Re: Add recovery to pg_control and remove backup_label

Hi,

On 2023-11-21 13:41:15 -0400, David Steele wrote:
> On 11/20/23 16:41, Andres Freund wrote:
> >
> > On 2023-11-20 15:56:19 -0400, David Steele wrote:
> > > I understand this is an option -- but does it need to be? What is the
> > > benefit of excluding the manifest?
> >
> > It's not free to create the manifest, particularly if checksums are enabled.
>
> It's virtually free, even with the basic CRCs.

Huh?

perf stat src/bin/pg_basebackup/pg_basebackup -h /tmp/ -p 5440 -D - -cfast -Xnone  --format=tar > /dev/null

          4,423.81 msec task-clock                       #    0.626 CPUs utilized
           433,475      context-switches                 #   97.987 K/sec
                 5      cpu-migrations                   #    1.130 /sec
               599      page-faults                      #  135.404 /sec
    12,208,261,153      cycles                           #    2.760 GHz
     6,805,401,520      instructions                     #    0.56  insn per cycle
     1,273,896,027      branches                         #  287.964 M/sec
        14,233,126      branch-misses                    #    1.12% of all branches

       7.068946385 seconds time elapsed

       1.106072000 seconds user
       3.403793000 seconds sys


perf stat src/bin/pg_basebackup/pg_basebackup -h /tmp/ -p 5440 -D - -cfast -Xnone  --format=tar
--manifest-checksums=CRC32C> /dev/null
 

          4,324.64 msec task-clock                       #    0.640 CPUs utilized
           433,306      context-switches                 #  100.195 K/sec
                 3      cpu-migrations                   #    0.694 /sec
               598      page-faults                      #  138.277 /sec
    11,952,475,908      cycles                           #    2.764 GHz
     6,816,888,845      instructions                     #    0.57  insn per cycle
     1,275,949,455      branches                         #  295.042 M/sec
        13,721,376      branch-misses                    #    1.08% of all branches

       6.760321433 seconds time elapsed

       1.113256000 seconds user
       3.302907000 seconds sys

perf stat src/bin/pg_basebackup/pg_basebackup -h /tmp/ -p 5440 -D - -cfast -Xnone  --format=tar --no-manifest >
/dev/null

          3,925.38 msec task-clock                       #    0.823 CPUs utilized
           257,467      context-switches                 #   65.590 K/sec
                 4      cpu-migrations                   #    1.019 /sec
               552      page-faults                      #  140.624 /sec
    11,577,054,842      cycles                           #    2.949 GHz
     5,933,731,797      instructions                     #    0.51  insn per cycle
     1,108,784,719      branches                         #  282.466 M/sec
        11,867,511      branch-misses                    #    1.07% of all branches

       4.770347012 seconds time elapsed

       1.002521000 seconds user
       2.991769000 seconds sys


I'd not call 7.06->4.77 or 6.76->4.77 "virtually free".


And this actually *under* selling the cost - we waste a lot of cycles due to
bad buffering decisions. Once we fix that, the cost differential increases
further.


> Anyway, would you really want a backup without a manifest? How would you
> know something is missing? In particular, for page incremental how do you
> know something is new (but not WAL logged) if there is no manifest? Is the
> plan to just recopy anything not WAL logged with each incremental?

Shrug. If you just want to create a new standby by copying the primary, I
don't think creating and then validating the manifest buys you much. Long term
backups are a different story, particularly if data files are stored
individually, rather than in a single checksummed file.


> > Also, for external backups, there's no manifest...
>
> There certainly is a manifest for many external backup solutions. Not having
> a manifest is just running with scissors, backup-wise.

You mean that you have an external solution gin up a backup manifest? I fail
to see how that's relevant here?

Greetings,

Andres Freund