Re: Rewriting the test of pg_upgrade as a TAP test - take three - remastered set

On Sat, May 15, 2021 at 02:22:24PM -0400, Andrew Dunstan wrote:
> PostgresNode is currently able to create nodes suitable for upgrade down
> to release 10. If we add '-w' to the 'pg_ctl start' flags that can
> extend down to release 9.5. (Just tested) I think we should do that
> forthwith. '-w' is now the default, but having it there explicitly does
> no harm.

Agreed.  When testing manually, I have personally never worked on any
patches that required binaries older than 9.4, so I would be fine if
the TAP tests are able to work easily down to 9.5, even if pg_upgrade
is supported down to 8.4.

> If people are interested in what's incompatible on older versions, they
> can look at
> <https://gitlab.com/adunstan/postgresnodeng/-/blob/master/PostgresNode.pm>
> starting at about line 2764.

We should really have adjust_conf() at some point in the in-core
module.

> I don't think this will work, though, unless there is enough data to
> exercise pg_upgrade fairly thoroughly. The approach taken by both
> test.sh and (somewhat more comprehensively) by the buildfarm cross
> version upgrade module is to test a cluster where the regression tests
> have been run.

Yeah, that's what my patch is doing with pg_regress, FWIW.  This
requires regress.so from the old version.

> That might be more difficult when testing against older
> versions, so I have published a snapshot of the dumps of each of the
> versions we tests against in the buildfarm animal crake. These could be
> loaded into PostgresNode instances and then an upgrade attempted. See
> <https://gitlab.com/adunstan/pg-old-bin/-/tree/master/data>. The data
> goes back to 9.2. These compressed dumps are a couple of megabytes each,
> not huge.

I agree that this can be simpler in some cases.  In your experience,
how much of an issue is it when it becomes necessary to keep around
binaries that rely on libraries older than what a system can support?
It is easy to counter issues in this area with OpenSSL and
non-necessary things, but we had in the past also cases where we had
code that conflicted with the kernel, aka 3e68686.

At the end of this exercise, what I think we should achieve is to:
1) Reduce the diff between the buildfarm code and the in-core code.
2) Get rid of test.sh.
3) Be able to run easily upgrade tests across major versions for
developers.

As of now, 3) requires some extra facilities depending on if this is
done by the buildfarm or the in-core tests:
1) Path to the source code of the old version.  This is required once
it becomes necessary to find out src/test/regress/ for the schedule,
the tests to run and its regress.so.  There is no need to do that if
you have a dump of the old instance.
2) Path to a custom dump to replace the run with pg_regress from 1).
3) Location of the old binaries, for pg_upgrade.  When it comes to
PostgresNode, we require an install path, so we cannot use directly
the location of the binaries.

Looking at the code of the buildfarm, its code does something smarter
than what my patch or HEAD's test.sh does now, as these require the
path for the old source.  The buildfarm code first scans for the
probin's used in the regression database and then updates any
references.  What test.sh and my patch do is using the path to the old
source code and run a single UPDATE.  The approach taken by the
buildfarm code is more portable, as a path to the old source code
becomes necessary only if running pg_regress manually.  So, what about
doing the following thing?
1) Update the TAP test so as probin entries are updated in the same way
as the buildfarm.
2) Allow one to specify a path to a custom dump, or a path to the old
source code for pg_regress.

If we do that, then it should be possible to reduce the code footprint
in the buildfarm code, while still allowing people to test major
upgrades in the same old-fashioned way, right?  That's assuming that
PostgresNode is made compatible down to 9.2, of course, as a first
step, as that's the range of the dumps you are keeping around for the
buildfarm.

Thoughts?
--
Michael