Обсуждение: Why do pg_upgrade's test use the serial schedule?

Hi,

The test I'm most commonly waiting for when doing a parallel make check
is pg_upgrade. And to a significant degree that's because it uses the
serial installcheck rather than installcheck-parallel.

Is there a reason to not use installcheck-parallel?

serial:
19.42user 7.77system 1:53.23elapsed 24%CPU (0avgtext+0avgdata 111420maxresident)k
parallel:
20.76user 7.72system 1:27.05elapsed 32%CPU (0avgtext+0avgdata 112300maxresident)k

So, a saving of ~25s.


ISTM we also should disable fsyncs for the postmaster instances. Wins
another ~5s on my laptop, and I suspect it makes a larger difference on
some of the buildfarm animals.

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> ISTM we also should disable fsyncs for the postmaster instances. Wins
> another ~5s on my laptop, and I suspect it makes a larger difference on
> some of the buildfarm animals.

Buildfarm did that long ago.

            regards, tom lane

On 2018-11-05 16:10:28 -0500, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > ISTM we also should disable fsyncs for the postmaster instances. Wins
> > another ~5s on my laptop, and I suspect it makes a larger difference on
> > some of the buildfarm animals.
> 
> Buildfarm did that long ago.

I don't think it did for pg_upgrade's test.sh?

POSTMASTER_OPTS="-F -c listen_addresses=\"$LISTEN_ADDRESSES\" -k \"$PGHOST\""
...
"$oldbindir"/pg_ctl start -l "$logdir/postmaster1.log" -o "$POSTMASTER_OPTS" -w

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> On 2018-11-05 16:10:28 -0500, Tom Lane wrote:
>> Andres Freund <andres@anarazel.de> writes:
>>> ISTM we also should disable fsyncs for the postmaster instances. Wins
>>> another ~5s on my laptop, and I suspect it makes a larger difference on
>>> some of the buildfarm animals.

>> Buildfarm did that long ago.

> I don't think it did for pg_upgrade's test.sh?

It's injected via the TEMP_CONFIG env variable.

            regards, tom lane

On 2018-11-05 16:32:20 -0500, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > On 2018-11-05 16:10:28 -0500, Tom Lane wrote:
> >> Andres Freund <andres@anarazel.de> writes:
> >>> ISTM we also should disable fsyncs for the postmaster instances. Wins
> >>> another ~5s on my laptop, and I suspect it makes a larger difference on
> >>> some of the buildfarm animals.
> 
> >> Buildfarm did that long ago.
> 
> > I don't think it did for pg_upgrade's test.sh?
> 
> It's injected via the TEMP_CONFIG env variable.

Ah, thanks.  Is there any good reason to not instead have it in the
script? Doesn't strike me as great that it slows down normal regression
runs during development?  Injecting TEMP_CONFIG w/ fsync=off globally
for those (rather than in a single test wrapper), could counteract
explicit tests where fsync is wanted - beside the fact that one has to
do so manually.  Normal regression tests have long done so.

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> On 2018-11-05 16:32:20 -0500, Tom Lane wrote:
>> It's injected via the TEMP_CONFIG env variable.

> Ah, thanks.  Is there any good reason to not instead have it in the
> script?

Why that script in particular?  If you don't want fsync, you likely
don't want it across the entire check-world run.  The TEMP_CONFIG
thing is a hack no doubt, but it gets the job done globally.  Also,
if you *do* want fsync, there's one place to turn it back on.  I'm
not a fan of individual tests deciding they know what to do.

            regards, tom lane

On 2018-11-05 17:08:12 -0500, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > On 2018-11-05 16:32:20 -0500, Tom Lane wrote:
> >> It's injected via the TEMP_CONFIG env variable.
> 
> > Ah, thanks.  Is there any good reason to not instead have it in the
> > script?
> 
> Why that script in particular?

Because just about everything else taking substantial time already
disables it. pg_regress forks off a postmaster with -F, the tap tests do
so for large portions via fsync=off in the config (c.f. PostgresNode.pm).

Greetings,

Andres Freund

On Mon, Nov 05, 2018 at 02:11:42PM -0800, Andres Freund wrote:
> Because just about everything else taking substantial time already
> disables it. pg_regress forks off a postmaster with -F, the tap tests do
> so for large portions via fsync=off in the config (c.f. PostgresNode.pm).

And in most cases people will forget to set up TEMP_CONFIG when setting
up a new environment.  If it is possible to reduce the I/O traffic when
running check-world by default and if we are able to make it faster with
a parallel schedule, my take is to do by default instead of expecting
the user to set up that all the time.  That's less to think about.

So I am +1 and +1 for Andres' suggestions on this thread.
--
Michael

On 05/11/2018 22:00, Andres Freund wrote:
> The test I'm most commonly waiting for when doing a parallel make check
> is pg_upgrade. And to a significant degree that's because it uses the
> serial installcheck rather than installcheck-parallel.
> 
> Is there a reason to not use installcheck-parallel?

Probably worth trying.  Using serial was probably just the conservative
choice at the time.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Hi,

I unfortunately copied the wrong message id into a bunch of commit
messages, and just noticed now. Messages pointing to this message's
parent via:
Discussion: https://postgr.es/m/20181105210039.hh4vvi4vwoq5ba2q@alap3.anarazel.de
ought to have been
Discussion: https://postgr.es/m/20180220224318.gw4oe5jadhpmcdnm@alap3.anarazel.de

Greetings,

Andres Freund