Обсуждение: meson and check-tests

On Fri May 31, 2024 at 12:02 PM CDT, Ashutosh Bapat wrote:
> Hi Tristan,
> Using make I can run only selected tests under src/test/regress using
> TESTS=... make check-tests. I didn't find any way to do that with meson.
> meson test --suite regress runs all the regression tests.
>
> We talked this off-list at the conference. It seems we have to somehow
> avoid passing pg_regress --schedule argument and instead pass the list of
> tests. Any idea how to do that?

I think there are 2 solutions to this.

1. Avoid passing --schedule by default, which doesn't sound like a great
   solution.

2. Teach pg_regress to ignore the --schedule option if specific tests
   are passed instead.

3. Add a --no-schedule option to pg_regress which would override the
   previously added --schedule option.

I personally prefer 2 or 3.

2: meson test -C build regress/regress --test-args my_specific_test
3: meson test -C build regress/regress --test-args "--no-schedule my_specific_test"

Does anyone have an opinion?

--
Tristan Partin
https://tristan.partin.io

On Sun, Jun 2, 2024 at 4:48 AM Tristan Partin <tristan@partin.io> wrote:
>
> On Fri May 31, 2024 at 12:02 PM CDT, Ashutosh Bapat wrote:
> > Hi Tristan,
> > Using make I can run only selected tests under src/test/regress using
> > TESTS=... make check-tests. I didn't find any way to do that with meson.
> > meson test --suite regress runs all the regression tests.
> >
> > We talked this off-list at the conference. It seems we have to somehow
> > avoid passing pg_regress --schedule argument and instead pass the list of
> > tests. Any idea how to do that?
>
> I think there are 2 solutions to this.
>
> 1. Avoid passing --schedule by default, which doesn't sound like a great
>    solution.
>
> 2. Teach pg_regress to ignore the --schedule option if specific tests
>    are passed instead.
>
> 3. Add a --no-schedule option to pg_regress which would override the
>    previously added --schedule option.
>
> I personally prefer 2 or 3.
>
> 2: meson test -C build regress/regress --test-args my_specific_test
> 3: meson test -C build regress/regress --test-args "--no-schedule my_specific_test"
>
> Does anyone have an opinion?
>

if each individual sql file can be tested separately, will
`test: test_setup`?
be aslo executed as a prerequisite?



in
# ----------
# The first group of parallel tests
# ----------
test: boolean char name varchar text int2 int4 int8 oid float4 float8
bit numeric txid uuid enum money rangetypes pg_lsn regproc

# ----------
# The second group of parallel tests
# multirangetypes depends on rangetypes
# multirangetypes shouldn't run concurrently with type_sanity
# ----------
test: strings md5 numerology point lseg line box path polygon circle
date time timetz timestamp timestamptz interval inet macaddr macaddr8
multirangetypes


If we can test each individual sql file via meson, that would be great.
but based on the above comments, we need to preserve the specified order.
like:
TEST=rangetypes, multirangetypes

will first run rangetypes then multirangetypes.


can we tag/name each src/test/regress/parallel_schedule groups
like:
group1:test: boolean char name varchar text int2 int4 int8 oid float4
float8 bit numeric txid uuid enum money rangetypes pg_lsn regproc
group2:test: strings md5 numerology point lseg line box path polygon
circle date time timetz timestamp timestamptz interval inet macaddr
macaddr8 multirangetypes

Then, we can test each individual group.
TEST=group1, group2.

"Tristan Partin" <tristan@partin.io> writes:
> On Fri May 31, 2024 at 12:02 PM CDT, Ashutosh Bapat wrote:
>> We talked this off-list at the conference. It seems we have to somehow
>> avoid passing pg_regress --schedule argument and instead pass the list of
>> tests. Any idea how to do that?

> I think there are 2 solutions to this.
> 1. Avoid passing --schedule by default, which doesn't sound like a great
>    solution.
> 2. Teach pg_regress to ignore the --schedule option if specific tests
>    are passed instead.
> 3. Add a --no-schedule option to pg_regress which would override the
>    previously added --schedule option.
> I personally prefer 2 or 3.

Just to refresh peoples' memory of what the Makefiles do:
src/test/regress/GNUmakefile has

check: all
    $(pg_regress_check) $(REGRESS_OPTS) --schedule=$(srcdir)/parallel_schedule $(MAXCONNOPT) $(EXTRA_TESTS)

check-tests: all | temp-install
    $(pg_regress_check) $(REGRESS_OPTS) $(MAXCONNOPT) $(TESTS) $(EXTRA_TESTS)

(and parallel cases for installcheck etc).  AFAICS, meson.build has
no equivalent to the EXTRA_TESTS add-on, nor does it have behavior
equivalent to check-tests' substitution of $(TESTS) for --schedule.
But I suggest that those behaviors have stood for a long time and
so the appropriate thing to do is duplicate them as best we can,
not invent something different.

            regards, tom lane

On 2024-06-02 Su 01:25, Tom Lane wrote:
> "Tristan Partin" <tristan@partin.io> writes:
>> On Fri May 31, 2024 at 12:02 PM CDT, Ashutosh Bapat wrote:
>>> We talked this off-list at the conference. It seems we have to somehow
>>> avoid passing pg_regress --schedule argument and instead pass the list of
>>> tests. Any idea how to do that?
>> I think there are 2 solutions to this.
>> 1. Avoid passing --schedule by default, which doesn't sound like a great
>>     solution.
>> 2. Teach pg_regress to ignore the --schedule option if specific tests
>>     are passed instead.
>> 3. Add a --no-schedule option to pg_regress which would override the
>>     previously added --schedule option.
>> I personally prefer 2 or 3.
> Just to refresh peoples' memory of what the Makefiles do:
> src/test/regress/GNUmakefile has
>
> check: all
>     $(pg_regress_check) $(REGRESS_OPTS) --schedule=$(srcdir)/parallel_schedule $(MAXCONNOPT) $(EXTRA_TESTS)
>
> check-tests: all | temp-install
>     $(pg_regress_check) $(REGRESS_OPTS) $(MAXCONNOPT) $(TESTS) $(EXTRA_TESTS)
>
> (and parallel cases for installcheck etc).  AFAICS, meson.build has
> no equivalent to the EXTRA_TESTS add-on, nor does it have behavior
> equivalent to check-tests' substitution of $(TESTS) for --schedule.
> But I suggest that those behaviors have stood for a long time and
> so the appropriate thing to do is duplicate them as best we can,
> not invent something different.
>
>             


+1


cheers


andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Hi,

On Sun, 2 Jun 2024 at 07:06, jian he <jian.universality@gmail.com> wrote:
>
> On Sun, Jun 2, 2024 at 4:48 AM Tristan Partin <tristan@partin.io> wrote:
> >
> > On Fri May 31, 2024 at 12:02 PM CDT, Ashutosh Bapat wrote:
> > > Hi Tristan,
> > > Using make I can run only selected tests under src/test/regress using
> > > TESTS=... make check-tests. I didn't find any way to do that with meson.
> > > meson test --suite regress runs all the regression tests.
> > >
> > > We talked this off-list at the conference. It seems we have to somehow
> > > avoid passing pg_regress --schedule argument and instead pass the list of
> > > tests. Any idea how to do that?
> >
> > I think there are 2 solutions to this.
> >
> > 1. Avoid passing --schedule by default, which doesn't sound like a great
> >    solution.
> >
> > 2. Teach pg_regress to ignore the --schedule option if specific tests
> >    are passed instead.
> >
> > 3. Add a --no-schedule option to pg_regress which would override the
> >    previously added --schedule option.
> >
> > I personally prefer 2 or 3.
> >
> > 2: meson test -C build regress/regress --test-args my_specific_test
> > 3: meson test -C build regress/regress --test-args "--no-schedule my_specific_test"
> >
> > Does anyone have an opinion?
> >
>
> if each individual sql file can be tested separately, will
> `test: test_setup`?
> be aslo executed as a prerequisite?

Yes, it is required to run at least two setup tests because regress
tests require a database to be created:

1- The initdb executable is needed, and it is installed by the
'tmp_install' test for the tests.

2- Although the initdb executable exists, it is not enough by itself.
Regress tests copies its database contents from the template
directory, instead of running initdb for each test [1] (This is the
default behaviour in the meson builds' regress tests). This template
directory is created by the 'initdb_cache' test.

[1] 252dcb3239

--
Regards,
Nazir Bilal Yavuz
Microsoft

On Sun Jun 2, 2024 at 12:25 AM CDT, Tom Lane wrote:
> "Tristan Partin" <tristan@partin.io> writes:
> > On Fri May 31, 2024 at 12:02 PM CDT, Ashutosh Bapat wrote:
> >> We talked this off-list at the conference. It seems we have to somehow
> >> avoid passing pg_regress --schedule argument and instead pass the list of
> >> tests. Any idea how to do that?
>
> > I think there are 2 solutions to this.
> > 1. Avoid passing --schedule by default, which doesn't sound like a great
> >    solution.
> > 2. Teach pg_regress to ignore the --schedule option if specific tests
> >    are passed instead.
> > 3. Add a --no-schedule option to pg_regress which would override the
> >    previously added --schedule option.
> > I personally prefer 2 or 3.
>
> Just to refresh peoples' memory of what the Makefiles do:
> src/test/regress/GNUmakefile has
>
> check: all
>     $(pg_regress_check) $(REGRESS_OPTS) --schedule=$(srcdir)/parallel_schedule $(MAXCONNOPT) $(EXTRA_TESTS)
>
> check-tests: all | temp-install
>     $(pg_regress_check) $(REGRESS_OPTS) $(MAXCONNOPT) $(TESTS) $(EXTRA_TESTS)
>
> (and parallel cases for installcheck etc).  AFAICS, meson.build has
> no equivalent to the EXTRA_TESTS add-on, nor does it have behavior
> equivalent to check-tests' substitution of $(TESTS) for --schedule.
> But I suggest that those behaviors have stood for a long time and
> so the appropriate thing to do is duplicate them as best we can,
> not invent something different.

In theory, this makes sense. In practice, this is hard to emulate. We
could make the check-tests a Meson run_target() instead of another
test(), which would end up running the same tests more than once.

We could also add the same functionality proposed in my email to the
Makefile, so they mirror each other. check-tests would then just become
a legacy target name.

--
Tristan Partin
https://tristan.partin.io