On Wed, Aug 12, 2015 at 7:20 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> FWIW, I've objected in the past to tests that would significantly
> increase the runtime of "make check", unless I thought they were
> especially valuable (which enumerating every minor behavior of a
> feature patch generally isn't IMO). I still think that that's an
> important consideration: every second you add to "make check" is
> multiplied many times over when you consider how many developers
> run that how many times a day.
>
> We've talked about having some sort of second rank of tests that
> people wouldn't necessarily run before committing, and that would
> be allowed to eat more time than the core regression tests would.
> I think that might be a valuable direction to pursue if people start
> submitting very bulky tests.
Maybe. Adding a whole new test suite is significantly more
administratively complex, because the BF client has to get updated to
run it. And if expected outputs in that test suite change very often
at all, then committers will have to run it before committing anyway.
The value of a core regression suite that takes less time to run has
to be weighed against the possibility that a better core regression
suite might cause us to find more bugs before committing. That could
easily be worth the price in runtime.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company