Обсуждение: Possible infinite loop on buildfarm animals

Between approximately 11:05 UTC and 13:25 UTC today (15 Mar 2024),
the Postgres git repo contained a buggy test recipe that caused
an infinite loop that will eventually exhaust disk space.
If you have any animals that might have launched a test run on
HEAD in that interval, you might want to check up on them.
A manual kill of the process that's consuming 100% CPU should
be enough to get out of it.

Another idea to consider is to set the wait_timeout parameter
in your animals' configuration files, to put an upper bound
on the total elapsed time for a run.  By default that's
infinite, since it's really hard to select a one-size-fits-all
value ... but it's a good backstop if you don't mind picking
machine-specific limits.

            regards, tom lane

On Fri, Mar 15, 2024 at 02:42:14PM -0400, Tom Lane wrote:
> Another idea to consider is to set the wait_timeout parameter
> in your animals' configuration files, to put an upper bound
> on the total elapsed time for a run.  By default that's
> infinite, since it's really hard to select a one-size-fits-all
> value ... but it's a good backstop if you don't mind picking
> machine-specific limits.

Other than CLOBBER_CACHE animals, the server rejects results older than 24h:
https://github.com/PGBuildFarm/server-code/blob/8572ac7/cgi-bin/pgstatus.pl#L273

The same 24h should probably be the default wait_timeout.  One might use a
longer timeout if wanting to attach a debugger to a process of a stuck run.
If one just wants an intervention-free buildfarm animal, 24h is good.

On 2024-Mar-17, Noah Misch wrote:

> Other than CLOBBER_CACHE animals, the server rejects results older than 24h:
> https://github.com/PGBuildFarm/server-code/blob/8572ac7/cgi-bin/pgstatus.pl#L273

> The same 24h should probably be the default wait_timeout.  One might use a
> longer timeout if wanting to attach a debugger to a process of a stuck run.
> If one just wants an intervention-free buildfarm animal, 24h is good.

Maybe that should be the default value embedded in the buildfarm client
script, which can be overridden for specific purposes such as
CLOBBER_CACHE animals?

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/
"No hay hombre que no aspire a la plenitud, es decir,
la suma de experiencias de que un hombre es capaz"

Alvaro Herrera <alvherre@alvh.no-ip.org> writes:
> On 2024-Mar-17, Noah Misch wrote:
>> Other than CLOBBER_CACHE animals, the server rejects results older than 24h:
>> https://github.com/PGBuildFarm/server-code/blob/8572ac7/cgi-bin/pgstatus.pl#L273

>> The same 24h should probably be the default wait_timeout.  One might use a
>> longer timeout if wanting to attach a debugger to a process of a stuck run.
>> If one just wants an intervention-free buildfarm animal, 24h is good.

> Maybe that should be the default value embedded in the buildfarm client
> script, which can be overridden for specific purposes such as
> CLOBBER_CACHE animals?

We don't normally hard-wire such choices in the script, but it could
be plausible to change build-farm.conf.sample, perhaps like:

    # max time in seconds allowed for a single branch run
    # undef/0 means unlimited
-    wait_timeout => undef,
+    wait_timeout => 24 * 60 * 60,

            regards, tom lane