Обсуждение: Broken pipe

Hi all,

I sent the following email to the php mailing list also but maybe this
is a more appropriate mailing list.

I wrote a php script which is running very long queries (hours) on a
database.
I seem to have a problem to run the code when there are single queries
which take long times (like 5 hours for an update query), from the log
of the database I received the following code:

2005-09-30 17:12:13 IDT postgres : LOG:  00000: duration: 18730038.678
ms  statement: UPDATE product_temp SET nleft=(SELECT
2005-09-30 17:12:13 IDT postgres : LOCATION:  exec_simple_query,
postgres.c:1035
2005-09-30 17:12:13 IDT postgres : LOG:  08006: could not send data to
client: Broken pipe
2005-09-30 17:12:13 IDT postgres : LOCATION:  internal_flush, pqcomm.c:1050
2005-09-30 17:12:13 IDT postgres : LOG:  08P01: unexpected EOF on client
connection
2005-09-30 17:12:13 IDT postgres : LOCATION:  SocketBackend, postgres.c:287
2005-09-30 17:12:13 IDT postgres : LOG:  00000: disconnection: session
time: 6:04:58.52
2005-09-30 17:12:13 IDT postgres : LOCATION:  log_disconnections,
postgres.c:3403

Now after the 5 hours update it need to echo into a log file a line
which say that it ended this command (just for me to know the times), my
assumption is that PHP read the code into memory at start and opened the
connection to the file, after a time which he waited to any given "life
sign" he gave up and closed the connection to the file, and when the
code came back to the file it encountered no connection to the file
(broken pipe).

Am I correct at my assumption? if so how can I set the PHP to wait how
much I tell him?
Ofcourse if im wrong I would like to know the reason also :)


Thanks in advance,
  Ben-Nes Yonatan

On Sun, Oct 02, 2005 at 12:07:18PM +0200, Ben-Nes Yonatan wrote:
> I wrote a php script which is running very long queries (hours) on a
> database.
> I seem to have a problem to run the code when there are single queries
> which take long times (like 5 hours for an update query), from the log
> of the database I received the following code:

My experience with long running backend queries is that IE gives up
after a while but Netscape hangs on a lot longer. I don't know if this
is still the case. It was a while while I worked out why it worked for
me but not my boss...

You need to work who is giving up, the client which causes PHP to give
up, or if PHP itself is giving up. PostgreSQL doesn't care overly much,
as long as you complete the transaction.

My suggestion would be to spawn the update into a seperate process and
send a "Processing, please wait" to the client. When the stuff is done,
set a flag and indicate to the client that processing is complete.

Hope this helps,
--
Martijn van Oosterhout   <kleptog@svana.org>   http://svana.org/kleptog/
> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a
> tool for doing 5% of the work and then sitting around waiting for someone
> else to do the other 95% so you can sue them.

Martijn van Oosterhout wrote:

>On Sun, Oct 02, 2005 at 12:07:18PM +0200, Ben-Nes Yonatan wrote:
>
>
>>I wrote a php script which is running very long queries (hours) on a
>>database.
>>I seem to have a problem to run the code when there are single queries
>>which take long times (like 5 hours for an update query), from the log
>>of the database I received the following code:
>>
>>
>
>My experience with long running backend queries is that IE gives up
>after a while but Netscape hangs on a lot longer. I don't know if this
>is still the case. It was a while while I worked out why it worked for
>me but not my boss...
>
>You need to work who is giving up, the client which causes PHP to give
>up, or if PHP itself is giving up. PostgreSQL doesn't care overly much,
>as long as you complete the transaction.
>
>My suggestion would be to spawn the update into a seperate process and
>send a "Processing, please wait" to the client. When the stuff is done,
>set a flag and indicate to the client that processing is complete.
>
>Hope this helps,
>
>
Well this is not the case (sadly.. :)) cause the process is running at
the server without any browser interface.

Am Sonntag, den 02.10.2005, 16:39 +0200 schrieb Ben-Nes Yonatan:
> Martijn van Oosterhout wrote:
...
> Well this is not the case (sadly.. :)) cause the process is running at
> the server without any browser interface.

Use the web as trigger only and a persistent long running process
to read the job table and do the action - including setting a flag
when its done.

----- Original Message -----
From: "Tino Wildenhain" <tino@wildenhain.de>
To: "Ben-Nes Yonatan" <da@canaan.co.il>
Cc: "Martijn van Oosterhout" <kleptog@svana.org>;
<pgsql-general@postgresql.org>
Sent: Sunday, October 02, 2005 4:26 PM
Subject: Re: [GENERAL] Broken pipe


> Am Sonntag, den 02.10.2005, 16:39 +0200 schrieb Ben-Nes Yonatan:
>> Martijn van Oosterhout wrote:
> ...
>> Well this is not the case (sadly.. :)) cause the process is running at
>> the server without any browser interface.
>
> Use the web as trigger only and a persistent long running process
> to read the job table and do the action - including setting a flag
> when its done.
>

Why should I use the web as a trigger (I assume that you mean with a
browser) when im running it from the command line?
I dont understand what do you mean with "a persistent long running process
to read the job table", Its one process which need to be run step by step,
with small files (total of less then 1.7 million rows) its working gr8 but
when I try to load it with more data it just stop with the error:
2005-09-30 17:12:13 IDT postgres : LOG:  00000: duration: 18730038.678
ms  statement: UPDATE product_temp SET nleft=(SELECT
2005-09-30 17:12:13 IDT postgres : LOCATION:  exec_simple_query,
postgres.c:1035
2005-09-30 17:12:13 IDT postgres : LOG:  08006: could not send data to
client: Broken pipe
2005-09-30 17:12:13 IDT postgres : LOCATION:  internal_flush, pqcomm.c:1050
2005-09-30 17:12:13 IDT postgres : LOG:  08P01: unexpected EOF on client
connection
2005-09-30 17:12:13 IDT postgres : LOCATION:  SocketBackend, postgres.c:287
2005-09-30 17:12:13 IDT postgres : LOG:  00000: disconnection: session
time: 6:04:58.52 user=postgres database=poweraise.com
2005-09-30 17:12:13 IDT postgres : LOCATION:  log_disconnections,
postgres.c:3403

I also tried persistent connection (thought that maybe you meant that even
if it unlikely) and it didnt work too.
About the flag I guess that its important only if the process need a browser
as a trigger.

Shana Tova! (hebrew happy new year :))
  Ben-Nes Yonatan