Обсуждение: LOG: could not send data to client: Broken pipe
Hello Postgres Support Team,
I need your help to identify the issue and apply the fix for it.
My client is running one query through Application and getting error like below.
11:20:46.298 [http-nio-8083-exec-7] ERROR c.s.s.g.a.alcor.pdi.dao.PdiDao - While getting pdi from 2019-12-21 to 2020-06-01: An I/O error occurred while sending to the backend.
org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
When I checked in the postgres log I noticed the below.
2020-06-02 02:09:21.333 CEST db:[xxx] user:[xxxxx] [xxxxxxxxxx(53643)] [08006] LOG: could not send data to client: Broken pipe
2020-06-02 02:09:21.333 CEST db:[xxx] user:[xxxxx] [xxxxxxxxxxxx(53643)] [08006] STATEMENT: select ………………………………. (it just a select query)
2020-06-02 02:09:21.333 CEST db:[xxx] user:[xxxxxx] [xxxxxxxxxx(53643)] [08006] FATAL: connection to client lost
Note: DB name, hostname, username, sql text morphed in above message for security reasons, so kindly consider it.
I ran the above select query from psql prompt and I got the output, Time: 7178.926 ms. Ofcource my one is local connection so no broken pipe, where as client is using the network so he is getting broken pipe.
We checked with Network team they are claiming no issues with the network, But not sure.
I worked with Unix team and they raised a case to Red hat support and got confirmation from them that “no issues observed from OS End”
In-fact Red hat looking for a feedback of Postgres Community.
Version of Postgres : PostgreSQL 9.6.5 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11), 64-bit
OS : Red Hat Enterprise Linux Server release 7.2 (Maipo)
We observed “LOG: could not send data to client: Broken pipe” in postgres log from long time , But recently client complained about it. So we are in process of fixing it for which we need your help.
Fyi.. I had performed VACUUM(FULL, ANALYZE, VERBOSE) on the table. So the last analyzed is latest.
I already checked in Postgres archive and found below links on the same topic, But I could not get the concreate answer to fix the issue, so please help.
checked 11,12,13,14,15,16,17,18 mail replies
Let me know if you need any more details from my end.
Regards,
Narresh
=========================================================
Ce message et toutes les pieces jointes (ci-apres le "message")
sont confidentiels et susceptibles de contenir des informations
couvertes par le secret professionnel. Ce message est etabli
a l'intention exclusive de ses destinataires. Toute utilisation
ou diffusion non autorisee interdite.
Tout message electronique est susceptible d'alteration. La SOCIETE GENERALE
et ses filiales declinent toute responsabilite au titre de ce message
s'il a ete altere, deforme falsifie.
=========================================================
This message and any attachments (the "message") are confidential,
intended solely for the addresses, and may contain legally privileged
information. Any unauthorized use or dissemination is prohibited.
E-mails are susceptible to alteration. Neither SOCIETE GENERALE nor any
of its subsidiaries or affiliates shall be liable for the message
if altered, changed or falsified.
=========================================================
On 6/1/20 8:41 PM, TALLURI Nareshkumar wrote: > Hello Postgres Support Team, > > I need your help to identify the issue and apply the fix for it. > > My client is running one query through Application and getting error > like below. > > 11:20:46.298 [http-nio-8083-exec-7] ERROR c.s.s.g.a.alcor.pdi.dao.PdiDao > - While getting pdi from 2019-12-21 to 2020-06-01: An I/O error occurred > while sending to the backend. > > org.postgresql.util.PSQLException: An I/O error occurred while sending > to the backend. > > When I checked in the postgres log I noticed the below. > > 2020-06-02 02:09:21.333 CEST db:[xxx] user:[xxxxx] [xxxxxxxxxx(53643)] > [08006] LOG: could not send data to client: Broken pipe > > 2020-06-02 02:09:21.333 CEST db:[xxx] user:[xxxxx] [xxxxxxxxxxxx(53643)] > [08006] STATEMENT: select ………………………………. (it just a select query) > > 2020-06-02 02:09:21.333 CEST db:[xxx] user:[xxxxxx] [xxxxxxxxxx(53643)] > [08006] FATAL: connection to client lost > > Note: DB name, hostname, username, sql text morphed in above message > for security reasons, so kindly consider it. > > I ran the above select query from psql prompt and I got the output, > Time: 7178.926 ms. Ofcource my one is local connection so no broken > pipe, where as client is using the network so he is getting broken pipe. > > We checked with Network team they are claiming no > issues with the network, But not sure. > > I worked with Unix team and they raised a case to Red hat support and > got confirmation from them that “no issues observed from OS End” > > In-fact Red hat looking for a feedback of Postgres Community. > > Version of Postgres : PostgreSQL 9.6.5 on > x86_64-redhat-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat > 4.8.5-11), 64-bit > > OS : Red Hat > Enterprise Linux Server release 7.2 (Maipo) > > We observed “LOG: could not send data to client: Broken pipe” in > postgres log from long time , But recently client complained about it. > So we are in process of fixing it for which we need your help. > > Fyi.. I had performed VACUUM(FULL, ANALYZE, VERBOSE) on > the table. So the last analyzed is latest. > > I already checked in Postgres archive and found below links on the same > topic, But I could not get the concreate answer to fix the issue, so > please help. > > https://www.postgresql.org/search/?m=1&q=LOG%3A++could+not+send+data+to+client%3A+Broken+pipe&l=2&d=-1&s=i > > checked 11,12,13,14,15,16,17,18 mail replies > > Let me know if you need any more details from my end. Is there anything relevant in the OS system log around the time period 2020-06-02 02:09:21.333 CEST? > > Regards, > > Narresh -- Adrian Klaver adrian.klaver@aklaver.com
On 6/1/20 8:41 PM, TALLURI Nareshkumar wrote: > Hello Postgres Support Team, > > I need your help to identify the issue and apply the fix for it. > > My client is running one query through Application and getting error > like below. > > 11:20:46.298 [http-nio-8083-exec-7] ERROR c.s.s.g.a.alcor.pdi.dao.PdiDao > - While getting pdi from 2019-12-21 to 2020-06-01: An I/O error occurred > while sending to the backend. > > org.postgresql.util.PSQLException: An I/O error occurred while sending > to the backend. > Should have included in previous post: What version of JDBC? What is the application stack? Client OS? What is the network distance between the client/application and the server? -- Adrian Klaver adrian.klaver@aklaver.com
Hello Adrian Klaver,
Here is the answers
What version of JDBC? : 9.3-1104-jdbc41
What is the application stack? : Java
Client OS? : windows 2016
What is the network distance between the client/application and the server?
Reply from 184.6.160.214: bytes=32 time=2ms TTL=57
Reply from 184.6.160.214: bytes=32 time=2ms TTL=57
Reply from 184.6.160.214: bytes=32 time=2ms TTL=57
Reply from 184.6.160.214: bytes=32 time=2ms TTL=57
Ping statistics for 184.6.160.214:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 2ms, Maximum = 2ms, Average = 2ms
Regards,
Narresh
-----Original Message-----
From: Adrian Klaver <adrian.klaver@aklaver.com>
Sent: Tuesday, June 2, 2020 7:50 PM
To: TALLURI Nareshkumar SgscGtsPasDre <nareshkumar.talluri@socgen.com>; pgsql-general@postgresql.org
Subject: Re: LOG: could not send data to client: Broken pipe
On 6/1/20 8:41 PM, TALLURI Nareshkumar wrote:
> Hello Postgres Support Team,
>
> I need your help to identify the issue and apply the fix for it.
>
> My client is running one query through Application and getting error
> like below.
>
> 11:20:46.298 [http-nio-8083-exec-7] ERROR
> c.s.s.g.a.alcor.pdi.dao.PdiDao
> - While getting pdi from 2019-12-21 to 2020-06-01: An I/O error
> occurred while sending to the backend.
>
> org.postgresql.util.PSQLException: An I/O error occurred while sending
> to the backend.
>
Should have included in previous post:
What version of JDBC?
What is the application stack?
Client OS?
What is the network distance between the client/application and the server?
--
Adrian Klaver
adrian.klaver@aklaver.com
On 6/2/20 11:18 AM, TALLURI Nareshkumar wrote: > Hello Adrian Klaver, > > Here is the answers > > > What version of JDBC? : 9.3-1104-jdbc41 > > What is the application stack? : Java That is the programming language. I was looking for the actual applications used. > > Client OS? : windows 2016 When I see Windows and interruptions to Postgres I think anti-virus software. Is there any running on the client? > > What is the network distance between the client/application and the server? I should have been more specific: Are the client and server on the same network? Are the client/server separated by a building, country, etc? > Reply from 184.6.160.214: bytes=32 time=2ms TTL=57 > Reply from 184.6.160.214: bytes=32 time=2ms TTL=57 > Reply from 184.6.160.214: bytes=32 time=2ms TTL=57 > Reply from 184.6.160.214: bytes=32 time=2ms TTL=57 > > Ping statistics for 184.6.160.214: > Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), > Approximate round trip times in milli-seconds: > Minimum = 2ms, Maximum = 2ms, Average = 2ms > > > Regards, > Narresh > > > > > -----Original Message----- > From: Adrian Klaver <adrian.klaver@aklaver.com> > Sent: Tuesday, June 2, 2020 7:50 PM > To: TALLURI Nareshkumar SgscGtsPasDre <nareshkumar.talluri@socgen.com>; pgsql-general@postgresql.org > Subject: Re: LOG: could not send data to client: Broken pipe > > On 6/1/20 8:41 PM, TALLURI Nareshkumar wrote: >> Hello Postgres Support Team, >> >> I need your help to identify the issue and apply the fix for it. >> >> My client is running one query through Application and getting error >> like below. >> >> 11:20:46.298 [http-nio-8083-exec-7] ERROR >> c.s.s.g.a.alcor.pdi.dao.PdiDao >> - While getting pdi from 2019-12-21 to 2020-06-01: An I/O error >> occurred while sending to the backend. >> >> org.postgresql.util.PSQLException: An I/O error occurred while sending >> to the backend. >> > > Should have included in previous post: > > What version of JDBC? > > What is the application stack? > > Client OS? > > What is the network distance between the client/application and the server? > > > -- > Adrian Klaver > adrian.klaver@aklaver.com > -- Adrian Klaver adrian.klaver@aklaver.com
Hello Adrian, What is the application stack? - I will check with Application team and get back to you. When I see Windows and interruptions to Postgres I think anti-virus software. Is there any running on the client? ----yes , symantec Are the client and server on the same network? yes Are the client/server separated by a building, country, etc? same country , But different Datacenter Regards, Narresh -----Original Message----- From: Adrian Klaver <adrian.klaver@aklaver.com> Sent: Wednesday, June 3, 2020 12:38 AM To: TALLURI Nareshkumar SgscGtsPasDre <nareshkumar.talluri@socgen.com>; pgsql-general@postgresql.org Subject: Re: LOG: could not send data to client: Broken pipe On 6/2/20 11:18 AM, TALLURI Nareshkumar wrote: > Hello Adrian Klaver, > > Here is the answers > > > What version of JDBC? : 9.3-1104-jdbc41 > > What is the application stack? : Java That is the programming language. I was looking for the actual applications used. > > Client OS? : windows 2016 When I see Windows and interruptions to Postgres I think anti-virus software. Is there any running on the client? > > What is the network distance between the client/application and the server? I should have been more specific: Are the client and server on the same network? Are the client/server separated by a building, country, etc? > Reply from 184.6.160.214: bytes=32 time=2ms TTL=57 Reply from > 184.6.160.214: bytes=32 time=2ms TTL=57 Reply from 184.6.160.214: > bytes=32 time=2ms TTL=57 Reply from 184.6.160.214: bytes=32 time=2ms > TTL=57 > > Ping statistics for 184.6.160.214: > Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate > round trip times in milli-seconds: > Minimum = 2ms, Maximum = 2ms, Average = 2ms > > > Regards, > Narresh > > > > > -----Original Message----- > From: Adrian Klaver <adrian.klaver@aklaver.com> > Sent: Tuesday, June 2, 2020 7:50 PM > To: TALLURI Nareshkumar SgscGtsPasDre > <nareshkumar.talluri@socgen.com>; pgsql-general@postgresql.org > Subject: Re: LOG: could not send data to client: Broken pipe > > On 6/1/20 8:41 PM, TALLURI Nareshkumar wrote: >> Hello Postgres Support Team, >> >> I need your help to identify the issue and apply the fix for it. >> >> My client is running one query through Application and getting error >> like below. >> >> 11:20:46.298 [http-nio-8083-exec-7] ERROR >> c.s.s.g.a.alcor.pdi.dao.PdiDao >> - While getting pdi from 2019-12-21 to 2020-06-01: An I/O error >> occurred while sending to the backend. >> >> org.postgresql.util.PSQLException: An I/O error occurred while >> sending to the backend. >> > > Should have included in previous post: > > What version of JDBC? > > What is the application stack? > > Client OS? > > What is the network distance between the client/application and the server? > > > -- > Adrian Klaver > adrian.klaver@aklaver.com > -- Adrian Klaver adrian.klaver@aklaver.com
On 6/2/20 12:20 PM, TALLURI Nareshkumar wrote: > Hello Adrian, > > > What is the application stack? - I will check with Application team and get back to you. > > When I see Windows and interruptions to Postgres I think anti-virus software. Is there any running on the client? ----yes , symantec Probably should check if AV software is logging anything at the time of the broken pipes. > > Are the client and server on the same network? yes > > Are the client/server separated by a building, country, etc? same country , But different Datacenter > > > Regards, > Narresh > > -----Original Message----- > From: Adrian Klaver <adrian.klaver@aklaver.com> > Sent: Wednesday, June 3, 2020 12:38 AM > To: TALLURI Nareshkumar SgscGtsPasDre <nareshkumar.talluri@socgen.com>; pgsql-general@postgresql.org > Subject: Re: LOG: could not send data to client: Broken pipe > > On 6/2/20 11:18 AM, TALLURI Nareshkumar wrote: >> Hello Adrian Klaver, >> >> Here is the answers >> >> >> What version of JDBC? : 9.3-1104-jdbc41 >> >> What is the application stack? : Java > > That is the programming language. I was looking for the actual applications used. > >> >> Client OS? : windows 2016 > > When I see Windows and interruptions to Postgres I think anti-virus software. Is there any running on the client? > >> >> What is the network distance between the client/application and the server? > > I should have been more specific: > > Are the client and server on the same network? > > Are the client/server separated by a building, country, etc? > >> Reply from 184.6.160.214: bytes=32 time=2ms TTL=57 Reply from >> 184.6.160.214: bytes=32 time=2ms TTL=57 Reply from 184.6.160.214: >> bytes=32 time=2ms TTL=57 Reply from 184.6.160.214: bytes=32 time=2ms >> TTL=57 >> >> Ping statistics for 184.6.160.214: >> Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate >> round trip times in milli-seconds: >> Minimum = 2ms, Maximum = 2ms, Average = 2ms >> >> >> Regards, >> Narresh >> >> >> >> >> -----Original Message----- >> From: Adrian Klaver <adrian.klaver@aklaver.com> >> Sent: Tuesday, June 2, 2020 7:50 PM >> To: TALLURI Nareshkumar SgscGtsPasDre >> <nareshkumar.talluri@socgen.com>; pgsql-general@postgresql.org >> Subject: Re: LOG: could not send data to client: Broken pipe >> >> On 6/1/20 8:41 PM, TALLURI Nareshkumar wrote: >>> Hello Postgres Support Team, >>> >>> I need your help to identify the issue and apply the fix for it. >>> >>> My client is running one query through Application and getting error >>> like below. >>> >>> 11:20:46.298 [http-nio-8083-exec-7] ERROR >>> c.s.s.g.a.alcor.pdi.dao.PdiDao >>> - While getting pdi from 2019-12-21 to 2020-06-01: An I/O error >>> occurred while sending to the backend. >>> >>> org.postgresql.util.PSQLException: An I/O error occurred while >>> sending to the backend. >>> >> >> Should have included in previous post: >> >> What version of JDBC? >> >> What is the application stack? >> >> Client OS? >> >> What is the network distance between the client/application and the server? >> >> >> -- >> Adrian Klaver >> adrian.klaver@aklaver.com >> > > > -- > Adrian Klaver > adrian.klaver@aklaver.com > -- Adrian Klaver adrian.klaver@aklaver.com