Обсуждение: Segmentation fault: pg_upgrade 9.1 to 9.3: pg_dump: row number 0 is out of range 0..-1

Поиск
Список
Период
Сортировка

Segmentation fault: pg_upgrade 9.1 to 9.3: pg_dump: row number 0 is out of range 0..-1

От
Robert Nix
Дата:
Running a pg_upgrade task is causing Segmentation fault:

command: "/usr/lib/postgresql/9.3/bin/pg_dump" --host "/var/lib/postgresql" --port 50432 --username "postgres" --schema-only --quote-all-identifiers --binary-upgrade --format=custom  --file="pg_upgrade_dump_6064585.custom" "u" >> "pg_upgrade_dump_6064585.log" 2>&1
pg_dump: row number 0 is out of range 0..-1
Segmentation fault (core dumped)

Running this command:

/usr/lib/postgresql/9.3/bin/pg_upgrade \
  -b /usr/lib/postgresql/9.1/bin       \
  -d /var/lib/postgresql/9.1/main      \
  -o "-c config_file=/etc/postgresql/9.1/main/postgresql.conf" \
  -B /usr/lib/postgresql/9.3/bin       \
  -D /var/lib/postgresql/9.3/main      \
  -O "-c config_file=/etc/postgresql/9.3/main/postgresql.conf"

Any tips on how to diagnose and/or what might be the problem?

I've verified that the same extensions are present in 9.3 as exist in 9.1. I'm not really sure how to proceed.

Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok
Checking database user is a superuser                       ok
Checking for prepared transactions                          ok
Checking for reg* system OID user data types                ok
Checking for contrib/isn with bigint-passing mismatch       ok
Creating dump of global objects                             ok
Creating dump of database schemas

It then goes through about 3 or 4 databases and then faults.

gdb doesn't seem helpful:

$ gdb /usr/lib/postgresql/9.3/bin/pg_dump core
GNU gdb (GDB) 7.5-ubuntu
Reading symbols from /usr/lib/postgresql/9.3/bin/pg_dump...(no debugging symbols found)...done.
[New LWP 1180]

warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/lib/postgresql/9.3/bin/pg_dump --host /var/lib/postgresql --port 50432 --u'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007fd05f7c2284 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x00007fd05f7c2284 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fd0601c1200 in ?? ()
#2  0x00007fd0601ca405 in ?? ()
#3  0x00007fd0601b5ffb in main ()

Thanks

Linux db 3.5.0-39-generic #60-Ubuntu SMP Tue Aug 13 18:33:05 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Distributor ID: Ubuntu
Description:    Ubuntu 12.10
Release:        12.10
Codename:       quantal

Re: Segmentation fault: pg_upgrade 9.1 to 9.3: pg_dump: row number 0 is out of range 0..-1

От
Jeff Janes
Дата:
On Sat, Sep 14, 2013 at 6:40 PM, Robert Nix <robert@urban4m.com> wrote:
Running a pg_upgrade task is causing Segmentation fault:

command: "/usr/lib/postgresql/9.3/bin/pg_dump" --host "/var/lib/postgresql" --port 50432 --username "postgres" --schema-only --quote-all-identifiers --binary-upgrade --format=custom  --file="pg_upgrade_dump_6064585.custom" "u" >> "pg_upgrade_dump_6064585.log" 2>&1
pg_dump: row number 0 is out of range 0..-1
Segmentation fault (core dumped)

...

Any tips on how to diagnose and/or what might be the problem?

What happens if you manually run the pg_dump command quoted above against a running 9.1 server, outside of the context of pg_upgrade?  (Your port will be probably be different from 50432)

If that still crashes, What if you drop the --binary-upgrade option?  The --format=custom option?

 

gdb doesn't seem helpful:

$ gdb /usr/lib/postgresql/9.3/bin/pg_dump core
GNU gdb (GDB) 7.5-ubuntu
Reading symbols from /usr/lib/postgresql/9.3/bin/pg_dump...(no debugging symbols found)...done.

You need to install the debugging symbols, which I have no experience doing under Ubuntu, but it should be easy to find instructions for.  Or you could compile pg_dump from source, after configuring with --enable-debug.

Cheers,

Jeff

Re: Segmentation fault: pg_upgrade 9.1 to 9.3: pg_dump: row number 0 is out of range 0..-1

От
Robert Nix
Дата:

What happens if you manually run the pg_dump command quoted above against a running 9.1 server, outside of the context of pg_upgrade?  (Your port will be probably be different from 50432)

If that still crashes, What if you drop the --binary-upgrade option?  The --format=custom option?
 
I started the 9.1 server normally and ran the above pg_dump (with the appropriate port for that instance).

| w/ -binary-upgrade | w/ --format=custom | crashed? |
|        yes         |         no         |   yes    |
|         no         |        yes         |    no    |
|        yes         |        yes         |   yes    |

So it seem the -binary-upgrade option has caused some problem. And the same crash happens regardless of whether i use the 9.1's or 9.3's pg_dump.
  

You need to install the debugging symbols, which I have no experience doing under Ubuntu, but it should be easy to find instructions for.  Or you could compile pg_dump from source, after configuring with --enable-debug.

I can handle the debug symbols build. I'll see what i can do if the pg_dump information doesn't lead anywhere useful.

Thanks for the help
 

Re: Segmentation fault: pg_upgrade 9.1 to 9.3: pg_dump: row number 0 is out of range 0..-1

От
Jeff Janes
Дата:
On Sun, Sep 15, 2013 at 5:06 PM, Robert Nix <robert@urban4m.com> wrote:

What happens if you manually run the pg_dump command quoted above against a running 9.1 server, outside of the context of pg_upgrade?  (Your port will be probably be different from 50432)

If that still crashes, What if you drop the --binary-upgrade option?  The --format=custom option?
 
I started the 9.1 server normally and ran the above pg_dump (with the appropriate port for that instance).

| w/ -binary-upgrade | w/ --format=custom | crashed? |
|        yes         |         no         |   yes    |
|         no         |        yes         |    no    |
|        yes         |        yes         |   yes    |

So it seem the -binary-upgrade option has caused some problem. And the same crash happens regardless of whether i use the 9.1's or 9.3's pg_dump.


If you do the dump using 9.1's pg_dump without --binary-upgrade, and then load that dump file into a new empty 9.1 server, then does it crash if you take a dump against *that* server?  If so, would you be allowed to post that dump file?
 
Cheers,

Jeff

Re: Segmentation fault: pg_upgrade 9.1 to 9.3: pg_dump: row number 0 is out of range 0..-1

От
Robert Nix
Дата:
If you do the dump using 9.1's pg_dump without --binary-upgrade, and then load that dump file into a new empty 9.1 server, then does it crash if you take a dump against *that* server?

I'll give it a try.

If so, would you be allowed to post that dump file?

I will not be able to provide the dump file but i will try to isolate the contents causing the issue and hopefully be able to provide a minimal dump file that doesn't contain any proprietary data yet still reproduces the crash. It's a fair amount of data, 45+G, so if i can reproduce it in a new database, isolating it may take some time.

Thanks

Re: Segmentation fault: pg_upgrade 9.1 to 9.3: pg_dump: row number 0 is out of range 0..-1

От
Jeff Janes
Дата:
On Sun, Sep 15, 2013 at 8:02 PM, Robert Nix <robert@urban4m.com> wrote:
If you do the dump using 9.1's pg_dump without --binary-upgrade, and then load that dump file into a new empty 9.1 server, then does it crash if you take a dump against *that* server?

I'll give it a try.

If so, would you be allowed to post that dump file?

I will not be able to provide the dump file but i will try to isolate the contents causing the issue and hopefully be able to provide a minimal dump file that doesn't contain any proprietary data yet still reproduces the crash. It's a fair amount of data, 45+G, so if i can reproduce it in a new database, isolating it may take some time.

The dump should just have the schema, not the data, because of --schema-only.
 

Re: Segmentation fault: pg_upgrade 9.1 to 9.3: pg_dump: row number 0 is out of range 0..-1

От
Bruce Momjian
Дата:
On Sat, Sep 14, 2013 at 09:40:01PM -0400, Robert Nix wrote:
> Running a pg_upgrade task is causing Segmentation fault:
>
> command: "/usr/lib/postgresql/9.3/bin/pg_dump" --host "/var/lib/postgresql"
> --port 50432 --username "postgres" --schema-only --quote-all-identifiers
> --binary-upgrade --format=custom  --file="pg_upgrade_dump_6064585.custom" "u"
> >> "pg_upgrade_dump_6064585.log" 2>&1
> pg_dump: row number 0 is out of range 0..-1
> Segmentation fault (core dumped)

This error is coming from libpq, specifically
check_tuple_field_number(), which is called by PQgetvalue(),
PQgetlength(), and PQgetisnull().  As I have never seen this error
before, there might be something wrong with the system catalogs on that
cluster.  It would be good to run pg_dump --binary-upgrade on that
database and try to identify the object being dumped that is causing the
crash.

--
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + Everyone has their own god. +


Re: Segmentation fault: pg_upgrade 9.1 to 9.3: pg_dump: row number 0 is out of range 0..-1

От
Alan Nilsson
Дата:
I ran into something tonight that seems relevant here, or certainly related:

I recently updated my app(s) libpq version from 9.1 to 9.3 and immediately I starting seeing:

row number 0 is out of range 0..-1

spewed to stdout.

I traced it down to this code:

    if (PQresultStatus(result) == PGRES_TUPLES_OK){
        if (!PQgetisnull(result, 0, 0)){
            ......
        }
    }

if the result contains 0 tuples, the message is spit to stdout.  This is new behavior, I went back and rebuilt using
libversions (according to PQlibVersion) 90102, 90203, and 90300.  Neither version 90102 & 90203 spewed to stdout. 

I don't know if this is causing the OP's crash, but it seems to me that there is something going on in libpq that needs
tobe looked at.  If nothing else than to get rid of the spewing to stdout. 

alan


On Oct 10, 2013, at 9:47 AM, Bruce Momjian <bruce@momjian.us> wrote:

> On Sat, Sep 14, 2013 at 09:40:01PM -0400, Robert Nix wrote:
>> Running a pg_upgrade task is causing Segmentation fault:
>>
>> command: "/usr/lib/postgresql/9.3/bin/pg_dump" --host "/var/lib/postgresql"
>> --port 50432 --username "postgres" --schema-only --quote-all-identifiers
>> --binary-upgrade --format=custom  --file="pg_upgrade_dump_6064585.custom" "u"
>>>> "pg_upgrade_dump_6064585.log" 2>&1
>> pg_dump: row number 0 is out of range 0..-1
>> Segmentation fault (core dumped)
>
> This error is coming from libpq, specifically
> check_tuple_field_number(), which is called by PQgetvalue(),
> PQgetlength(), and PQgetisnull().  As I have never seen this error
> before, there might be something wrong with the system catalogs on that
> cluster.  It would be good to run pg_dump --binary-upgrade on that
> database and try to identify the object being dumped that is causing the
> crash.
>
> --
>  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
>  EnterpriseDB                             http://enterprisedb.com
>
>  + Everyone has their own god. +
>
>
> --
> Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-general



Re: Segmentation fault: pg_upgrade 9.1 to 9.3: pg_dump: row number 0 is out of range 0..-1

От
Tom Lane
Дата:
Alan Nilsson <anilsson@apple.com> writes:
> I ran into something tonight that seems relevant here, or certainly related:
> I recently updated my app(s) libpq version from 9.1 to 9.3 and immediately I starting seeing:

> row number 0 is out of range 0..-1

> spewed to stdout.

> I traced it down to this code:

>     if (PQresultStatus(result) == PGRES_TUPLES_OK){
>         if (!PQgetisnull(result, 0, 0)){
>             ......
>         }
>     }

I can find no code like that in the PG sources --- everyplace that does
"PQgetisnull(result, 0, 0)" prefaces it with a check that there's at least
one row in the result.  I think you're blaming the messenger.

            regards, tom lane


Re: Segmentation fault: pg_upgrade 9.1 to 9.3: pg_dump: row number 0 is out of range 0..-1

От
Alan Nilsson
Дата:
oops sorry to be unclear, the code below is my code in my app.

What I am saying is that something changed in 90300 that causes libpq to spew to stdout where it had not in libpq 90102
&90203. 

I guess i am blaming the messenger because there should be no messenger.  Regardless of how badly I mangle the use of
libpq,it should not be sending anything to stdout. 

alan

On Oct 26, 2013, at 6:53 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

> Alan Nilsson <anilsson@apple.com> writes:
>> I ran into something tonight that seems relevant here, or certainly related:
>> I recently updated my app(s) libpq version from 9.1 to 9.3 and immediately I starting seeing:
>
>> row number 0 is out of range 0..-1
>
>> spewed to stdout.
>
>> I traced it down to this code:
>
>>     if (PQresultStatus(result) == PGRES_TUPLES_OK){
>>         if (!PQgetisnull(result, 0, 0)){
>>             ......
>>         }
>>     }
>
> I can find no code like that in the PG sources --- everyplace that does
> "PQgetisnull(result, 0, 0)" prefaces it with a check that there's at least
> one row in the result.  I think you're blaming the messenger.
>
>             regards, tom lane
>
>
> --
> Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-general



Re: Segmentation fault: pg_upgrade 9.1 to 9.3: pg_dump: row number 0 is out of range 0..-1

От
Tom Lane
Дата:
Alan Nilsson <anilsson@apple.com> writes:
> What I am saying is that something changed in 90300 that causes libpq to spew to stdout where it had not in libpq
90102& 90203. 

Well, that's interesting, but the issue is not in libpq, and you've still
provided no information that would help anyone diagnose where it is
(like, say, the query that's producing different results).

> I guess i am blaming the messenger because there should be no messenger.  Regardless of how badly I mangle the use of
libpq,it should not be sending anything to stdout. 

Would you prefer it dumped core?  It doesn't have very many choices for
reporting that you've passed invalid arguments.  In any case, the short
answer is that if you'd like this code fragment to do X when the query
has returned zero rows, you should explicitly code a test for that.
Otherwise you're at the mercy of somebody else's idea of what to do
with the error condition.

Also, if your gripe is not so much that it resorted to printing a message
as exactly how it printed it, see libpq's options for inserting your own
message processing hooks.

            regards, tom lane