Обсуждение: OSX doesn't accept identical source/target for strcpy() anymore
Hi, On -bugs it was reported that initdb of 9.3 failed with a assertion. On 2013-10-28 16:52:13 +0100, Matthias Schmitt wrote: > > In that case, could you enable coredumps and get a backtrace from that > > coredump? I unfortunately have zero clue about OSX, so I can't really > > help you with that. > > Yes, you are right. I totally forgot about the crash logs in OS X. Here it is: Ah, ok. I see the problem... > Process: postgres [68949] > Path: /Users/*/postgres > Identifier: postgres > Version: 0 > Code Type: X86-64 (Native) > Parent Process: sh [68948] > Responsible: Terminal [411] > User ID: 502 > > Date/Time: 2013-10-28 16:46:28.188 +0100 > OS Version: Mac OS X 10.9 (13A603) > Thread 0 Crashed:: Dispatch queue: com.apple.main-thread > 0 libsystem_kernel.dylib 0x00007fff93e00866 __pthread_kill + 10 > 1 libsystem_pthread.dylib 0x00007fff92c7335c pthread_kill + 92 > 2 libsystem_c.dylib 0x00007fff8c5a5bba abort + 125 > 3 libsystem_c.dylib 0x00007fff8c5a5d31 abort_report_np + 181 > 4 libsystem_c.dylib 0x00007fff8c5c98c5 __chk_fail + 48 > 5 libsystem_c.dylib 0x00007fff8c5c98d5 __chk_fail_overlap + 16 > 6 libsystem_c.dylib 0x00007fff8c5c9906 __chk_overlap + 49 > 7 libsystem_c.dylib 0x00007fff8c5c9a59 __strncpy_chk + 78 > 8 postgres 0x000000010b4c9045 namestrcpy + 86 > 9 postgres 0x000000010b1901f2 TupleDescInitEntry + 99 It seems the newest version of OSX is more strict about strcpy than previous ones. So the issue is likely the upgraded OS version. Which means we're going to see this more frequently now. There have been previous discussions about fixing strcpy calls with identical source/destination (same for memcpy) but it was deemed not worth the effort. I don't really see an alternative to fixing it now. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Andres Freund <andres@2ndquadrant.com> writes: > There have been previous discussions about fixing strcpy calls with > identical source/destination (same for memcpy) but it was deemed not > worth the effort. I don't really see an alternative to fixing it now. I'm not seeing this with bare-bones ./configure --enable-debug --enable-cassert on an OS X Mavericks machine with Xcode downloaded today. So there is something unidentified as yet about Matthias's configuration. regards, tom lane
On Mon, Oct 28, 2013 at 12:11 PM, Andres Freund <andres@2ndquadrant.com> wrote: > Hi, > > On -bugs it was reported that initdb of 9.3 failed with a > assertion. > > On 2013-10-28 16:52:13 +0100, Matthias Schmitt wrote: >> > In that case, could you enable coredumps and get a backtrace from that >> > coredump? I unfortunately have zero clue about OSX, so I can't really >> > help you with that. >> >> Yes, you are right. I totally forgot about the crash logs in OS X. Here it is: > > Ah, ok. I see the problem... > >> Process: postgres [68949] >> Path: /Users/*/postgres >> Identifier: postgres >> Version: 0 >> Code Type: X86-64 (Native) >> Parent Process: sh [68948] >> Responsible: Terminal [411] >> User ID: 502 >> >> Date/Time: 2013-10-28 16:46:28.188 +0100 >> OS Version: Mac OS X 10.9 (13A603) > >> Thread 0 Crashed:: Dispatch queue: com.apple.main-thread >> 0 libsystem_kernel.dylib 0x00007fff93e00866 __pthread_kill + 10 >> 1 libsystem_pthread.dylib 0x00007fff92c7335c pthread_kill + 92 >> 2 libsystem_c.dylib 0x00007fff8c5a5bba abort + 125 >> 3 libsystem_c.dylib 0x00007fff8c5a5d31 abort_report_np + 181 >> 4 libsystem_c.dylib 0x00007fff8c5c98c5 __chk_fail + 48 >> 5 libsystem_c.dylib 0x00007fff8c5c98d5 __chk_fail_overlap + 16 >> 6 libsystem_c.dylib 0x00007fff8c5c9906 __chk_overlap + 49 >> 7 libsystem_c.dylib 0x00007fff8c5c9a59 __strncpy_chk + 78 >> 8 postgres 0x000000010b4c9045 namestrcpy + 86 >> 9 postgres 0x000000010b1901f2 TupleDescInitEntry + 99 > > It seems the newest version of OSX is more strict about strcpy than > previous ones. So the issue is likely the upgraded OS version. Which > means we're going to see this more frequently now. > > There have been previous discussions about fixing strcpy calls with > identical source/destination (same for memcpy) but it was deemed not > worth the effort. I don't really see an alternative to fixing it now. Ugh. Why in the world would Apple break this? We could try to go through our code and fix this, but catching all of the instances is likely to be hard, and everyone else who writes software of any complexity is going to have the same darn problme. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Robert Haas <robertmhaas@gmail.com> writes: > On Mon, Oct 28, 2013 at 12:11 PM, Andres Freund <andres@2ndquadrant.com> wrote: >> There have been previous discussions about fixing strcpy calls with >> identical source/destination (same for memcpy) but it was deemed not >> worth the effort. I don't really see an alternative to fixing it now. > Ugh. Why in the world would Apple break this? It's broken already; the C and POSIX standards say of strncpy: If copying takes place between objects that overlap, the behavior is undefined. Both gcc and glibc have been moving steadily in the direction of aggressively exploiting "undefined behavior" cases for optimization purposes. I don't know if there is yet a platform where strncpy with src == dest behaves oddly, but we'd be foolish to imagine that it's not going to happen eventually. If anything, Apple is probably doing us a service by making it obvious where we're failing to adhere to spec. However ... I still can't replicate this here, and as you say, there's about zero chance of keeping our code clean of this problem unless we can set up a buildfarm member that will catch it. regards, tom lane
On 2013-10-28 14:11:12 -0400, Tom Lane wrote: > Robert Haas <robertmhaas@gmail.com> writes: > > On Mon, Oct 28, 2013 at 12:11 PM, Andres Freund <andres@2ndquadrant.com> wrote: > >> There have been previous discussions about fixing strcpy calls with > >> identical source/destination (same for memcpy) but it was deemed not > >> worth the effort. I don't really see an alternative to fixing it now. > > > Ugh. Why in the world would Apple break this? > > It's broken already; the C and POSIX standards say of strncpy: > > If copying takes place between objects that overlap, the behavior is undefined. > > Both gcc and glibc have been moving steadily in the direction of > aggressively exploiting "undefined behavior" cases for optimization > purposes. I don't know if there is yet a platform where strncpy with > src == dest behaves oddly, but we'd be foolish to imagine that it's > not going to happen eventually. If anything, Apple is probably doing > us a service by making it obvious where we're failing to adhere to spec. > > However ... I still can't replicate this here, and as you say, there's > about zero chance of keeping our code clean of this problem unless we > can set up a buildfarm member that will catch it. It'd be neat if we could get a buildfarm animal up that uses valgrind - which would catch such and lots of other errors. That's where the topic has come up in the past: http://www.postgresql.org/message-id/20110312133224.GA7833%40tornado.gateway.2wire.net http://www.postgresql.org/message-id/20130217142209.GA5073@awork2.anarazel.de Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Hi,
On Mon, Oct 28, 2013 at 7:11 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
I haven't a 10.9 box for double checking but there's a gcc command line triggering the same assert for strcpy and gcc at http://lists.gnu.org/archive/html/bug-bash/2013-07/msg00011.html.
Didier
If copying takes place between objects that overlap, the behavior is undefined.
Both gcc and glibc have been moving steadily in the direction of
aggressively exploiting "undefined behavior" cases for optimization
purposes. I don't know if there is yet a platform where strncpy with
src == dest behaves oddly, but we'd be foolish to imagine that it's
not going to happen eventually. If anything, Apple is probably doing
us a service by making it obvious where we're failing to adhere to spec.
However ... I still can't replicate this here, and as you say, there's
about zero chance of keeping our code clean of this problem unless we
can set up a buildfarm member that will catch it.
regards, tom lane
On 10/28/2013 02:26 PM, Andres Freund wrote: > > It'd be neat if we could get a buildfarm animal up that uses valgrind - > which would catch such and lots of other errors. That's where the topic > has come up in the past: > http://www.postgresql.org/message-id/20110312133224.GA7833%40tornado.gateway.2wire.net > http://www.postgresql.org/message-id/20130217142209.GA5073@awork2.anarazel.de > How exactly is it going to do that? Fundamentally, the buildfarm client is simply glue to run existing build and test code, collect the results, and send them to the server. AFAICT there are no configure or make targets for running under valgrind. If someone provides the requisite support in the build system for this I'll be happy to add buildfarm support for it. cheers andrew
On 2013-10-28 15:20:20 -0400, Andrew Dunstan wrote: > > On 10/28/2013 02:26 PM, Andres Freund wrote: > > > >It'd be neat if we could get a buildfarm animal up that uses valgrind - > >which would catch such and lots of other errors. That's where the topic > >has come up in the past: > >http://www.postgresql.org/message-id/20110312133224.GA7833%40tornado.gateway.2wire.net > >http://www.postgresql.org/message-id/20130217142209.GA5073@awork2.anarazel.de > > > > > How exactly is it going to do that? The easiest method - somewhat slower than necessary - is to just run "valgrind --suppressions=$srcdir/src/tools/valgrind.supp make check". But the buildfarm supports running a postgres install before installcheck, right? If we could run just that step using valgrind we'd be very well of I think because we'd not run valgrind (slow!) if there are plain regression failures around. [.. looking for sources ...] start_db in https://github.com/PGBuildFarm/client-code/blob/master/run_build.pl is where the server's run, right? Hm. That uses pg_ctl and not the server itself and relies on pg_ctl -w returning when the server is started... So it's not easy to make it use valgrind properly. We could hack it by replacing bin/postgres with a wrapper around valgrind that invokes postgres, but ick. Maybe we can make pg_ctl start $PG_POSTGRES_BINARY instead of postgres if defined? Better ideas? > Fundamentally, the buildfarm client is simply glue to run existing build and > test code, collect the results, and send them to the server. AFAICT there > are no configure or make targets for running under valgrind. > If someone provides the requisite support in the build system for this I'll > be happy to add buildfarm support for it. It'd be relatively easy to add support for make check (not installcheck) wrapping postgres in valgrind via pg_regress, but I am not sure that's the best way to go. I think defining an additional CFLAG (USE_VALGRIND) shouldn't be a problem? Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Andres Freund <andres@2ndquadrant.com> writes: > It'd be relatively easy to add support for make check (not installcheck) > wrapping postgres in valgrind via pg_regress, but I am not sure that's > the best way to go. > I think defining an additional CFLAG (USE_VALGRIND) shouldn't be a > problem? CFLAGS doesn't seem to have anything to do with this. I'd be more inclined to add a "--wrapper=prog" argument to pg_regress and invoke it with something like --wrapper="valgrind --trace-children=yes". The larger problem though is what you'd do with the output. There's enough false-positive noise from valgrind that I can't see having the buildfarm run just fail if there are any messages. What to do instead isn't very clear. Anyway, to get back to the original problem, I've confirmed that valgrind complains about the particular case at hand: ==9497== Source and destination overlap in strncpy(0xe1d5a3d, 0xe1d5a3d, 64) ==9497== at 0x4A081EF: strncpy (mc_replace_strmem.c:476) ==9497== by 0x6D2398: namestrcpy (name.c:221) ==9497== by 0x45F478: TupleDescInitEntry (tupdesc.c:507) ==9497== by 0x756FE1: internal_get_result_type (funcapi.c:557) ==9497== by 0x75727C: get_expr_result_type (funcapi.c:235) ==9497== by 0x534146: expandRecordVariable (parse_target.c:1524) ==9497== by 0x52C267: ParseFuncOrColumn (parse_func.c:1494) ==9497== by 0x5285CF: transformExprRecurse (parse_expr.c:463) ==9497== by 0x528DB1: transformExpr (parse_expr.c:117) ==9497== by 0x5350C5: transformTargetEntry (parse_target.c:94) ==9497== by 0x535AD4: transformTargetList (parse_target.c:167) ==9497== by 0x505FFF: transformStmt (analyze.c:929) It seems to me the most reasonable fix for this is to make TupleDescInitEntry notice that the passed "attributeName" points at the tupdesc's name field and not call namestrcpy if so. This would go with an API clarification stating that callers can pass that if they want the name field to be unchanged. (Or we could invent some other way to signal that, but note that NULL is already in use for a different purpose.) Another possibly-useful approach would be to hack namestrcpy itself to handle name == str specially. However, that's legitimizing a usage that's really a type cheat, so I don't like it as much, even though it might fix more cases besides this one. Any other thoughts about it? regards, tom lane
On Mon, Oct 28, 2013 at 6:11 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Both gcc and glibc have been moving steadily in the direction of > aggressively exploiting "undefined behavior" cases for optimization > purposes. I don't know if there is yet a platform where strncpy with > src == dest behaves oddly, but we'd be foolish to imagine that it's > not going to happen eventually. If anything, Apple is probably doing > us a service by making it obvious where we're failing to adhere to spec. It's worth being aware of the fact that the upcoming GCC 4.9 release is expected to ship with an "Undefined Behavior Sanitizer", as described here: http://gcc.gnu.org/gcc-4.9/changes.html -- Peter Geoghegan
On 2013-10-28 16:02:36 -0400, Tom Lane wrote: > Andres Freund <andres@2ndquadrant.com> writes: > > It'd be relatively easy to add support for make check (not installcheck) > > wrapping postgres in valgrind via pg_regress, but I am not sure that's > > the best way to go. > > > I think defining an additional CFLAG (USE_VALGRIND) shouldn't be a > > problem? > > CFLAGS doesn't seem to have anything to do with this. I'd be more > inclined to add a "--wrapper=prog" argument to pg_regress and invoke > it with something like --wrapper="valgrind --trace-children=yes". Err. I am *obviously* not saying that it makes the backend run under valgrind. But if we're going to have a buildfarm animal running valgrind, it'd be useful to run to let it catch all errors it can instead of hiding many of them via mcxt/aset.c? Right? > The larger problem though is what you'd do with the output. There's > enough false-positive noise from valgrind that I can't see having > the buildfarm run just fail if there are any messages. What to do > instead isn't very clear. The false positives should be gone using the suppressions file we ship these days (--suppressions=/path/to/pg/src/tools/valgrind.supp). We might miss some more cases there, but it should be fairly easy to extend it. > It seems to me the most reasonable fix for this is to make > TupleDescInitEntry notice that the passed "attributeName" points > at the tupdesc's name field and not call namestrcpy if so. > This would go with an API clarification stating that callers can > pass that if they want the name field to be unchanged. +1 > Another possibly-useful approach would be to hack namestrcpy itself > to handle name == str specially. However, that's legitimizing a > usage that's really a type cheat, so I don't like it as much, even > though it might fix more cases besides this one. -1 Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
On 10/28/13, 4:11 PM, Peter Geoghegan wrote: > On Mon, Oct 28, 2013 at 6:11 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: >> Both gcc and glibc have been moving steadily in the direction of >> aggressively exploiting "undefined behavior" cases for optimization >> purposes. I don't know if there is yet a platform where strncpy with >> src == dest behaves oddly, but we'd be foolish to imagine that it's >> not going to happen eventually. If anything, Apple is probably doing >> us a service by making it obvious where we're failing to adhere to spec. > > It's worth being aware of the fact that the upcoming GCC 4.9 release > is expected to ship with an "Undefined Behavior Sanitizer", as > described here: > > http://gcc.gnu.org/gcc-4.9/changes.html Address Sanitizer, which is already in clang and gcc, does catch the case were are talking about (as was previously discussed). But its only reaction is crashing, and in my testing it's crashing already before it gets to this place, so we'd have to put in some more work before it will be useful.
Andres Freund <andres@2ndquadrant.com> writes: > On 2013-10-28 16:02:36 -0400, Tom Lane wrote: >> The larger problem though is what you'd do with the output. There's >> enough false-positive noise from valgrind that I can't see having >> the buildfarm run just fail if there are any messages. What to do >> instead isn't very clear. > The false positives should be gone using the suppressions file we ship > these days (--suppressions=/path/to/pg/src/tools/valgrind.supp). We > might miss some more cases there, but it should be fairly easy to extend > it. They're not all gone according to my testing; but there are far worse problems: 1. The output goes to stderr which means it's mixed in with the backend's normal log chatter. 2. valgrind causes autovacuum to dump core, at least on my box (RHEL6). I'm prepared to believe that this is some relatively old bug that Red Hat hasn't gotten round to including a patch for, but still it doesn't leave me with any warm fuzzy feeling about the practicality of routine valgrind testing. regards, tom lane
On 2013-10-28 21:14:48 -0400, Tom Lane wrote: > Andres Freund <andres@2ndquadrant.com> writes: > > On 2013-10-28 16:02:36 -0400, Tom Lane wrote: > >> The larger problem though is what you'd do with the output. There's > >> enough false-positive noise from valgrind that I can't see having > >> the buildfarm run just fail if there are any messages. What to do > >> instead isn't very clear. > > > The false positives should be gone using the suppressions file we ship > > these days (--suppressions=/path/to/pg/src/tools/valgrind.supp). We > > might miss some more cases there, but it should be fairly easy to extend > > it. > > They're not all gone according to my testing; but there are far worse > problems: Spurious or real bugs? Inside PG or libc? > 1. The output goes to stderr which means it's mixed in with the backend's > normal log chatter. That's relatively easy to fix. We could just pass --log-file redirecting the errors somewhere special and display them there. What I've done so far is to tell valgrind to let child processes with errors exit with a non-zero exitcode using the --error-exitcode parameter and specify -q to reduce the chatter upon normal process exit. That gives at least some correlation to the errors in the failed tests. > 2. valgrind causes autovacuum to dump core, at least on my box (RHEL6). > I'm prepared to believe that this is some relatively old bug that Red Hat > hasn't gotten round to including a patch for, but still it doesn't leave > me with any warm fuzzy feeling about the practicality of routine valgrind > testing. Yea, I know which bug that is, I've pushed the valgrind guys into fixing it... valgrind used to get confused about stack alignment in signal handlers causing instructions that care about that (mostly xmm* register using ones) to fail. elog() fails because we frequently pass many parameters. Since we fork processes from inside signal handlers, that situation happens pretty often. https://bugs.kde.org/show_bug.cgi?id=280114 3. valgrind gets floating point computations for exp(larger_negative_double) wrong and returns the wrong error message: regression=# SELECT exp(-808.3::float8); ERROR: value out of range: overflow exp sets errno=ERANGE and returns inf. That's not supposed to happen according to my exp(3)... Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Andres Freund <andres@2ndquadrant.com> writes: > On 2013-10-28 21:14:48 -0400, Tom Lane wrote: >> They're not all gone according to my testing; but there are far worse >> problems: > Spurious or real bugs? Inside PG or libc? I saw a bunch of uninitialized-value complaints in initdb, apparently from places in BootstrapXLog that write out uninitialized pad bytes. I didn't get far in testing the main regression tests because of the autovacuum crash problem. > 3. valgrind gets floating point computations for > exp(larger_negative_double) wrong and returns the wrong error message: > regression=# SELECT exp(-808.3::float8); > ERROR: value out of range: overflow Ugh ... regards, tom lane
On Mon, Oct 28, 2013 at 09:14:48PM -0400, Tom Lane wrote: > Andres Freund <andres@2ndquadrant.com> writes: > > On 2013-10-28 16:02:36 -0400, Tom Lane wrote: > >> The larger problem though is what you'd do with the output. There's > >> enough false-positive noise from valgrind that I can't see having > >> the buildfarm run just fail if there are any messages. What to do > >> instead isn't very clear. > > > The false positives should be gone using the suppressions file we ship > > these days (--suppressions=/path/to/pg/src/tools/valgrind.supp). We > > might miss some more cases there, but it should be fairly easy to extend > > it. > > They're not all gone according to my testing True. Getting a clean Valgrind report is similar to getting the build warning-free. Variations of compiler version, optimization level, host OS, and CPU architecture can all affect the set of errors Valgrind reports, just as they affect compiler warnings. Valgrind cleanliness has a lot of catching up to do. I never ran initdb under Valgrind, just "make installcheck", so that's novel territory. > 1. The output goes to stderr which means it's mixed in with the backend's > normal log chatter. As Andres explained, this is strictly a local configuration choice. > 2. valgrind causes autovacuum to dump core, at least on my box (RHEL6). Don't bother with versions older than Valgrind 3.8.1. Besides having a fix for that bug, it runs PostgreSQL an order of magnitude faster, per the comment in pg_config_manual.h. nm -- Noah Misch EnterpriseDB http://www.enterprisedb.com
On Mon, Oct 28, 2013 at 04:02:36PM -0400, Tom Lane wrote: > It seems to me the most reasonable fix for this is to make > TupleDescInitEntry notice that the passed "attributeName" points > at the tupdesc's name field and not call namestrcpy if so. +1 -- Noah Misch EnterpriseDB http://www.enterprisedb.com
Noah Misch <noah@leadboat.com> writes: > On Mon, Oct 28, 2013 at 09:14:48PM -0400, Tom Lane wrote: >> 2. valgrind causes autovacuum to dump core, at least on my box (RHEL6). > Don't bother with versions older than Valgrind 3.8.1. $ rpm -qa | grep valgrind valgrind-3.8.1-3.2.el6.x86_64 regards, tom lane
On 2013-10-28 22:20:02 -0400, Noah Misch wrote: > > 2. valgrind causes autovacuum to dump core, at least on my box (RHEL6). > > Don't bother with versions older than Valgrind 3.8.1. Besides having a fix > for that bug, it runs PostgreSQL an order of magnitude faster, per the comment > in pg_config_manual.h. I don't think that bugfix is in upstream's 3.8.1 given that was released months earlier than the bugfix was committed... Might be backported in your package though... Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
On Mon, Oct 28, 2013 at 10:30:10PM -0400, Tom Lane wrote: > Noah Misch <noah@leadboat.com> writes: > > On Mon, Oct 28, 2013 at 09:14:48PM -0400, Tom Lane wrote: > >> 2. valgrind causes autovacuum to dump core, at least on my box (RHEL6). > > > Don't bother with versions older than Valgrind 3.8.1. > > $ rpm -qa | grep valgrind > valgrind-3.8.1-3.2.el6.x86_64 Apparently I only dreamt the bug was gone in Valgrind 3.8.1; my usual testing configuration has autovacuum=off. Sorry for the noise. -- Noah Misch EnterpriseDB http://www.enterprisedb.com
Andres Freund <andres@2ndquadrant.com> writes: > On 2013-10-28 21:14:48 -0400, Tom Lane wrote: >> 2. valgrind causes autovacuum to dump core, at least on my box (RHEL6). > Yea, I know which bug that is, I've pushed the valgrind guys into fixing > it... > https://bugs.kde.org/show_bug.cgi?id=280114 Thanks, I whined to Red Hat at https://bugzilla.redhat.com/show_bug.cgi?id=1024162 and also updated our wiki page about Valgrind (where somebody overhastily removed all mention of the issue ...) regards, tom lane
On 2013-10-29 02:29:03 +0100, Andres Freund wrote: > 3. valgrind gets floating point computations for > exp(larger_negative_double) wrong and returns the wrong error message: > > regression=# SELECT exp(-808.3::float8); > ERROR: value out of range: overflow > > exp sets errno=ERANGE and returns inf. That's not supposed to happen > according to my exp(3)... I've reported this as a valgrind bug... https://bugs.kde.org/show_bug.cgi?id=326821 Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services