Обсуждение: 7.5 beta version

I am having some trouble interfacing the 7.5 server built with MINGW
with tools generated using other compilers.

I suspect that the issue is one of default structure packing.  In the
old version we were using, we built PostgreSQL using Intel C++ or MS
VC++ and the same for the libpq and other interfaces.

Now that I have built the server and associated tools with Mingw and a
version of the interface system with MS VC++ and Intel C++, I have
problems using (for instance) libpq.

Some character strings returned early in the struct are fine, but the
numeric fields occurring later are garbled.

Now, this issue seems like it must have happened before, since the
server might be generated on some 64 bit platform and the interface
library on some other platform.

In particular, this function:

char *PQgetvalue(const PGresult *res, int tup_num, int field_num)
{if (!check_tuple_field_number(res, tup_num, field_num))    return NULL;return res->tuples[tup_num][field_num].value;
}

Seems to return nonsense when I get to pointers to non-character data.

Any thoughts?

I don't think you can mix libs/binaries from different compilers.

---------------------------------------------------------------------------

Dann Corbit wrote:
> I am having some trouble interfacing the 7.5 server built with MINGW
> with tools generated using other compilers.
> 
> I suspect that the issue is one of default structure packing.  In the
> old version we were using, we built PostgreSQL using Intel C++ or MS
> VC++ and the same for the libpq and other interfaces.
> 
> Now that I have built the server and associated tools with Mingw and a
> version of the interface system with MS VC++ and Intel C++, I have
> problems using (for instance) libpq.
> 
> Some character strings returned early in the struct are fine, but the
> numeric fields occurring later are garbled.
> 
> Now, this issue seems like it must have happened before, since the
> server might be generated on some 64 bit platform and the interface
> library on some other platform.
> 
> In particular, this function:
> 
> char *PQgetvalue(const PGresult *res, int tup_num, int field_num)
> {
>     if (!check_tuple_field_number(res, tup_num, field_num))
>         return NULL;
>     return res->tuples[tup_num][field_num].value;
> }
> 
> Seems to return nonsense when I get to pointers to non-character data.
> 
> Any thoughts?
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 6: Have you searched our list archives?
> 
>                http://archives.postgresql.org
> 

--  Bruce Momjian                        |  http://candle.pha.pa.us pgman@candle.pha.pa.us               |  (610)
359-1001+  If your life is a hard drive,     |  13 Roberts Road +  Christ can be your backup.        |  Newtown Square,
Pennsylvania19073
 

Hi!

1) Is this depending on the server, or the fact that there is a
different libpq.dll in the path? Does a non-win32 client work against
the win32 server, and vice versa?

2) Are you using the import library from mingw, or the one from the old
visual C++ compile?  If you are using the import library from the old
Visual compile, what happens if you change to the mingw one?


//Magnus


> -----Original Message-----
> From: Dann Corbit [mailto:DCorbit@connx.com]
> Sent: Tuesday, April 06, 2004 2:43 AM
> To: pgsql-hackers@postgresql.org
> Subject: [HACKERS] 7.5 beta version
>
>
> I am having some trouble interfacing the 7.5 server built
> with MINGW with tools generated using other compilers.
>
> I suspect that the issue is one of default structure packing.
>  In the old version we were using, we built PostgreSQL using
> Intel C++ or MS
> VC++ and the same for the libpq and other interfaces.
>
> Now that I have built the server and associated tools with
> Mingw and a version of the interface system with MS VC++ and
> Intel C++, I have problems using (for instance) libpq.
>
> Some character strings returned early in the struct are fine,
> but the numeric fields occurring later are garbled.
>
> Now, this issue seems like it must have happened before,
> since the server might be generated on some 64 bit platform
> and the interface library on some other platform.
>
> In particular, this function:
>
> char *PQgetvalue(const PGresult *res, int tup_num, int field_num) {
>     if (!check_tuple_field_number(res, tup_num, field_num))
>         return NULL;
>     return res->tuples[tup_num][field_num].value;
> }
>
> Seems to return nonsense when I get to pointers to non-character data.
>
> Any thoughts?
>
> ---------------------------(end of
> broadcast)---------------------------
> TIP 6: Have you searched our list archives?
>
>                http://archives.postgresql.org
>

On Mon, Apr 05, 2004 at 09:38:13PM -0400, Bruce Momjian wrote:
> 
> I don't think you can mix libs/binaries from different compilers.

As long as it's plain old C, and the compilers adhere to the platform's
ABI standards, why not?  Even if you compile the C code using a C++
compiler, as in this case, any C structs will be PODs and so should be
compiled according to the C layout rules.


Jeroen

Jeroen T. Vermeulen wrote:
> On Mon, Apr 05, 2004 at 09:38:13PM -0400, Bruce Momjian wrote:
> > 
> > I don't think you can mix libs/binaries from different compilers.
> 
> As long as it's plain old C, and the compilers adhere to the platform's
> ABI standards, why not?  Even if you compile the C code using a C++
> compiler, as in this case, any C structs will be PODs and so should be
> compiled according to the C layout rules.

I was not sure if Win32 had standard alignment for C.

--  Bruce Momjian                        |  http://candle.pha.pa.us pgman@candle.pha.pa.us               |  (610)
359-1001+  If your life is a hard drive,     |  13 Roberts Road +  Christ can be your backup.        |  Newtown Square,
Pennsylvania19073
 

On Sun, Apr 11, 2004 at 10:21:30PM -0400, Bruce Momjian wrote:
> I was not sure if Win32 had standard alignment for C.

Good point.  There's standards, and then there's Windows.  It's possible
that separate "tight-packing" and "regular" pragmas are used there, just
for structs that are expected to be used across linkage boundaries.


Jeroen

"Jeroen T. Vermeulen" <jtv@xs4all.nl> writes:
> On Sun, Apr 11, 2004 at 10:21:30PM -0400, Bruce Momjian wrote:
>> I was not sure if Win32 had standard alignment for C.

> Good point.  There's standards, and then there's Windows.  It's possible
> that separate "tight-packing" and "regular" pragmas are used there, just
> for structs that are expected to be used across linkage boundaries.

"#pragma pack" is one of the worst, most dangerous inventions in the
history of software development :-(.  However, I don't think we should
be at any great risk from it, since we don't use it in our own code.
The case that's really hazardous is where you have separately compiled
libraries interfacing through nontrivial struct declarations...  I still
have the scars from making libjpeg do things that way...
        regards, tom lane

> -----Original Message-----
> From: Jeroen T. Vermeulen [mailto:jtv@xs4all.nl]
> Sent: Sunday, April 11, 2004 7:28 AM
> To: Bruce Momjian
> Cc: Dann Corbit; pgsql-hackers@postgresql.org
> Subject: Re: [HACKERS] 7.5 beta version
>
>
> On Mon, Apr 05, 2004 at 09:38:13PM -0400, Bruce Momjian wrote:
> >
> > I don't think you can mix libs/binaries from different compilers.
>
> As long as it's plain old C, and the compilers adhere to the
> platform's ABI standards, why not?  Even if you compile the C
> code using a C++ compiler, as in this case, any C structs
> will be PODs and so should be compiled according to the C
> layout rules.

1.
The C language does not define alignment of structs.

The C language does not specify that an integer shall be the same size
on two different platforms.

The C language does not specify a portable way even to read and write
structs to disk and preserve alignment across platforms.

The C language does not specify that IEEE arithmetic must be used or
even that a double shall be able to represent a value larger than a
float.

2.
It is a mistake to use a C++ compiler to compile C code, because the
languages are different.  Stroustrup's statement that good C code tends
to be valid C++ code is not correct.

Mingw GCC is both a C and a C++ compiler.  Fortunately, the C compiler
is used, because there is no way that the code base would compile as C
code.  A trivial example to show why not is the frequent use of the C++
keyword 'new' in the code base.  Here is an example:
regc_color.c (225):  struct colordesc *new;
regc_color.c (251):  new = (struct colordesc *) MALLOC(n *
regc_color.c (253):  if (new != NULL)
regc_color.c (254):  memcpy(VS(new), VS(cm->cdspace), cm->ncds *
regc_color.c (258):  new = (struct colordesc *) REALLOC(cm->cd,
regc_color.c (260):  if (new == NULL)
regc_color.c (265):  cm->cd = new;

On Mon, Apr 12, 2004 at 11:55:45AM -0700, Dann Corbit wrote:
> 1.
> The C language does not define alignment of structs.
Platform ABI standards do, though (hence the "as long as it adheres to..."
clause in my previous post).  Whether it's in the C language or in the
platform's ABI standards is important when writing portable code; it should
not matter when compiling an existing product--provided that the platform
defines the ABI unambiguously.  The latter is what's in question here.


> The C language does not specify that an integer shall be the same size
> on two different platforms.

That is not a problem when linking across compilers, only when linking
across ABIs.  I would expect that doing so would fail at link time.
Besides, as a practical matter, even the 64-bit platforms current today
normally specify that int is a 32-bit type so this should not come into
play in this case, even when linking e.g. Itanium code to x86 code.


> The C language does not specify that IEEE arithmetic must be used or
> even that a double shall be able to represent a value larger than a
> float.
Again, I doubt this is relevant in this case where the problem is linking
across compilers on the same platform, not across platforms.


> Mingw GCC is both a C and a C++ compiler.  Fortunately, the C compiler
> is used, because there is no way that the code base would compile as C
> code.  A trivial example to show why not is the frequent use of the C++
> keyword 'new' in the code base.  Here is an example:

With gcc at least, the matter is even simpler than that.  It selects an
appropriate front-end (C, C++, ...) based on the names of the source files
being compiled.  This may break down at link time however; g++ links to
the C++ standard library, which gcc by default doesn't.


Jeroen

> -----Original Message-----
> From: Jeroen T. Vermeulen [mailto:jtv@xs4all.nl]
> Sent: Monday, April 12, 2004 12:25 PM
> To: Dann Corbit
> Cc: Bruce Momjian; pgsql-hackers@postgresql.org
> Subject: Re: [HACKERS] 7.5 beta version
>
>
> On Mon, Apr 12, 2004 at 11:55:45AM -0700, Dann Corbit wrote:
>
> > 1.
> > The C language does not define alignment of structs.
>
> Platform ABI standards do, though (hence the "as long as it
> adheres to..." clause in my previous post).  Whether it's in
> the C language or in the platform's ABI standards is
> important when writing portable code; it should not matter
> when compiling an existing product--provided that the
> platform defines the ABI unambiguously.  The latter is what's
> in question here.

Where is the definition for a platform ABI?  (Sounds like Application
Binary Interface).
At any rate, there is nothing to force any C compiler to adhere to it.

> > The C language does not specify that an integer shall be
> the same size
> > on two different platforms.
>
> That is not a problem when linking across compilers, only
> when linking across ABIs.  I would expect that doing so would
> fail at link time. Besides, as a practical matter, even the
> 64-bit platforms current today normally specify that int is a
> 32-bit type so this should not come into play in this case,
> even when linking e.g. Itanium code to x86 code.
>
>
> > The C language does not specify that IEEE arithmetic must
> be used or
> > even that a double shall be able to represent a value larger than a
> > float.
>
> Again, I doubt this is relevant in this case where the
> problem is linking across compilers on the same platform, not
> across platforms.

I do know of important differences in compilers in this regard.  You can
(for instance) have 80 bit floating point on one compiler using double
but it is only 64 bits on another.

The point being that there is no such thing as a binary interface for
alignments or data types that is defined by the C or C++ ANSI/ISO
language standards.  If there is another standard, I would like to hear
about it.


> > Mingw GCC is both a C and a C++ compiler.  Fortunately, the
> C compiler
> > is used, because there is no way that the code base would
> compile as C
> > code.  A trivial example to show why not is the frequent use of the
> > C++ keyword 'new' in the code base.  Here is an example:
>
> With gcc at least, the matter is even simpler than that.  It
> selects an appropriate front-end (C, C++, ...) based on the
> names of the source files being compiled.  This may break
> down at link time however; g++ links to the C++ standard
> library, which gcc by default doesn't.

Most compilers do something like that.

Here is my puzzlement...
If I compile a PostgreSQL database on some 64 bit machine, I should be
able to access it from a 32 bit machine.  For instance, I can access
DB/2 on our 3090 or Rdb on our Alpha from a 32 bit workstation and I
have no problems of this nature.  Surely it is an issue with PostgreSQL
that has been recognized before.

If I change compilers or if I even completely change architectures it
should not matter.  The interface to the database should be architecture
independent.  Said another way:
I should have no concerns about what sort of architecture the server is
on or what compiler was used.

On Mon, Apr 12, 2004 at 12:35:15PM -0700, Dann Corbit wrote:
> I do know of important differences in compilers in this regard.  You can
> (for instance) have 80 bit floating point on one compiler using double
> but it is only 64 bits on another.
But in the case of x86 (among others) that's the in-register
representation, no?  IIRC they are stored to memory as 64-bit doubles at
best.


> The point being that there is no such thing as a binary interface for
> alignments or data types that is defined by the C or C++ ANSI/ISO
> language standards.  If there is another standard, I would like to hear
> about it.
That depends on the platform vendor.  Which depending on the platform may
actually be whoever specified the CPU architecture and/or whoever supplied
the OS.  As you say, compilers may deviate from it although in many cases
it would render them useless.

In this particular case, it would most likely be Microsoft as the dominant
{OS,compiler,everything else} vendor.  I *think* (but I'm not sure) that
Microsoft set an ABI even at the C++ level (as Intel also did with the
Itanium, BTW), although it's more common to specify the C level only.

In C++, ABI compatibility is normally protected through a side effect of
name mangling.  By maintaining different name mangling schemes for
different ABI conventions, compiler vendors ensure that object files will
refuse to link to other object files that adhere to different ABIs.

> Here is my puzzlement...
> If I compile a PostgreSQL database on some 64 bit machine, I should be
> able to access it from a 32 bit machine.  For instance, I can access
> DB/2 on our 3090 or Rdb on our Alpha from a 32 bit workstation and I
> have no problems of this nature.  Surely it is an issue with PostgreSQL
> that has been recognized before.
I would say yes, definitely!  That part is not in question here, only the
linking-across-compilers part.  But see below.


> If I change compilers or if I even completely change architectures it
> should not matter.  The interface to the database should be architecture
> independent.  Said another way:
> I should have no concerns about what sort of architecture the server is
> on or what compiler was used.
Unless you use the binary mode of data transfer perhaps; I think that's been
rationalized in 7.4 and is now portable.  No idea what happens if you
convert tables written in an older version (say, 7.3) to 7.5 and then
read them from a wildly different platform than you wrote them from, but
that may be a bit far-fetched.

Jeroen

> -----Original Message-----
> From: Jeroen T. Vermeulen [mailto:jtv@xs4all.nl]
> Sent: Monday, April 12, 2004 1:00 PM
> To: Dann Corbit
> Cc: Bruce Momjian; pgsql-hackers@postgresql.org
> Subject: Re: [HACKERS] 7.5 beta version
>
>
> On Mon, Apr 12, 2004 at 12:35:15PM -0700, Dann Corbit wrote:
>
> > I do know of important differences in compilers in this
> regard.  You
> > can (for instance) have 80 bit floating point on one compiler using
> > double but it is only 64 bits on another.
>
> But in the case of x86 (among others) that's the in-register
> representation, no?  IIRC they are stored to memory as 64-bit
> doubles at best.

Depends on the compiler.  The Intel C++ compiler can compile 80 bit
doubles.

> > The point being that there is no such thing as a binary
> interface for
> > alignments or data types that is defined by the C or C++ ANSI/ISO
> > language standards.  If there is another standard, I would like to
> > hear about it.
>
> That depends on the platform vendor.  Which depending on the
> platform may actually be whoever specified the CPU
> architecture and/or whoever supplied the OS.  As you say,
> compilers may deviate from it although in many cases it would
> render them useless.
>
> In this particular case, it would most likely be Microsoft as
> the dominant {OS,compiler,everything else} vendor.  I *think*
> (but I'm not sure) that Microsoft set an ABI even at the C++
> level (as Intel also did with the Itanium, BTW), although
> it's more common to specify the C level only.
>
> In C++, ABI compatibility is normally protected through a
> side effect of name mangling.  By maintaining different name
> mangling schemes for different ABI conventions, compiler
> vendors ensure that object files will refuse to link to other
> object files that adhere to different ABIs.

As far as I know, name mangling only pertains to C++ compilers.  And the
PostgreSQL project is entirely written in C.
> > Here is my puzzlement...
> > If I compile a PostgreSQL database on some 64 bit machine,
> I should be
> > able to access it from a 32 bit machine.  For instance, I
> can access
> > DB/2 on our 3090 or Rdb on our Alpha from a 32 bit
> workstation and I
> > have no problems of this nature.  Surely it is an issue with
> > PostgreSQL that has been recognized before.
>
> I would say yes, definitely!  That part is not in question
> here, only the linking-across-compilers part.  But see below.
>
>
> > If I change compilers or if I even completely change
> architectures it
> > should not matter.  The interface to the database should be
> > architecture independent.  Said another way: I should have
> no concerns
> > about what sort of architecture the server is on or what
> compiler was
> > used.
>
> Unless you use the binary mode of data transfer perhaps; I
> think that's been rationalized in 7.4 and is now portable.
> No idea what happens if you convert tables written in an
> older version (say, 7.3) to 7.5 and then read them from a
> wildly different platform than you wrote them from, but that
> may be a bit far-fetched.

Actually, I do want to use binary mode if possible.  The data stream
will be more compact and that will be a stupendous advantage if I can
get it working right.

I am doing a port of the entire server code base to both the MS and
Intel compilers.  Going pretty well so far.

On Mon, Apr 12, 2004 at 10:00:05PM +0200, Jeroen T. Vermeulen wrote:
> On Mon, Apr 12, 2004 at 12:35:15PM -0700, Dann Corbit wrote:
>  
> > I do know of important differences in compilers in this regard.  You can
> > (for instance) have 80 bit floating point on one compiler using double
> > but it is only 64 bits on another.
>  
> But in the case of x86 (among others) that's the in-register
> representation, no?  IIRC they are stored to memory as 64-bit doubles at
> best.

You also have "long double"s on some compilers which could be 80 bit.

> In C++, ABI compatibility is normally protected through a side effect of
> name mangling.  By maintaining different name mangling schemes for
> different ABI conventions, compiler vendors ensure that object files will
> refuse to link to other object files that adhere to different ABIs.

We gave up trying to make C++ dlls on windows because of ABI/name
mangling problems, never tried it again though.

The compilers from Microsoft and Borland atleast aren't
compatible.


Kurt

On Wed, Apr 14, 2004 at 12:22:18AM +0200, Kurt Roeckx wrote:

> > But in the case of x86 (among others) that's the in-register
> > representation, no?  IIRC they are stored to memory as 64-bit doubles at
> > best.
> 
> You also have "long double"s on some compilers which could be 80 bit.
Actually, they're a supported feature in both C99 and C++ IIRC.  But I
still suspect they're not *actually* 80 bits, at least not in memory.


> We gave up trying to make C++ dlls on windows because of ABI/name
> mangling problems, never tried it again though.
> 
> The compilers from Microsoft and Borland atleast aren't
> compatible.

But that shows up as link errors, not at runtime, right?


Jeroen

> > The compilers from Microsoft and Borland atleast aren't
> > compatible.
>
> But that shows up as link errors, not at runtime, right?

Correct.  Microsoft and Borland use different library packaging formats,
COFF and OMF.  However (non C++) DLLs are compatible and you can extract
a static lib from a dll in the format of your choice.

Merlin