Обсуждение: [GENERAL] 64bit initdb failure on macOS 10.11 and 10.12

Hi,

I got this initdb error for a 64bit-build on macOS El Capitan and Sierra:

initdb -D /data/pg/hawk -E UTF8 --locale=C"
The files belonging to this database system will be owned by user "rjhb".
This user must also own the server process.

The database cluster will be initialized with locale "C".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /data/pg/hawk ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
creating template1 database in /data/pg/hawk/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating collations ... ok
creating conversions ... FATAL:  could not load library "/opt/bb/170704/lib/postgresql/ascii_and_mic.so":
dlopen(/opt/bb/170704/lib/postgresql/ascii_and_mic.so,10): Symbol not found: _check_encoding_conversion_args 
      Referenced from: /opt/bb/170704/lib/postgresql/ascii_and_mic.so
      Expected in: /opt/bb/170704/bin/postgres
     in /opt/bb/170704/lib/postgresql/ascii_and_mic.so
STATEMENT:  CREATE OR REPLACE FUNCTION ascii_to_mic (INTEGER, INTEGER, CSTRING, INTERNAL, INTEGER) RETURNS VOID AS
'$libdir/ascii_and_mic','ascii_to_mic' LANGUAGE C STRICT; 

child process exited with exit code 1
initdb: removing contents of data directory "/data/pg/hawk"
Exit 1


This is reproducible with 9.4.5, 9.5.4, 9.6.3, and 10 beta.
I configured with ./configure CC='gcc -m64' --prefix=/opt/bb/170704
and also tried with CFLAGS=-m64 LDFLAGS=-m64.

With the same procedures, but 32-bit-builds, I don't get this error,
and everything works fine.  Unfortunately, I need the 64-bit version.

Does the error ring a bell somewhere?

Thanks, Rainer

--
Email: rjhb@bb-c.de
Telefon: 0172/9593205

Brandt & Brandt Computer GmbH
Am Wiesenpfad 6, 53340 Meckenheim
Geschäftsführer: Rainer J.H. Brandt und Volker A. Brandt
Handelsregister: Amtsgericht Bonn, HRB 10513

rjhb@bb-c.de (Rainer J.H. Brandt) writes:
> I got this initdb error for a 64bit-build on macOS El Capitan and Sierra:

> creating conversions ... FATAL:  could not load library "/opt/bb/170704/lib/postgresql/ascii_and_mic.so":
dlopen(/opt/bb/170704/lib/postgresql/ascii_and_mic.so,10): Symbol not found: _check_encoding_conversion_args 
>       Referenced from: /opt/bb/170704/lib/postgresql/ascii_and_mic.so
>       Expected in: /opt/bb/170704/bin/postgres
>      in /opt/bb/170704/lib/postgresql/ascii_and_mic.so

check_encoding_conversion_args() should certainly be there in any PG
version released since 2009 (see src/backend/utils/mb/wchar.c).  But it's
unreferenced in the core Postgres executable, only in the loadable
conversion libraries.  I wonder if you have somehow enabled a link-time
optimization to remove "unreferenced" symbols, or at least not export them
to libraries.  Are you using Apple's stock toolchain, or something else?

> I configured with ./configure CC='gcc -m64' --prefix=/opt/bb/170704
> and also tried with CFLAGS=-m64 LDFLAGS=-m64.
> With the same procedures, but 32-bit-builds, I don't get this error,
> and everything works fine.  Unfortunately, I need the 64-bit version.

This makes little sense to me.  64-bit builds have been the default on
macOS for some time.

It's possible that by overriding LDFLAGS you're removing linker switches
that need to be there ...

            regards, tom lane

Tom Lane writes:
> rjhb@bb-c.de (Rainer J.H. Brandt) writes:
> > I got this initdb error for a 64bit-build on macOS El Capitan and Sierra:
>
> > creating conversions ... FATAL:  could not load library "/opt/bb/170704/lib/postgresql/ascii_and_mic.so":
dlopen(/opt/bb/170704/lib/postgresql/ascii_and_mic.so,10): Symbol not found: _check_encoding_conversion_args 
> >       Referenced from: /opt/bb/170704/lib/postgresql/ascii_and_mic.so
> >       Expected in: /opt/bb/170704/bin/postgres
> >      in /opt/bb/170704/lib/postgresql/ascii_and_mic.so
>
> check_encoding_conversion_args() should certainly be there in any PG
> version released since 2009 (see src/backend/utils/mb/wchar.c).  But it's
> unreferenced in the core Postgres executable, only in the loadable
> conversion libraries.  I wonder if you have somehow enabled a link-time
> optimization to remove "unreferenced" symbols, or at least not export them
> to libraries.  Are you using Apple's stock toolchain, or something else?
Yes: current Xcode, nothing else.

> > I configured with ./configure CC='gcc -m64' --prefix=/opt/bb/170704
> > and also tried with CFLAGS=-m64 LDFLAGS=-m64.
> > With the same procedures, but 32-bit-builds, I don't get this error,
> > and everything works fine.  Unfortunately, I need the 64-bit version.
>
> This makes little sense to me.  64-bit builds have been the default on
> macOS for some time.
Good to know.  I removed those options and tried again.

> It's possible that by overriding LDFLAGS you're removing linker switches
> that need to be there ...
Thanks for that suggestion.  I'm now using this (with 9.6.3):

./configure PERL=/opt/bb/170704/bin/perl --with-perl --prefix=/opt/bb/170704

I still get the error.  I don't see how Perl can make the difference,
but I guess it's not relevant that I can't see it ;-(

Thanks for taking the time to answer.
Rainer
--
Email: rjhb@bb-c.de
Telefon: 0172/9593205

Brandt & Brandt Computer GmbH
Am Wiesenpfad 6, 53340 Meckenheim
Geschäftsführer: Rainer J.H. Brandt und Volker A. Brandt
Handelsregister: Amtsgericht Bonn, HRB 10513

On 07/04/2017 01:29 PM, Rainer J.H. Brandt wrote:
> Tom Lane writes:
>> rjhb@bb-c.de (Rainer J.H. Brandt) writes:

>>
>> This makes little sense to me.  64-bit builds have been the default on
>> macOS for some time.
> Good to know.  I removed those options and tried again.

Did you run make clean before re-running ./configure?

>
>> It's possible that by overriding LDFLAGS you're removing linker switches
>> that need to be there ...
> Thanks for that suggestion.  I'm now using this (with 9.6.3):
>
> ./configure PERL=/opt/bb/170704/bin/perl --with-perl --prefix=/opt/bb/170704
>
> I still get the error.  I don't see how Perl can make the difference,
> but I guess it's not relevant that I can't see it ;-(
>
> Thanks for taking the time to answer.
> Rainer
>


--
Adrian Klaver
adrian.klaver@aklaver.com

Adrian Klaver writes:
> On 07/04/2017 01:29 PM, Rainer J.H. Brandt wrote:
> > Tom Lane writes:
> >> rjhb@bb-c.de (Rainer J.H. Brandt) writes:
>
> >>
> >> This makes little sense to me.  64-bit builds have been the default on
> >> macOS for some time.
> > Good to know.  I removed those options and tried again.
>
> Did you run make clean before re-running ./configure?
I removed everything, unpacked the sources, and ran the command given
below.

> >> It's possible that by overriding LDFLAGS you're removing linker switches
> >> that need to be there ...
> > Thanks for that suggestion.  I'm now using this (with 9.6.3):
> >
> > ./configure PERL=/opt/bb/170704/bin/perl --with-perl --prefix=/opt/bb/170704

Rainer
--
Email: rjhb@bb-c.de
Telefon: 0172/9593205

Brandt & Brandt Computer GmbH
Am Wiesenpfad 6, 53340 Meckenheim
Geschäftsführer: Rainer J.H. Brandt und Volker A. Brandt
Handelsregister: Amtsgericht Bonn, HRB 10513

Adrian Klaver <adrian.klaver@aklaver.com> writes:
> On 07/04/2017 01:29 PM, Rainer J.H. Brandt wrote:
>> Good to know.  I removed those options and tried again.

> Did you run make clean before re-running ./configure?

Personally I do "make distclean" before changing any configure options.
I'm not sure how much difference that really makes, but why waste brain
cells chasing such issues?  Build cycles are cheap.

The whole thing's odd though --- certainly many people are building
PG successfully on macOS.  There's got to be something unusual about
Rainer's build environment, but what?

            regards, tom lane

Tom Lane writes:
> Adrian Klaver <adrian.klaver@aklaver.com> writes:
> > On 07/04/2017 01:29 PM, Rainer J.H. Brandt wrote:
> >> Good to know.  I removed those options and tried again.
>
> > Did you run make clean before re-running ./configure?
>
> Personally I do "make distclean" before changing any configure options.
> I'm not sure how much difference that really makes, but why waste brain
> cells chasing such issues?  Build cycles are cheap.
Right, and it's not the issue here.  I always do each build in a freshly
unpacked source tree.

> The whole thing's odd though --- certainly many people are building
> PG successfully on macOS.  There's got to be something unusual about
> Rainer's build environment, but what?
I thought so, too, but I'm beginning to doubt it.  I'm now down to

  ./configure --prefix=/opt/bb/170705

and the initdb failure is the same.  The build machine is a few months old
and has current OS and Xcode, and certainly no other build tools or other
relevant stuff.  I have no special environment variables set.

It's been a while since I built PG on macOS, but I regularly do on other
operating systems, and haven't had any trouble in a very long time.

Rainer
--
Email: rjhb@bb-c.de
Telefon: 0172/9593205

Brandt & Brandt Computer GmbH
Am Wiesenpfad 6, 53340 Meckenheim
Geschäftsführer: Rainer J.H. Brandt und Volker A. Brandt
Handelsregister: Amtsgericht Bonn, HRB 10513

On 07/04/2017 11:56 PM, Rainer J.H. Brandt wrote:
> Tom Lane writes:
>> Adrian Klaver <adrian.klaver@aklaver.com> writes:
>>> On 07/04/2017 01:29 PM, Rainer J.H. Brandt wrote:
>>>> Good to know.  I removed those options and tried again.
>>
>>> Did you run make clean before re-running ./configure?
>>
>> Personally I do "make distclean" before changing any configure options.
>> I'm not sure how much difference that really makes, but why waste brain
>> cells chasing such issues?  Build cycles are cheap.
> Right, and it's not the issue here.  I always do each build in a freshly
> unpacked source tree.
>
>> The whole thing's odd though --- certainly many people are building
>> PG successfully on macOS.  There's got to be something unusual about
>> Rainer's build environment, but what?
> I thought so, too, but I'm beginning to doubt it.  I'm now down to
>
>    ./configure --prefix=/opt/bb/170705
>
> and the initdb failure is the same.  The build machine is a few months old
> and has current OS and Xcode, and certainly no other build tools or other
> relevant stuff.  I have no special environment variables set.

I do not build on OS X so this is a bit of a reach for me, still here it
goes. From your original post the error was:

"reating conversions ... FATAL:  could not load library
"/opt/bb/170704/lib/postgresql/ascii_and_mic.so":
dlopen(/opt/bb/170704/lib/postgresql/ascii_and_mic.so, 10): Symbol not
found: _check_encoding_conversion_args
      Referenced from: /opt/bb/170704/lib/postgresql/ascii_and_mic.so
      Expected in: /opt/bb/170704/bin/postgres
     in /opt/bb/170704/lib/postgresql/ascii_and_mic.so"

You also mentioned you have done 32 bit builds that worked.

Now when I do 64 bit build on Linux the libraries end up in ~/lib64/:

/usr/local/pgsql/lib64> l ascii_and_mic.so
-rwxr-xr-x 1 root root 9760 Jun 14 07:32 ascii_and_mic.so*

Yet in your case I see ~/lib/. Not sure how OS X handles 32bit/64bit,
which is where I am reaching. Just wondering if there is cross
contamination going on?

Another thought, what does:

ldd ascii_and_mic.so

show?


>
> It's been a while since I built PG on macOS, but I regularly do on other
> operating systems, and haven't had any trouble in a very long time.
>
> Rainer
>


--
Adrian Klaver
adrian.klaver@aklaver.com

rjhb@bb-c.de (Rainer J.H. Brandt) writes:
> Tom Lane writes:
>> The whole thing's odd though --- certainly many people are building
>> PG successfully on macOS.  There's got to be something unusual about
>> Rainer's build environment, but what?

> I thought so, too, but I'm beginning to doubt it.  I'm now down to

>   ./configure --prefix=/opt/bb/170705

> and the initdb failure is the same.  The build machine is a few months old
> and has current OS and Xcode, and certainly no other build tools or other
> relevant stuff.  I have no special environment variables set.

Hmph.  I confess bafflement --- it works fine for me on curremt macOS.

After reading the clang & ld man pages for a bit I wondered if I could
reproduce the problem by adding -flto to the compiler switches.  I was
right that that causes "unused" functions to get stripped out of the
postgres executable, but then you get a pile of failures when these
libraries get built:

undef: _pg_mic2ascii
undef: _check_encoding_conversion_args
undef: _pg_ascii2mic
Undefined symbols for architecture x86_64:
  "_pg_mic2ascii", referenced from:
      _mic_to_ascii in lto.o
  "_check_encoding_conversion_args", referenced from:
      _ascii_to_mic in lto.o
      _mic_to_ascii in lto.o
  "_pg_ascii2mic", referenced from:
      _ascii_to_mic in lto.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[3]: *** [ascii_and_mic.so] Error 1

Given that macOS's linker checks the resolvability of such symbols at
dylib build time, I really don't see how you could get through the
build successfully and then have a failure at run time.  It's weird.

Lacking any better ideas to offer, I can only suggest that maybe
removing/reinstalling Xcode would help.  (Hm, are you sure you've
installed Xcode's command line tools?  Try
xcode-select --install
to be sure.)

            regards, tom lane

Hi,

I'm coming back to an old thread.  My original issue isn't resolved yet;
this is just to answer your questions below.

Adrian Klaver writes:
> [...]
> I do not build on OS X so this is a bit of a reach for me, still here it
> goes. From your original post the error was:
>
> "reating conversions ... FATAL:  could not load library
> "/opt/bb/170704/lib/postgresql/ascii_and_mic.so":
> dlopen(/opt/bb/170704/lib/postgresql/ascii_and_mic.so, 10): Symbol not
> found: _check_encoding_conversion_args
>       Referenced from: /opt/bb/170704/lib/postgresql/ascii_and_mic.so
>       Expected in: /opt/bb/170704/bin/postgres
>      in /opt/bb/170704/lib/postgresql/ascii_and_mic.so"
>
> You also mentioned you have done 32 bit builds that worked.
I was wrong about that.  Everything was 64 bit.

> Now when I do 64 bit build on Linux the libraries end up in ~/lib64/:
On macOS, everything goes into the same $PREFIX/lib.  A library can be
32bit, or 64bit, or both (a "universal binary").  macOS has a tool
named "lipo" that can create those.  Usually, you get 64bit-only libraries.
Once I realized this (thanks to Tom Lane), I dropped all configure
arguments related to that.

> /usr/local/pgsql/lib64> l ascii_and_mic.so
> -rwxr-xr-x 1 root root 9760 Jun 14 07:32 ascii_and_mic.so*
>
> Yet in your case I see ~/lib/. Not sure how OS X handles 32bit/64bit,
> which is where I am reaching. Just wondering if there is cross
> contamination going on?
No, that's not the problem.  I only have 64 bit stuff.

> ldd ascii_and_mic.so
On macOS, there's no ldd.  The equivalent is "otool -L".
It looks like this (line breaks inserted for readability):

  % file /opt/pg/lib/postgresql/ascii_and_mic.so
  /opt/pg/lib/postgresql/ascii_and_mic.so: Mach-O 64-bit bundle x86_64

  % otool -L /opt/pg/lib/postgresql/ascii_and_mic.so
  /opt/pg/lib/postgresql/ascii_and_mic.so:
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0,
      current version 1238.60.2)

BTW, libSystem.B.dylib is a universal binary:

  % file /usr/lib/libSystem.B.dylib
  /usr/lib/libSystem.B.dylib: Mach-O universal binary with 2 architectures:
    [x86_64: Mach-O 64-bit dynamically linked shared library x86_64]
    [i386: Mach-O dynamically linked shared library i386]
  /usr/lib/libSystem.B.dylib (for architecture x86_64):
    Mach-O 64-bit dynamically linked shared library x86_64
  /usr/lib/libSystem.B.dylib (for architecture i386):
    Mach-O dynamically linked shared library i386

This 32/64 business isn't the problem, though.

I've found out that everything works if I type the configure/make commands
in my shell (which isn't what I said before, sorry about that), but stops
working if I put them into a Perl script (which is how I automate things
across OSes).  The weird thing is that the commands are absolutely the same,
and environment is the same, too.  Anyway, it seems clear that the error is
on my side.  Given enough time, I will find it and report to you.

Thanks, Rainer
--
Email: rjhb@bb-c.de  +  Telefon: 02448/919126  +  Mobiltelefon: 0172/9593205
Brandt & Brandt Computer GmbH  +  Am Wiesenpfad 6, 53340 Meckenheim
Geschäftsführer: Rainer J.H. Brandt und Volker A. Brandt
Handelsregister: Amtsgericht Bonn, HRB 10513

rjhb@bb-c.de (Rainer J.H. Brandt) writes:
> This 32/64 business isn't the problem, though.

> I've found out that everything works if I type the configure/make commands
> in my shell (which isn't what I said before, sorry about that), but stops
> working if I put them into a Perl script (which is how I automate things
> across OSes).

Ooooh, that's interesting ...

> The weird thing is that the commands are absolutely the same,
> and environment is the same, too.

I bet not.  We've seen problems with macOS unexpectedly deciding to
filter away inherited environment variables in some situations.
It might be useful to put "env >somefile" into the PG makefile and
compare results between the two ways of invoking it.

            regards, tom lane

Tom Lane writes:
> rjhb@bb-c.de (Rainer J.H. Brandt) writes:
> > This 32/64 business isn't the problem, though.
>
> > I've found out that everything works if I type the configure/make commands
> > in my shell (which isn't what I said before, sorry about that), but stops
> > working if I put them into a Perl script (which is how I automate things
> > across OSes).
>
> Ooooh, that's interesting ...
>
> > The weird thing is that the commands are absolutely the same,
> > and environment is the same, too.
>
> I bet not.  We've seen problems with macOS unexpectedly deciding to
> filter away inherited environment variables in some situations.
> It might be useful to put "env >somefile" into the PG makefile and
> compare results between the two ways of invoking it.

Between the configure and make steps, I put it into the top level
GNUmakefile.  Here's the diff:

18c18
< PWD=/private/tmp/buildbot.9365/postgresql-9.6.3
---
> PWD=/tmp/w/postgresql-9.6.3
21c21
< SHLVL=3
---
> SHLVL=2

This subshell level isn't relevant, or is it?
Here's the full output of the manual version:

Apple_PubSub_Socket_Render=/private/tmp/com.apple.launchd.tMcsjxst4H/Render
DISPLAY=/private/tmp/com.apple.launchd.EahVZI82Sz/org.macosforge.xquartz:0
EDITOR=emacsclient
GROUP=people
HOME=/Users/rjhb
HOST=hawk.local
HOSTTYPE=unknown
LANG=en_US.UTF-8
LOGNAME=rjhb
MACHTYPE=x86_64
MAKEFLAGS=
MAKELEVEL=1
MANPATH=/usr/share/man:/Users/rjhb/doc/man:/opt/local/share/man
MFLAGS=
MYDISPLAY=/private/tmp/com.apple.launchd.EahVZI82Sz/org.macosforge.xquartz:0
OSTYPE=darwin
PATH=/usr/bin:/usr/sbin:/bin:/sbin
PWD=/tmp/w/postgresql-9.6.3
SECURITYSESSIONID=186a6
SHELL=/bin/tcsh
SHLVL=2
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.yAcr2eFkb9/Listeners
TERM=xterm-256color
TERM_PROGRAM=Apple_Terminal
TERM_PROGRAM_VERSION=388.1.1
TERM_SESSION_ID=3BEAA846-F8B4-46AC-9CB6-17ADC709E34F
TMPDIR=/var/folders/75/60nrw1lx53b5jwl1dz7g5v_40000z9/T/
USER=rjhb
VENDOR=apple
XPC_FLAGS=0x0
XPC_SERVICE_NAME=0
_=/usr/bin/env
__CF_USER_TEXT_ENCODING=0x3E9:0x0:0x0

Regards, Rainer

rjhb@bb-c.de (Rainer J.H. Brandt) writes:
> Tom Lane writes:
>> I bet not.  We've seen problems with macOS unexpectedly deciding to
>> filter away inherited environment variables in some situations.
>> It might be useful to put "env >somefile" into the PG makefile and
>> compare results between the two ways of invoking it.

> Between the configure and make steps, I put it into the top level
> GNUmakefile.  Here's the diff:
> 18c18
> < PWD=/private/tmp/buildbot.9365/postgresql-9.6.3
> ---
> > PWD=/tmp/w/postgresql-9.6.3
> 21c21
> < SHLVL=3
> ---
> > SHLVL=2
> This subshell level isn't relevant, or is it?

No, probably not --- so that theory seems like a failure.

> Here's the full output of the manual version:

Hmmm ...

> SHELL=/bin/tcsh

Mine's bash ... I wonder whether that could make a difference here?
I'm pretty sure the PG makefiles aren't set up for csh syntax.
But I can't see how that would make it work in the manual case
and not when going through Perl.

            regards, tom lane

Hello everybody,

I have found my mistake.  I apologize for not telling the whole truth
about my build process.  The answer is what it had to be:  There was one
step I had forgotten about:  My build script stripped all binaries, i.e.
it ran /usr/bin/strip on all of them.

That's especially embarrasing because you had already asked whether I had
'somehow enabled a link-time optimization to remove "unreferenced" symbols'.

I've grown up during times when disk space was precious, and it seems that
I have to reconsider all my old habits.  It's interesting that it's the
first time after building hundreds of software packages on many different
systems over many years, that I have broken something by using strip.  It
still doesn't cause trouble on the other OSes that I use, but maybe I'm
going to abandon it.

Thank you for your patience.
Regards, Rainer
--
Email: rjhb@bb-c.de  +  Telefon: 02448/919126  +  Mobiltelefon: 0172/9593205
Brandt & Brandt Computer GmbH  +  Am Wiesenpfad 6, 53340 Meckenheim
Geschäftsführer: Rainer J.H. Brandt und Volker A. Brandt
Handelsregister: Amtsgericht Bonn, HRB 10513

rjhb@bb-c.de (Rainer J.H. Brandt) writes:
> I have found my mistake.  I apologize for not telling the whole truth
> about my build process.  The answer is what it had to be:  There was one
> step I had forgotten about:  My build script stripped all binaries, i.e.
> it ran /usr/bin/strip on all of them.

Hah ... so then there wasn't anything specific to
check_encoding_conversion_args, that just happened to be the first symbol
the dynamic linker tried to look up.

I see that Apple's version of strip(1) has an option to supply a file
listing symbols to be kept.  So if we had an authoritative list of the
symbols meant to be exported from core Postgres to extensions, you could
use that to filter out unnecessary symbols.  Unfortunately, no such
list has ever been prepared :-(

            regards, tom lane