Обсуждение: Misidentification of Python shared library
While chasing down last night's failure on buildfarm member longfin,
I came across an interesting factoid. If you just do
./configure --enable-sharedmakemake install
in unmodified Python sources, what you will get is an install tree in
which libpython.so (or local equivalent such as .dylib) is installed
in /usr/local/lib, while libpython.a is installed in a directory named
like /usr/local/lib/python3.5/config-3.5m. I've verified this behavior
on both Linux and macOS.
The trouble is that we are picking up /usr/local/lib/python3.5/config-3.5m
as the python_libdir. This means that we think we are using a python
shared library (because we see that Python was configured to build one)
but what we are actually linking with is a static library. This happens
to work, because the code in the static library was built with -fpic,
but it results in an undesirably bloated plpython.so.
Or at least it did work until commit d51924be8. With that, plpython.so
and hstore_plpython.so are each bloated with their very own copy of the
Python code. And they fail to interoperate at all, which is why longfin
crashed. I didn't investigate very hard, but it looks to me like Python
assumes that symbols like Py_None will have unique addresses, which they
don't in this scenario.
There's a comment in config/python.m4 that correctly describes what
Python is exposing:
# Note: selecting libpython from python_configdir works in all Python
# releases, but it generally finds a non-shared library, which means
# that we are binding the python interpreter right into libplpython.so.
# In Python 2.3 and up there should be a shared library available in
# the main library location.
but the code below this is not coming to the right conclusions, at least
not in Python 3.5. What I'm seeing on macOS is that we get
"libpython3.5m.dylib" for python_ldlibrary, which seems right, but
python_so comes out as ".cpython-35m-darwin.so", whereas the code looks
like it's expecting that to be ".dylib". The lack of match causes
the if-test just below that to make the wrong choice. So some sort of
fix is needed here.
Also, I propose that we'd be better off to not rely on Py_ENABLE_SHARED
at all, but just to run the code in configure.in that insists on
finding something in $python_libdir that is named like a shared library,
because installation trees like this mean that Py_ENABLE_SHARED is
totally untrustworthy as a guide to what is actually in any particular
library directory. Moreover, this stanza:
if test "$PORTNAME" = darwin; then # macOS does supply a .dylib even though Py_ENABLE_SHARED does # not get
set. The file detection logic below doesn't succeed # on older macOS versions, so make it explicit.
python_enable_shared=1
is completely misguided because it presumes we are using the
Apple-supplied python library and not a hand-installed one. I think
the similar exception for Windows is probably bogus for the same reason.
In short: I propose replacing all of this logic with "if there's something
in $python_libdir that has the right name to be a python shared library,
use that, else try the same in $python_configdir, else fail". Thoughts?
regards, tom lane
On 10/04/2016 10:43 AM, Tom Lane wrote: > While chasing down last night's failure on buildfarm member longfin, > I came across an interesting factoid. If you just do > > ./configure --enable-shared > make > make install > > in unmodified Python sources, what you will get is an install tree in > which libpython.so (or local equivalent such as .dylib) is installed > in /usr/local/lib, while libpython.a is installed in a directory named > like /usr/local/lib/python3.5/config-3.5m. I've verified this behavior > on both Linux and macOS. Wow, these modules have uncovered a number of cans of worms. > > > In short: I propose replacing all of this logic with "if there's something > in $python_libdir that has the right name to be a python shared library, > use that, else try the same in $python_configdir, else fail". Thoughts? > > Seems reasonable. cheers andrew
Andrew Dunstan <andrew@dunslane.net> writes:
> On 10/04/2016 10:43 AM, Tom Lane wrote:
>> In short: I propose replacing all of this logic with "if there's something
>> in $python_libdir that has the right name to be a python shared library,
>> use that, else try the same in $python_configdir, else fail". Thoughts?
> Seems reasonable.
Actually, given that we no longer support Python < 2.3, I am not sure
we need to bother with looking in $python_configdir at all.
But looking at the installations on my older Apple boxes, it seems that
*neither* $python_libdir nor $python_configdir contain any library at all;
rather libpython.dylib sits in good old /usr/lib and is found there
despite the -L switch pointing to noplace useful. So I'm now thinking
that we should do the above dance in a list of directories like
"$python_libdir /usr/lib64 /usr/lib" (too bad there's no easy
platform-independent way to find out the linker's default search path).
We can add $python_configdir to that if it proves necessary, but I'll try
it without first.
regards, tom lane
On Tue, Oct 4, 2016 at 11:45 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > "$python_libdir /usr/lib64 /usr/lib" (too bad there's no easy > platform-independent way to find out the linker's default search path). "too bad" sounds like a grave understatement of the perils involved using a fixed list of paths which may or may not bear any resemblance at all to what the platform actually uses. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Robert Haas <robertmhaas@gmail.com> writes:
> On Tue, Oct 4, 2016 at 11:45 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> "$python_libdir /usr/lib64 /usr/lib" (too bad there's no easy
>> platform-independent way to find out the linker's default search path).
> "too bad" sounds like a grave understatement of the perils involved
> using a fixed list of paths which may or may not bear any resemblance
> at all to what the platform actually uses.
Sure, but it will only matter given a more-or-less-broken Python
installation. Anyway, if you have a better idea, let's hear it.
regards, tom lane
On Tue, Oct 4, 2016 at 12:57 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Robert Haas <robertmhaas@gmail.com> writes: >> On Tue, Oct 4, 2016 at 11:45 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: >>> "$python_libdir /usr/lib64 /usr/lib" (too bad there's no easy >>> platform-independent way to find out the linker's default search path). > >> "too bad" sounds like a grave understatement of the perils involved >> using a fixed list of paths which may or may not bear any resemblance >> at all to what the platform actually uses. > > Sure, but it will only matter given a more-or-less-broken Python > installation. Anyway, if you have a better idea, let's hear it. Sorry, no clue.... -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Robert Haas <robertmhaas@gmail.com> writes:
> On Tue, Oct 4, 2016 at 12:57 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Sure, but it will only matter given a more-or-less-broken Python
>> installation. Anyway, if you have a better idea, let's hear it.
> Sorry, no clue....
So I pushed that, and most of the Debian-based buildfarm critters
don't like it. Where does Debian keep the python shlib, pray tell?
regards, tom lane
On Tue, Oct 4, 2016 at 3:38 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Robert Haas <robertmhaas@gmail.com> writes: >> On Tue, Oct 4, 2016 at 12:57 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: >>> Sure, but it will only matter given a more-or-less-broken Python >>> installation. Anyway, if you have a better idea, let's hear it. > >> Sorry, no clue.... > > So I pushed that, and most of the Debian-based buildfarm critters > don't like it. Where does Debian keep the python shlib, pray tell? ...must...not...say...I...told...you...so... Even if you eventually get this working on every flavor of Linux that's represented in the buildfarm, it's likely to be unreliable on other versions of Linux and/or other UNIX-like operating systems because operating system packagers *love* to find new places to store things, and system administrators expect to be able to compile stuff with --prefix and have things work after that, at least if they then pass the right configure flags to the next thing they want to install. I bet you a nickle that if you include a hard-coded list of paths in any form, at least one operating system packager is going to have to patch that hard-coded list in order to get things to work -- and a quarter that at least one user will have to patch it. Like I already said, I don't have a better solution, but all of my experience says that this one is not going to be reliable no matter how hard you beat it. I will be happy if you can prove me wrong, though. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Robert Haas <robertmhaas@gmail.com> writes:
> On Tue, Oct 4, 2016 at 3:38 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> So I pushed that, and most of the Debian-based buildfarm critters
>> don't like it. Where does Debian keep the python shlib, pray tell?
> ...must...not...say...I...told...you...so...
Meh. How about you don't say it, until we get some evidence that that
actually has anything to do with it?
My immediate guess is that Debian decided that sticking the shlib in
${python_configdir} would be a cute idea after all, because it's hard to
think of what else they might've done that would have succeeded with the
old code and not the new. But I'd rather not waste time on blind fix
attempts when there are people around here who can look at their
installations and tell me.
Also, I notice that on the animals that are now failing, the last
successful runs reported
checking how to link an embedded Python application... -L/usr/lib -lpython2.7 -lpthread -ldl -lutil -lm
^^^^^^^^
which would if anything suggest that the shlib *is* in /usr/lib on
these machines. So maybe I fat-fingered some other aspect entirely,
but I can't yet guess what.
regards, tom lane
Aidan Van Dyk <aidan@highrise.ca> writes: > https://packages.debian.org/jessie/amd64/libpython3.4/filelist > The ".so.<SOVERSION>" is in /usr/lib/<system>, but ".so" symlink is in the > config directory... Hah. I guess that explains why they are setting LIBDIR to /usr/lib. Thanks for the info! regards, tom lane