Обсуждение: zlib detection in Meson on Windows broken?

Hi

I'm working on updating the build of PostgreSQL that pgAdmin uses in its Windows installers to use Meson ready for the v17 release. I'm using Visual Studio 2022, on Windows Server 2022.

I've been unable to persuade Meson to detect zlib, whilst OpenSSL seems to be fine.

The dependencies have been built and installed as follows:

 mkdir c:\build64

 wget https://zlib.net/zlib-1.3.2.tar.gz
 tar -zxvf zlib-1.3.2.tar.gz
 cd zlib-1.3.2
 cmake -DCMAKE_INSTALL_PREFIX=C:/build64/zlib -G "Visual Studio 17 2022" .
 msbuild ALL_BUILD.vcxproj /p:Configuration=Release
 msbuild RUN_TESTS.vcxproj /p:Configuration=Release
 msbuild INSTALL.vcxproj /p:Configuration=Release
 cd ..

 wget https://www.openssl.org/source/openssl-3.0.13.tar.gz
 tar -zxvf openssl-3.0.13.tar.gz
 cd openssl-3.0.013
 perl Configure VC-WIN64A no-asm --prefix=C:\build64\openssl no-ssl3 no-comp
 nmake
 nmake test
 nmake install
 cd ..

This results in the following headers and libraries being installed for zlib:

C:\Users\dpage\git\postgresql>dir C:\build64\zlib\include
 Volume in drive C has no label.
 Volume Serial Number is 3AAD-5864

 Directory of C:\build64\zlib\include

17/05/2024  15:56    <DIR>          .
17/05/2024  15:56    <DIR>          ..
17/05/2024  15:54            17,096 zconf.h
22/01/2024  19:32            96,829 zlib.h
               2 File(s)        113,925 bytes
               2 Dir(s)  98,842,726,400 bytes free

C:\Users\dpage\git\postgresql>dir C:\build64\zlib\lib
 Volume in drive C has no label.
 Volume Serial Number is 3AAD-5864

 Directory of C:\build64\zlib\lib

17/05/2024  17:01    <DIR>          .
17/05/2024  15:56    <DIR>          ..
17/05/2024  15:55            16,638 zlib.lib
17/05/2024  15:55           184,458 zlibstatic.lib
               2 File(s)        201,096 bytes
               2 Dir(s)  98,842,726,400 bytes free

I then attempt to build PostgreSQL:

 meson setup build -Dextra_include_dirs=C:/build64/openssl/include,C:/build64/zlib/include -Dextra_lib_dirs=C:/build64/openssl/lib,C:/build64/zlib/lib -Dssl=openssl -Dzlib=enabled --prefix=c:/build64/pgsql

Which results in the output in output.txt, indicating that OpenSSL was correctly found, but zlib was not. I've also attached the meson log.

I have very little experience with Meson, and even less interpreting it's logs, but it seems to me that it's not including the extra lib and include directories when it runs the test compile, given the command line it's reporting:

cl C:\Users\dpage\git\postgresql\build\meson-private\tmpg_h4xcue\testfile.c /nologo /showIncludes /utf-8 /EP /nologo /showIncludes /utf-8 /EP /Od /Oi-

Bug, or am I doing something silly?

--

Dave Page

pgAdmin: https://www.pgadmin.org

PostgreSQL: https://www.postgresql.org

EDB: https://www.enterprisedb.com

Hi,

On Tue, 21 May 2024 at 10:20, Sandeep Thakkar
<sandeep.thakkar@enterprisedb.com> wrote:
>
> Hi Dave,
>
> Is the .pc file generated after the successful build of zlib? If yes, then meson should be able to detect the installation ideally

If meson is not able to find the .pc file automatically, using 'meson
setup ... --pkg-config-path $ZLIB_PC_PATH' might help.

Hi Sandeep, Nazir,

On Tue, 21 May 2024 at 10:14, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:

Hi,

On Tue, 21 May 2024 at 10:20, Sandeep Thakkar
<sandeep.thakkar@enterprisedb.com> wrote:
>
> Hi Dave,
>
> Is the .pc file generated after the successful build of zlib? If yes, then meson should be able to detect the installation ideally

If meson is not able to find the .pc file automatically, using 'meson
setup ... --pkg-config-path $ZLIB_PC_PATH' might help.

The problem is that on Windows there are no standard locations for a Unix-style development library installation such as this, so the chances are that the .pc file will point to entirely the wrong location.

For example, please see https://github.com/dpage/winpgbuild/actions/runs/9172187335 which is a Github action that builds a completely vanilla zlib using VC++. If you look at the uploaded artefact containing the build output and example the .pc file, you'll see it references /zlib as the location, which is simply where I built it in that action. On a developer's machine that's almost certainly not going to be where it actually ends up. For example, on the pgAdmin build farm, the dependencies all end up in C:\build64\[whatever]. On the similar Github action I'm building for PostgreSQL, that artefact will be unpacked into /build/zlib.

Of course, for my own builds I can easily make everything use consistent directories, however most people who are likely to want to build PostgreSQL may not want to also build all the dependencies themselves as well, as some are a lot more difficult than zlib. So what tends to happen is people find third party builds or upstream official builds. 

I would therefore argue that if the .pc file that's found doesn't provide correct paths for us, then Meson should fall back to searching in the paths specified on its command line for the appropriate libraries/headers (which is what it does for OpenSSL for example, as that doesn't include a .pc file). This is also what happens with PG16 and earlier.

One other thing I will note is that PG16 and earlier try to use the wrong filename for the import library. For years, it's been a requirement to do something like this: "copy \zlib\lib\zlib.lib \zlib\lib\zdll.lib" to make a build succeed against a "vanilla" zlib build. I haven't got as far as figuring out if the same is true with Meson yet.

--

Dave Page

pgAdmin: https://www.pgadmin.org

PostgreSQL: https://www.postgresql.org

EDB: https://www.enterprisedb.com

Hi Dave,

On Tue, May 21, 2024 at 3:12 PM Dave Page <dpage@pgadmin.org> wrote:

Hi Sandeep, Nazir,

On Tue, 21 May 2024 at 10:14, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:

Hi,

On Tue, 21 May 2024 at 10:20, Sandeep Thakkar
<sandeep.thakkar@enterprisedb.com> wrote:
>
> Hi Dave,
>
> Is the .pc file generated after the successful build of zlib? If yes, then meson should be able to detect the installation ideally

If meson is not able to find the .pc file automatically, using 'meson
setup ... --pkg-config-path $ZLIB_PC_PATH' might help.

The problem is that on Windows there are no standard locations for a Unix-style development library installation such as this, so the chances are that the .pc file will point to entirely the wrong location.

For example, please see https://github.com/dpage/winpgbuild/actions/runs/9172187335 which is a Github action that builds a completely vanilla zlib using VC++. If you look at the uploaded artefact containing the build output and example the .pc file, you'll see it references /zlib as the location, which is simply where I built it in that action. On a developer's machine that's almost certainly not going to be where it actually ends up. For example, on the pgAdmin build farm, the dependencies all end up in C:\build64\[whatever]. On the similar Github action I'm building for PostgreSQL, that artefact will be unpacked into /build/zlib.

The above link returned 404. But I found a successful build at https://github.com/dpage/winpgbuild/actions/runs/9175426807. I downloaded the artifact but didn't find .pc file as I wanted to look into the content of that file.

I had a word with Murali who mentioned he encountered a similar issue while building PG17 on windows. He worked-around is by using a template .pc file that includes these lines:

--

prefix=${pcfiledir}/../..
exec_prefix=${prefix}
libdir=${prefix}/lib
sharedlibdir=${prefix}/lib

includedir=${prefix}/include

--

But in general I agree with you on the issue of Meson's dependency on pkgconfig files to detect the third party libraries. 

Of course, for my own builds I can easily make everything use consistent directories, however most people who are likely to want to build PostgreSQL may not want to also build all the dependencies themselves as well, as some are a lot more difficult than zlib. So what tends to happen is people find third party builds or upstream official builds. 

I would therefore argue that if the .pc file that's found doesn't provide correct paths for us, then Meson should fall back to searching in the paths specified on its command line for the appropriate libraries/headers (which is what it does for OpenSSL for example, as that doesn't include a .pc file). This is also what happens with PG16 and earlier.

One other thing I will note is that PG16 and earlier try to use the wrong filename for the import library. For years, it's been a requirement to do something like this: "copy \zlib\lib\zlib.lib \zlib\lib\zdll.lib" to make a build succeed against a "vanilla" zlib build. I haven't got as far as figuring out if the same is true with Meson yet.

--

Dave Page

pgAdmin: https://www.pgadmin.org

PostgreSQL: https://www.postgresql.org

EDB: https://www.enterprisedb.com

--

Sandeep Thakkar

Hi,

On 2024-05-20 11:58:05 +0100, Dave Page wrote:
> I have very little experience with Meson, and even less interpreting it's
> logs, but it seems to me that it's not including the extra lib and include
> directories when it runs the test compile, given the command line it's
> reporting:
>
> cl C:\Users\dpage\git\postgresql\build\meson-private\tmpg_h4xcue\testfile.c
> /nologo /showIncludes /utf-8 /EP /nologo /showIncludes /utf-8 /EP /Od /Oi-
>
> Bug, or am I doing something silly?

It's a buglet. We rely on meson's internal fallback detection of zlib, if it's
not provided via pkg-config or cmake. But it doesn't know about our
extra_include_dirs parameter. We should probably fix that...

Hi,

On 2024-05-20 11:58:05 +0100, Dave Page wrote:
> I then attempt to build PostgreSQL:
>
>  meson setup build
> -Dextra_include_dirs=C:/build64/openssl/include,C:/build64/zlib/include
> -Dextra_lib_dirs=C:/build64/openssl/lib,C:/build64/zlib/lib -Dssl=openssl
> -Dzlib=enabled --prefix=c:/build64/pgsql
>
> Which results in the output in output.txt, indicating that OpenSSL was
> correctly found, but zlib was not. I've also attached the meson log.

I forgot to mention that earlier: Assuming you're building something to be
distributed, I'd recommend --auto-features=enabled/disabled and specifying
specifically which dependencies you want to be used.

On 2024-05-21 Tu 11:04, Andres Freund wrote:
> Hi,
>
> On 2024-05-20 11:58:05 +0100, Dave Page wrote:
>> I have very little experience with Meson, and even less interpreting it's
>> logs, but it seems to me that it's not including the extra lib and include
>> directories when it runs the test compile, given the command line it's
>> reporting:
>>
>> cl C:\Users\dpage\git\postgresql\build\meson-private\tmpg_h4xcue\testfile.c
>> /nologo /showIncludes /utf-8 /EP /nologo /showIncludes /utf-8 /EP /Od /Oi-
>>
>> Bug, or am I doing something silly?
> It's a buglet. We rely on meson's internal fallback detection of zlib, if it's
> not provided via pkg-config or cmake. But it doesn't know about our
> extra_include_dirs parameter. We should probably fix that...
>

Yeah. Meanwhile, what I got working on a totally fresh Windows + VS
install was instead of using extra_include_dirs etc to add the relevant
directories to the environment LIB and INCLUDE settings before calling
`"meson setup".

Hi,

On Tue, 21 May 2024 at 18:24, Dave Page <dpage@pgadmin.org> wrote:
>
>
>
> On Tue, 21 May 2024 at 16:04, Andres Freund <andres@anarazel.de> wrote:
>>
>> Hi,
>>
>> On 2024-05-20 11:58:05 +0100, Dave Page wrote:
>> > I have very little experience with Meson, and even less interpreting it's
>> > logs, but it seems to me that it's not including the extra lib and include
>> > directories when it runs the test compile, given the command line it's
>> > reporting:
>> >
>> > cl C:\Users\dpage\git\postgresql\build\meson-private\tmpg_h4xcue\testfile.c
>> > /nologo /showIncludes /utf-8 /EP /nologo /showIncludes /utf-8 /EP /Od /Oi-
>> >
>> > Bug, or am I doing something silly?
>>
>> It's a buglet. We rely on meson's internal fallback detection of zlib, if it's
>> not provided via pkg-config or cmake. But it doesn't know about our
>> extra_include_dirs parameter. We should probably fix that...
>
>
> Oh good, then I'm not going bonkers. I'm still curious about how it works for Andrew but not me, however fixing that buglet should solve my issue, and would be sensible behaviour.
>
> Thanks!

I tried to install your latest zlib artifact (nmake one) to the
Windows CI images (not the official ones) [1]. Then, I used the
default meson.build file to build but meson could not find the zlib.
After that, I modified it like you suggested before; I used a
'cc.find_library()' to find zlib as a fallback method and it seems it
worked [2]. Please see meson setup logs below [3], does something
similar to the attached solve your problem?

The interesting thing is, I also tried this 'cc.find_library' method
with your old artifact (cmake one). It was able to find zlib but all
tests failed [4].

Experimental zlib meson.build diff is attached.

[1] https://cirrus-ci.com/task/6736867247259648
[2] https://cirrus-ci.com/build/5286228755480576
[3]
Run-time dependency zlib found: NO (tried pkgconfig, cmake and system)
Has header "zlib.h" : YES
Library zlib found: YES
...
  External libraries
...
    zlib                   : YES
...
[4] https://cirrus-ci.com/task/5208433811521536

--
Regards,
Nazir Bilal Yavuz
Microsoft

Hi,

On Wed, 22 May 2024 at 17:21, Dave Page <dpage@pgadmin.org> wrote:
>
> Hi
>
> On Wed, 22 May 2024 at 14:11, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
>>
>>
>> I tried to install your latest zlib artifact (nmake one) to the
>> Windows CI images (not the official ones) [1]. Then, I used the
>> default meson.build file to build but meson could not find the zlib.
>> After that, I modified it like you suggested before; I used a
>> 'cc.find_library()' to find zlib as a fallback method and it seems it
>> worked [2]. Please see meson setup logs below [3], does something
>> similar to the attached solve your problem?
>
>
> That patch does solve my problem - thank you!

I am glad that it worked!

Do you think that we need to have this patch in the upstream Postgres?
I am not sure because:
- There is a case that meson is able to find zlib but tests fail.
- This might be a band-aid fix rather than a permanent fix.