Обсуждение: Re: [PATCH] audo-detect and use -moutline-atomics compilation flag for aarch64
Re: [PATCH] audo-detect and use -moutline-atomics compilation flag for aarch64
Hi Christoph,
This is a follow-up email based on recommendations from Heikki Linnakangas:
https://www.postgresql.org/message-id/5f048a5a-c471-50ae-30ad-385dcafc616b%40iki.fi
and Tom Lane:
https://www.postgresql.org/message-id/888692.1601489331%40sss.pgh.pa.us
> Maybe you should speak to the distribution vendors or the folk packaging PostgreSQL for those distributions, instead.
Following the initial post by Tsahi Zidenberg:
https://www.postgresql.org/message-id/099F69EE-51D3-4214-934A-1F28C0A1A7A7%40amazon.com
> The patch was tested to improve pgbench simple-update by 10% and sysbench write-only by 3% on a 64-core armv8.2 machine (AWS m6g.16xlarge).
The change is to compile PostgreSQL for arm64 with an extra flag -moutline-atomics when the compiler is gcc-8.5 or gcc-9.4.
-moutline-atomics is enabled by default in all newer compilers gcc-{10, 11, trunk}, and on clang-{12, 13, trunk}.
The change is needed only for a limited number of Debian and Ubuntu releases compiled by gcc versions with the outline-atomics flag disabled by default.
Re: [PATCH] audo-detect and use -moutline-atomics compilation flag for aarch64
Re: Pop, Sebastian > ?Hi Christoph, > > > This is a follow-up email based on recommendations from Heikki Linnakangas: > > https://www.postgresql.org/message-id/5f048a5a-c471-50ae-30ad-385dcafc616b%40iki.fi > > and Tom Lane: > > https://www.postgresql.org/message-id/888692.1601489331%40sss.pgh.pa.us Well I wouldn't exactly call these messages "recommendations". > ?The change is to compile PostgreSQL for arm64 with an extra flag -moutline-atomics when the compiler is gcc-8.5 or gcc-9.4. > > -moutline-atomics is enabled by default in all newer compilers gcc-{10, 11, trunk}, and on clang-{12, 13, trunk}. > > The change is needed only for a limited number of Debian and Ubuntu releases compiled by gcc versions with the outline-atomicsflag disabled by default. None of the Debian releases are using 8.5 or 9.4, and on the Ubuntu side focal is using 9.3. Is that gcc version even affected? Christoph
Re: [PATCH] audo-detect and use -moutline-atomics compilation flag for aarch64
> on the Ubuntu side focal is using 9.3. Is that gcc version even affected? Yes, we have backported the outline-atomics flag to Ubuntu 20.04 gcc-9.3. On arm64 you can check whether a compiler has the option with: $ gcc -v --help -Q 2>/dev/null | grep outline-atomics -moutline-atomics [disabled] $ gcc --version gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0 [...] Sebastian
Re: [PATCH] audo-detect and use -moutline-atomics compilation flag for aarch64
Re: Pop, Sebastian > > on the Ubuntu side focal is using 9.3. Is that gcc version even affected? > > Yes, we have backported the outline-atomics flag to Ubuntu 20.04 gcc-9.3. > > On arm64 you can check whether a compiler has the option with: > $ gcc -v --help -Q 2>/dev/null | grep outline-atomics > -moutline-atomics [disabled] > $ gcc --version > gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0 Not here: $ gcc --version gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0 $ gcc -v --help -Q 2>/dev/null | grep outline-atomics Did you rebuild the package locally without changing the Debian version? Christoph
Re: [PATCH] audo-detect and use -moutline-atomics compilation flag for aarch64
> Not here: > $ gcc --version > gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0 > $ gcc -v --help -Q 2>/dev/null | grep outline-atomics This needs to be on an arm64 machine. The outline-atomics flag does not exist when targeting x86_64. The flag has been back-ported to Ubuntu 20.04 a year and a half ago: IIRC, it went in just before the release in April 2020. > Did you rebuild the package locally without changing the Debian version? I have not re-compiled gcc. The gcc-9.3 is the one distributed as part of Ubuntu 20.04. If you need access to Graviton2 instances, we have a system of credits that we can use to fund compute time for open-sourceCI projects. Let me know if you are interested in getting those compute credits, and I will take the request to my management. Sebastian
Re: [PATCH] audo-detect and use -moutline-atomics compilation flag for aarch64
Hi Christoph, Could you please let me know what is the process to enable -moutline-atomics for arm64 binaries distributed by PostgreSQLfor Ubuntu 20.04? Using the right LSE atomic instructions is very important for the performance of PostgreSQL on all new arm64 processors. Please let me know how I can help to enable the flag, test, and benchmark the new packages. Thanks, Sebastian
Re: [PATCH] audo-detect and use -moutline-atomics compilation flag for aarch64
Re: Pop, Sebastian > > Not here: > > $ gcc --version > > gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0 > > $ gcc -v --help -Q 2>/dev/null | grep outline-atomics > > This needs to be on an arm64 machine. Oh, sure. Sorry I was missing a lot of context that wasn't immediately clear. > The outline-atomics flag does not exist when targeting x86_64. > The flag has been back-ported to Ubuntu 20.04 a year and a half ago: IIRC, it went in just before the release in April2020. It is indeed present there, but not enabled: $ for chroot in *; do echo $chroot:; schroot -c $chroot -- gcc -v --help -Q 2>&1 | grep outline-atomics; done bionic-arm64: bookworm-arm64: -moutline-atomics [enabled] bullseye-arm64: -moutline-atomics [enabled] buster-arm64: focal-arm64: -moutline-atomics [disabled] sid-arm64: -moutline-atomics [enabled] The (newer) Debian dists have it enabled by default. > > Did you rebuild the package locally without changing the Debian version? > > I have not re-compiled gcc. The gcc-9.3 is the one distributed as part of Ubuntu 20.04. Again, that was a lot of context I didn't have. > If you need access to Graviton2 instances, we have a system of credits that we can use to fund compute time for open-sourceCI projects. > Let me know if you are interested in getting those compute credits, and I will take the request to my management. Thanks for the offer - we already have a arm64 build node sponsored by Huawei that does the job for now. Re: Pop, Sebastian > Could you please let me know what is the process to enable -moutline-atomics for arm64 binaries distributed by PostgreSQLfor Ubuntu 20.04? > Using the right LSE atomic instructions is very important for the performance of PostgreSQL on all new arm64 processors. > Please let me know how I can help to enable the flag, test, and benchmark the new packages. First, we need to be convinced it's a good idea. From that I got from the messages you were linking to, Heikki "definitely do[es]n't want to override that decision", and Tom doesn't "find this argument terribly convincing". The benchmarks in https://www.postgresql.org/message-id/1C8D0E58-FB33-4105-AC00-8FA07621F5DD%40amazon.com don't look like the option is a game-changer. Even if we are convinced, we don't have the infrastructure in the package yet to do architecture- and distribution-specific optimizations. Would you (Amazon?) be willing to sponsor the required package engineering? Christoph
Re: [PATCH] audo-detect and use -moutline-atomics compilation flag for aarch64
> The benchmarks in > https://www.postgresql.org/message-id/1C8D0E58-FB33-4105-AC00-8FA07621F5DD%40amazon.com > don't look like the option is a game-changer. Since then we have seen benchmarks that are several orders of magnitude faster with LSE instructions. I will get the new data and share. > Even if we are convinced, we don't have the infrastructure in the > package yet to do architecture- and distribution-specific > optimizations. Would you (Amazon?) be willing to sponsor the required > package engineering? Yes, I am willing to do whatever is necessary to enable the flag on PostgreSQL builds for Ubuntu 20.04. Please point me to the (git?) repo and the files that need to be changed. Thanks, Sebastian
Re: [PATCH] audo-detect and use -moutline-atomics compilation flag for aarch64
Re: Pop, Sebastian > > The benchmarks in > > https://www.postgresql.org/message-id/1C8D0E58-FB33-4105-AC00-8FA07621F5DD%40amazon.com > > don't look like the option is a game-changer. > > Since then we have seen benchmarks that are several orders of magnitude faster with LSE instructions. > I will get the new data and share. > > > Even if we are convinced, we don't have the infrastructure in the > > package yet to do architecture- and distribution-specific > > optimizations. Would you (Amazon?) be willing to sponsor the required > > package engineering? > > Yes, I am willing to do whatever is necessary to enable the flag on PostgreSQL builds for Ubuntu 20.04. > Please point me to the (git?) repo and the files that need to be changed. https://salsa.debian.org/postgresql/postgresql-common.git https://salsa.debian.org/postgresql/postgresql.git https://git.postgresql.org/gitweb/?p=pgapt.git;a=summary Christoph
Re: [PATCH] audo-detect and use -moutline-atomics compilation flag for aarch64
On 11/11/21, 11:14 AM, "Christoph Berg" <myon@debian.org> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you canconfirm the sender and know the content is safe. Re: Pop, Sebastian > > The benchmarks in > > https://www.postgresql.org/message-id/1C8D0E58-FB33-4105-AC00-8FA07621F5DD%40amazon.com > > don't look like the option is a game-changer. > > Since then we have seen benchmarks that are several orders of magnitude faster with LSE instructions. > I will get the new data and share. > > > Even if we are convinced, we don't have the infrastructure in the > > package yet to do architecture- and distribution-specific > > optimizations. Would you (Amazon?) be willing to sponsor the required > > package engineering? > > Yes, I am willing to do whatever is necessary to enable the flag on PostgreSQL builds for Ubuntu 20.04. > Please point me to the (git?) repo and the files that need to be changed. The binary compiled with -moutline-atomics works for all arm64 architecture versions. I don't believe there would be newpackage engineering required unless I'm missing something, but if there is, yes, we're absolutely happy to help. Thanks, Ali
Re: [PATCH] audo-detect and use -moutline-atomics compilation flag for aarch64
Thanks Christoph for the pointers. I see that the source code of PostgreSQL is patched before builds: i.e., the configure.in file has this patch applied to it https://salsa.debian.org/postgresql/postgresql/-/blob/13/debian/patches/autoconf2.69 Following the original patch from Tsahi Zidenberg, https://www.postgresql.org/message-id/attachment/111983/0001-Support-outline-atomics-on-aarch64.patch I would like to add the attached patch to the list of patches applied before build. Christoph, could you please let me know how I should test the patch? Thanks, Sebastian
Вложения
Re: [PATCH] audo-detect and use -moutline-atomics compilation flag for aarch64
Re: Pop, Sebastian > Thanks Christoph for the pointers. > > I see that the source code of PostgreSQL is patched before builds: > i.e., the configure.in file has this patch applied to it https://salsa.debian.org/postgresql/postgresql/-/blob/13/debian/patches/autoconf2.69 > > Following the original patch from Tsahi Zidenberg, > https://www.postgresql.org/message-id/attachment/111983/0001-Support-outline-atomics-on-aarch64.patch > I would like to add the attached patch to the list of patches applied before build. That will fail on bionic. > Christoph, could you please let me know how I should test the patch? Apply it, and run "make check-world" on focal. The others should be unaffected since the flag is on by default. Christoph
Re: [PATCH] audo-detect and use -moutline-atomics compilation flag for aarch64
>> The benchmarks in >> https://www.postgresql.org/message-id/1C8D0E58-FB33-4105-AC00-8FA07621F5DD%40amazon.com >> don't look like the option is a game-changer. > > Since then we have seen benchmarks that are several orders of magnitude faster with LSE instructions. > I will get the new data and share. On an m6gd.metal with PostgreSQL v11 test: pgbench - select-only Measure tps (transactions/s), higher is better. build without LSE: 132143 tps with LSE atomics: 844552 tps That is a 6.39x speedup. Sebastian
Re: [PATCH] audo-detect and use -moutline-atomics compilation flag for aarch64
> That will fail on bionic. The autoconf code does a check that the compiler supports the -moutline-atomics flag. The change will still be fine on bionic and all other distros with a gcc without the flag. > > Christoph, could you please let me know how I should test the patch? > Apply it, and run "make check-world" on focal. The others should be > unaffected since the flag is on by default. Thanks for the instructions. I will test the patch and let you know. Sebastian
Re: [PATCH] audo-detect and use -moutline-atomics compilation flag for aarch64
> Apply it, and run "make check-world" on focal. The others should be > unaffected since the flag is on by default. Here are the steps in my testing on focal - ubuntu 20.04: git clone https://github.com/postgres/postgres cd postgres patch -p1 < a.diff autoconf make make check-world All checks passed on a c6g Graviton2 instance. configure.in was renamed configure.ac in July 2020. For older postgresql versions we need to patch configure.in. See attached the patch a.diff that I used on trunk postgresql. Thanks, Sebastian
Вложения
Re: [PATCH] audo-detect and use -moutline-atomics compilation flag for aarch64
sudo apt install build-essential libreadline-dev zlib1g-dev bison flex > git clone https://github.com/postgres/postgres > cd postgres > patch -p1 < a.diff > autoconf ./configure > make > make check-world > > All checks passed on a c6g Graviton2 instance. I also run the above steps on bionic - ubuntu 18.04 - to check that the patch passes when gcc does not have the -moutline-atomicsflag. All checks pass on bionic on a c6g instance. Thanks Ali for recommending the extra test. Sebastian
Re: [PATCH] audo-detect and use -moutline-atomics compilation flag for aarch64
Hi Christoph, a quick follow-up email to make sure you have everything you need to enable -moutline-atomics flag on Ubuntu 20.04 buildsof PostgreSQL. Please let me know if you need more experiments or tests with the patch. Thanks, Sebastian
Re: [PATCH] audo-detect and use -moutline-atomics compilation flag for aarch64
Re: Pop, Sebastian > I also run the above steps on bionic - ubuntu 18.04 - to check that the patch passes when gcc does not have the -moutline-atomicsflag. > All checks pass on bionic on a c6g instance. > Thanks Ali for recommending the extra test. Thanks for verifying that part as well, I'll try throwing in the patch now. Christoph
Re: [PATCH] audo-detect and use -moutline-atomics compilation flag for aarch64
Re: To Pop, Sebastian > > I also run the above steps on bionic - ubuntu 18.04 - to check that the patch passes when gcc does not have the -moutline-atomicsflag. > > All checks pass on bionic on a c6g instance. > > Thanks Ali for recommending the extra test. > > Thanks for verifying that part as well, I'll try throwing in the patch > now. Things are looking good on PG14, but PG10 and 9.6 fail: configure.in:485: error: possibly undefined macro: PGAC_PROG_CXX_CFLAGS_OPT https://pgdgbuild.dus.dg-i.net/job/postgresql-10-binaries/architecture=ppc64el,distribution=focal/76/console So we'll have it on PG 11 .. 14, unless the regression tests act up. Christoph
Re: [PATCH] audo-detect and use -moutline-atomics compilation flag for aarch64
> Things are looking good on PG14, but PG10 and 9.6 fail: > configure.in:485: error: possibly undefined macro: PGAC_PROG_CXX_CFLAGS_OPT Thanks Christoph for the heads up. I will investigate the autoconf issue and send you a patch specific to PG10 and 9. Most likely the PGAC_PROG_CXX_CFLAGS_OPT macro has been introduced after these releases. Sebastian
Re: [PATCH] audo-detect and use -moutline-atomics compilation flag for aarch64
> I will investigate the autoconf issue and send you a patch specific to PG10 and 9. > Most likely the PGAC_PROG_CXX_CFLAGS_OPT macro has been introduced after these releases. The above macro has been added in March 2018: https://github.com/postgres/postgres/commit/6869b4f2584787d9e4cefaab8a4bae1ecbe63766 the patch also adds CXXFLAGS, and so before this patch there is no need to detect CXXFLAGS. Please see attached a patch tested on Ubuntu 20.04 on the following branches: remotes/origin/REL9_6_STABLE remotes/origin/REL_10_STABLE I checked that REL_11_STABLE has the CXXFLAGS and the above macro. Thanks, Sebastian
Вложения
Re: [PATCH] audo-detect and use -moutline-atomics compilation flag for aarch64
Thanks Christoph for the quick turn-around. I see that builds with -moutline-atomics succeeded for Focal arm64 builds for all the branches: https://pgdgbuild.dus.dg-i.net/view/PostgreSQL/job/postgresql-9.6-binaries/architecture=arm64,distribution=focal/88/ https://pgdgbuild.dus.dg-i.net/job/postgresql-10-binaries/77/architecture=arm64,distribution=focal/ https://pgdgbuild.dus.dg-i.net/view/PostgreSQL/job/postgresql-11-binaries/architecture=arm64,distribution=focal/686/ https://pgdgbuild.dus.dg-i.net/view/PostgreSQL/job/postgresql-12-binaries/architecture=arm64,distribution=focal/875/ https://pgdgbuild.dus.dg-i.net/view/PostgreSQL/job/postgresql-13-binaries/architecture=arm64,distribution=focal/858/ https://pgdgbuild.dus.dg-i.net/view/PostgreSQL/job/postgresql-14-binaries/architecture=arm64,distribution=focal/651/ I see that the patch is still missing on branch 15 that is the development branch: https://pgdgbuild.dus.dg-i.net/view/PostgreSQL/job/postgresql-15-binaries/architecture=amd64,distribution=focal/345/ Maybe we need to add the patch to that branch as well? How are the above binaries distributed to users? I see for instance that pgsql 14 binaries have last been updated in 30-Sep-2021: https://apt.postgresql.org/pub/repos/apt/dists/focal-pgdg/14/binary-arm64/ Is there another step needed to make those binaries available through the PostgreSQL Apt Repository ( https://www.postgresql.org/download/linux/ubuntu/) Does Canonical rely on these builds to update the Focal PostgreSQL packages? Thanks, Sebastian
Re: [PATCH] audo-detect and use -moutline-atomics compilation flag for aarch64
Thanks Christoph for the quick turn-around.
I see that builds with -moutline-atomics succeeded for Focal arm64 builds for all the branches:
https://pgdgbuild.dus.dg-i.net/view/PostgreSQL/job/postgresql-9.6-binaries/architecture=arm64,distribution=focal/88/
https://pgdgbuild.dus.dg-i.net/job/postgresql-10-binaries/77/architecture=arm64,distribution=focal/
https://pgdgbuild.dus.dg-i.net/view/PostgreSQL/job/postgresql-11-binaries/architecture=arm64,distribution=focal/686/
https://pgdgbuild.dus.dg-i.net/view/PostgreSQL/job/postgresql-12-binaries/architecture=arm64,distribution=focal/875/
https://pgdgbuild.dus.dg-i.net/view/PostgreSQL/job/postgresql-13-binaries/architecture=arm64,distribution=focal/858/
https://pgdgbuild.dus.dg-i.net/view/PostgreSQL/job/postgresql-14-binaries/architecture=arm64,distribution=focal/651/
I see that the patch is still missing on branch 15 that is the development branch:
https://pgdgbuild.dus.dg-i.net/view/PostgreSQL/job/postgresql-15-binaries/architecture=amd64,distribution=focal/345/
Maybe we need to add the patch to that branch as well?
How are the above binaries distributed to users?
I see for instance that pgsql 14 binaries have last been updated in 30-Sep-2021:
https://apt.postgresql.org/pub/repos/apt/dists/focal-pgdg/14/binary-arm64/
Is there another step needed to make those binaries available through the PostgreSQL Apt Repository ( https://www.postgresql.org/download/linux/ubuntu/ )
Does Canonical rely on these builds to update the Focal PostgreSQL packages?
Thanks,
Sebastian
https://apt.postgresql.org/pub/repos/apt/pool/main/p/postgresql-14/
Re: [PATCH] audo-detect and use -moutline-atomics compilation flag for aarch64
> The packages should be accessible via the 'focal-pgdg-testing' distribution, where they initially land before eventual promotion to the 'focal-pgdg' distribution.
Thanks Aaron for the explanation.
Re: [PATCH] audo-detect and use -moutline-atomics compilation flag for aarch64
Hi, Re: Pop, Sebastian > I see that the patch is still missing on branch 15 that is the development branch: > https://pgdgbuild.dus.dg-i.net/view/PostgreSQL/job/postgresql-15-binaries/architecture=amd64,distribution=focal/345/ That's amd64. https://salsa.debian.org/postgresql/postgresql/-/blob/15/debian/patches/focal-arm64-outline-atomics > Does Canonical rely on these builds to update the Focal PostgreSQL packages? I don't know what Ubuntu's policy on updating released packages is. Christoph
Re: [PATCH] audo-detect and use -moutline-atomics compilation flag for aarch64
> That's amd64. > https://salsa.debian.org/postgresql/postgresql/-/blob/15/debian/patches/focal-arm64-outline-atomics Sorry about that typo. Arm64 branch 15 looks good. Thanks Christoph. > I don't know what Ubuntu's policy on updating released packages is. I took the issue to Canonical and I will follow up with them to update the PostgreSQL packages. Thanks, Sebastian