Обсуждение: apt.postgresql.org repo via https will fail will some users starting 2021-10-01
apt.postgresql.org repo via https will fail will some users starting 2021-10-01
От
Stefan Huehner
Дата:
Hello, sending this here as looks like https://apt.postgresql.org is affected by this so this could trigger some support/user questions. Note this only (!) happens when using https:// in sources.list for the pgdg repo. Benefit of that is debatable (see recent debian-devel discussion) but i would not be surprised if some/many people use it. Trigger: https://community.letsencrypt.org/t/openssl-client-compatibility-changes-for-let-s-encrypt-certificates/143816 End of this month some CA cert will expire related to Let's Encrypt which will trigger an bug in clients using old openssl/gnutls. apt is using gnutls backend and at least the version in Ubuntu <= 18.04 are affected and "apt update" will already fail forpeople starting that date. Note that canonical is working in patching gnutls so if that finishes in time and (!) if people update before that date allgood. If not they will get error similar to: Err:9 https://apt.postgresql.org/pub/repos/apt focal-pgdg Release Certificate verification failed: The certificate is NOT trusted. The revocation or OCSP data are old and have been superseded.The certificate chain uses expired certificate. Could not handshake: Error in the certificate verification. [IP:87.238.57.227 443] Can be triggered today i.e. with: faketime "2021-10-01" apt update Ideas: - Do nothing apt.postgresql suggest http:// in the instructions - Some on the website - Think on reconfiguring certbot/Let's Encrypt on the server to switch to the alternative chain (avoiding this bug but breakingcompatibility with old Android - Raise as bug to debian also (against openssl/gnutls) to maybe patch both in stable also to avoid this ? - Not sure if that is a interesting/acceptable material for stable/old-stable? Please CC me on replies i'm not on the list Stefan
Re: apt.postgresql.org repo via https will fail will some users starting 2021-10-01
От
Christoph Berg
Дата:
Re: Stefan Huehner > sending this here as looks like https://apt.postgresql.org is affected by this so this could trigger some support/userquestions. > > Note this only (!) happens when using https:// in sources.list for the pgdg repo. Hi, thanks for sharing this. We aren't advertising https:// for apt.postgresql.org anywhere, but the download instructions tell users to "wget" the repository key from https://www.postgresql.org, so we are at least somewhat affected. (wget is using gnutls at least in unstable.) > Ideas: > - Do nothing apt.postgresql suggest http:// in the instructions > - Some on the website > - Think on reconfiguring certbot/Let's Encrypt on the server to switch to the alternative chain (avoiding this bug butbreaking compatibility with old Android That's probably rather the ca-certificates package? > - Raise as bug to debian also (against openssl/gnutls) to maybe patch both in stable also to avoid this ? > - Not sure if that is a interesting/acceptable material for stable/old-stable? If stretch/buster/bullseye are affected, these should be fixed, yes. Though none of this is material for the PostgreSQL packages, can you raise the issue with the LTS team? Christoph
Re: apt.postgresql.org repo via https will fail will some users starting 2021-10-01
От
Stefan Huehner
Дата:
On Thu, Sep 09, 2021 at 02:33:49PM +0200, Christoph Berg wrote: > Re: Stefan Huehner > > sending this here as looks like https://apt.postgresql.org is affected by this so this could trigger some support/userquestions. > > > > Note this only (!) happens when using https:// in sources.list for the pgdg repo. > > Hi, > > thanks for sharing this. > > We aren't advertising https:// for apt.postgresql.org anywhere, but > the download instructions tell users to "wget" the repository key from > https://www.postgresql.org, so we are at least somewhat affected. > (wget is using gnutls at least in unstable.) > > > Ideas: > > - Do nothing apt.postgresql suggest http:// in the instructions > > - Some on the website > > - Think on reconfiguring certbot/Let's Encrypt on the server to switch to the alternative chain (avoiding this bug butbreaking compatibility with old Android > > That's probably rather the ca-certificates package? Not in this case, i know a bit confusing. That upstream article has more details: https://community.letsencrypt.org/t/openssl-client-compatibility-changes-for-let-s-encrypt-certificates/143816 Part: How to support older OpenSSL versions In (not so) short: ca-certificates is fine to have trust anchor for Lets Encrypt. However not everybody directly trust Let's Encrypt (missing entry in their equivalent of ca-certificates (i.e. old Android). To keep those other clients supported they employed a bit of a trick which has an 'expired root certificates' in the chainfrom your server-cert to their root. At the same time there is 2nd valid path. But old version of software (openssl,gnutls)just stop + fail on seeing 'expired'. Best they could do if offer server owner (certbot parameter when requesting ssl certificate to select): a.) Default chain (compatible still with old android) but triggering this bug b.) Alternative chain (ignore old android) but keep compatible with old openssl/gnutls That link goes into much more detail but hopefully now clearer. That is also why i raised this here as a choice for apt.postgresql.org hosting (if you think it's a useful workaround) > > > - Raise as bug to debian also (against openssl/gnutls) to maybe patch both in stable also to avoid this ? > > - Not sure if that is a interesting/acceptable material for stable/old-stable? > > If stretch/buster/bullseye are affected, these should be fixed, yes. > > Though none of this is material for the PostgreSQL packages, can you > raise the issue with the LTS team? Will raise there. Hopefuly above also clarified why i sent that here (not about any PostgreSQL package, but apt.postgresql.org server admintopic). Stefan
Re: apt.postgresql.org repo via https will fail will some users starting 2021-10-01
От
Christoph Berg
Дата:
Re: Stefan Huehner > > > - Some on the website > > > - Think on reconfiguring certbot/Let's Encrypt on the server to switch to the alternative chain (avoiding this bugbut breaking compatibility with old Android > > > > That's probably rather the ca-certificates package? > > Not in this case, i know a bit confusing. > That upstream article has more details: > https://community.letsencrypt.org/t/openssl-client-compatibility-changes-for-let-s-encrypt-certificates/143816 > Part: How to support older OpenSSL versions > > In (not so) short: ca-certificates is fine to have trust anchor for Lets Encrypt. > However not everybody directly trust Let's Encrypt (missing entry in their equivalent of ca-certificates (i.e. old Android). > > To keep those other clients supported they employed a bit of a trick which has an 'expired root certificates' in the chainfrom your server-cert to their root. At the same time there is 2nd valid path. But old version of software (openssl,gnutls)just stop + fail on seeing 'expired'. > > Best they could do if offer server owner (certbot parameter when requesting ssl certificate to select): Ah, I thought you meant the end-users servers running PostgreSQL when you said "server". For changing the webservers, we'd need to get pginfra on board, Cc'ed now. Christoph
Re: apt.postgresql.org repo via https will fail will some users starting 2021-10-01
От
Stefan Huehner
Дата:
On Thu, Sep 09, 2021 at 05:33:51PM +0200, Christoph Berg wrote: > Re: Stefan Huehner > > > > - Some on the website > > > > - Think on reconfiguring certbot/Let's Encrypt on the server to switch to the alternative chain (avoiding this bugbut breaking compatibility with old Android > > > > > > That's probably rather the ca-certificates package? > > > > Not in this case, i know a bit confusing. > > That upstream article has more details: > > https://community.letsencrypt.org/t/openssl-client-compatibility-changes-for-let-s-encrypt-certificates/143816 > > Part: How to support older OpenSSL versions > > > > In (not so) short: ca-certificates is fine to have trust anchor for Lets Encrypt. > > However not everybody directly trust Let's Encrypt (missing entry in their equivalent of ca-certificates (i.e. old Android). > > > > To keep those other clients supported they employed a bit of a trick which has an 'expired root certificates' in thechain from your server-cert to their root. At the same time there is 2nd valid path. But old version of software (openssl,gnutls)just stop + fail on seeing 'expired'. > > > > Best they could do if offer server owner (certbot parameter when requesting ssl certificate to select): > > Ah, I thought you meant the end-users servers running PostgreSQL when > you said "server". Sorry for the confusion. But now thinking they could be affected (but special cases only) - There may be a case for running PostgreSQL instances - As this affects any 'client' using above older libraries - libpq linked with old ssl, connecting via SSL to remote pg having Let's Encrypt and client validating certificate (verify-ca,verify-full) - or outgoing connections using fdw Note: This is just me trying to construct a flow which might fail. Again kind of 'info for supporting users' as bug is libssl we link against only. Maybe also interesting to spread to the yum-side of things (Devrim?) to check thei rpm using packaging/distros. But also not sure if that is common enough even to make a big topic out of... > > For changing the webservers, we'd need to get pginfra on board, Cc'ed > now. For the sysadmins: - 'changing' would avoid the bug described here + not for debian https://lists.debian.org/debian-lts/2021/09/msg00008.html for people running old distros (no fix, or not updated, ...) - But it will break using the webserver with other clients (i.e. older android) - Need to pick the 'smaller problem' based on the concrete site and its users/clients - This is generic Let's Encrypt topic (not just apt.postgresql.org host) - Above letsencrypt link has bigger explanation + all the background Stefan
Re: apt.postgresql.org repo via https will fail will some users starting 2021-10-01
От
Christoph Berg
Дата:
Re: Stefan Huehner > But now thinking they could be affected (but special cases only) > - There may be a case for running PostgreSQL instances > - As this affects any 'client' using above older libraries On stretch and later, PG is linked against libssl1.1 which is safe, so we are not affected. (jessie is EOL.) Christoph
Re: apt.postgresql.org repo via https will fail will some users starting 2021-10-01
От
Magnus Hagander
Дата:
On Wed, Sep 8, 2021 at 6:42 PM Stefan Huehner <stefan@huehner.org> wrote: > > Hello, > > sending this here as looks like https://apt.postgresql.org is affected by this so this could trigger some support/userquestions. > > Note this only (!) happens when using https:// in sources.list for the pgdg repo. > > Benefit of that is debatable (see recent debian-devel discussion) but i would not be surprised if some/many people useit. > > Trigger: > https://community.letsencrypt.org/t/openssl-client-compatibility-changes-for-let-s-encrypt-certificates/143816 > > End of this month some CA cert will expire related to Let's Encrypt which will trigger an bug in clients using old openssl/gnutls. > > apt is using gnutls backend and at least the version in Ubuntu <= 18.04 are affected and "apt update" will already failfor people starting that date. > > Note that canonical is working in patching gnutls so if that finishes in time and (!) if people update before that dateall good. > > If not they will get error similar to: > Err:9 https://apt.postgresql.org/pub/repos/apt focal-pgdg Release > Certificate verification failed: The certificate is NOT trusted. The revocation or OCSP data are old and have been superseded.The certificate chain uses expired certificate. Could not handshake: Error in the certificate verification. [IP:87.238.57.227 443] > > Can be triggered today i.e. with: > > faketime "2021-10-01" apt update > > Ideas: > - Do nothing apt.postgresql suggest http:// in the instructions > - Some on the website > - Think on reconfiguring certbot/Let's Encrypt on the server to switch to the alternative chain (avoiding this bug butbreaking compatibility with old Android > > - Raise as bug to debian also (against openssl/gnutls) to maybe patch both in stable also to avoid this ? > - Not sure if that is a interesting/acceptable material for stable/old-stable? Hi! We've started looking into what can and should be done on the infra side to see if we can get this working. One question though. In my attempts to reproduce, it seems that *wget* on Ubuntu 18.04 has no problem with the current chain, just apt-get, does that match with your testing? So if one follows our instructions of getting the gpg key with https but the actual repo with http, it never actually presents a problem? That's not saying we don't need to do anything about it, just to reconfirm our tests. For example, this appears to also break RedHat 6 as well... -- Magnus Hagander Me: https://www.hagander.net/ Work: https://www.redpill-linpro.com/
Re: apt.postgresql.org repo via https will fail will some users starting 2021-10-01
От
Stefan Huehner
Дата:
On Tue, Sep 14, 2021 at 04:33:01PM +0200, Magnus Hagander wrote: > On Wed, Sep 8, 2021 at 6:42 PM Stefan Huehner <stefan@huehner.org> wrote: > > > > Hello, > > > > sending this here as looks like https://apt.postgresql.org is affected by this so this could trigger some support/userquestions. > > > > Note this only (!) happens when using https:// in sources.list for the pgdg repo. > > > > Benefit of that is debatable (see recent debian-devel discussion) but i would not be surprised if some/many people useit. > > > > Trigger: > > https://community.letsencrypt.org/t/openssl-client-compatibility-changes-for-let-s-encrypt-certificates/143816 > > > > End of this month some CA cert will expire related to Let's Encrypt which will trigger an bug in clients using old openssl/gnutls. > > > > apt is using gnutls backend and at least the version in Ubuntu <= 18.04 are affected and "apt update" will already failfor people starting that date. > > > > Note that canonical is working in patching gnutls so if that finishes in time and (!) if people update before that dateall good. > > > > If not they will get error similar to: > > Err:9 https://apt.postgresql.org/pub/repos/apt focal-pgdg Release > > Certificate verification failed: The certificate is NOT trusted. The revocation or OCSP data are old and have beensuperseded. The certificate chain uses expired certificate. Could not handshake: Error in the certificate verification.[IP: 87.238.57.227 443] > > > > Can be triggered today i.e. with: > > > > faketime "2021-10-01" apt update > > > > Ideas: > > - Do nothing apt.postgresql suggest http:// in the instructions > > - Some on the website > > - Think on reconfiguring certbot/Let's Encrypt on the server to switch to the alternative chain (avoiding this bug butbreaking compatibility with old Android > > > > - Raise as bug to debian also (against openssl/gnutls) to maybe patch both in stable also to avoid this ? > > - Not sure if that is a interesting/acceptable material for stable/old-stable? > > Hi! > > We've started looking into what can and should be done on the infra > side to see if we can get this working. > > One question though. In my attempts to reproduce, it seems that *wget* > on Ubuntu 18.04 has no problem with the current chain, just apt-get, > does that match with your testing? So if one follows our instructions > of getting the gpg key with https but the actual repo with http, it > never actually presents a problem? Hi Magnus, sorry for the delay i was out travelling. I just checked and and wget for Ubuntu 18.04/bionic is linked to libssl1.1 not having the problem in the first place. Which explains why is it not affected here. I think wget can have both gnutls+openssl backend so need to see which is usedper distro/release (As can change) # apt show wget | grep Depends Depends: libc6 (>= 2.17), libidn2-0 (>= 0.6), libpcre3, libpsl5 (>= 0.16.0), libssl1.1 (>= 1.1.0), libuuid1 (>= 2.16) Also for apt itself in 18.04 Ubuntu has backported the fix into gnutls30 and it just reached the normal update repository. Package/version containing the fix is: libgnutsl30 3.5.18-1ubuntu1.5 So here if people install (non-security) updates they will be not affected. Or telling them "run system updates" if they get the bug is enough. Note for older Ubuntu 16.04+14.04 Ubuntu offers paid updates still which are not yet done /published for this issue but alsoin progress. For debian also fixes are moving along after i reached our to debian-lts list ass Christoph suggested earlier. > That's not saying we don't need to do anything about it, just to > reconfirm our tests. For example, this appears to also break RedHat 6 > as well... But good news is also impact is getting smaller with less distro/releases affected as some are getting fixed. Also with few combinations being affected maybe just some extra note/section in the wiki could be enough. wget http:// for affected + extra step to verify key fingerprint For other distros (RHEL,Centos,Suse, ...) do we have anyone having contact's there nearby here? I think there was yum.postgresql.org with Devrim active on that (but don't remember). As probably reaching out to them maybe they are also interested in just backporting the libary fixes on their side. Stefan > > -- > Magnus Hagander > Me: https://www.hagander.net/ > Work: https://www.redpill-linpro.com/