Обсуждение: Download failures on apt archive

Поиск
Список
Период
Сортировка

Download failures on apt archive

От
James Coleman
Дата:
Hello,

We've been trying to do direct downloads from the apt archive (e.g.
[1]) and many of the files fail to download with an AccessDenied error
with XML from S3 like:

<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>FRN2GJREB814S5FD</RequestId>
<HostId>
47zx5Bji8uWyhDlF+59x0dJhK/8+8TInBCVeQqwhIZecGE2pZfPlFZEIO2/aVtam+My7h/WY9ls=
</HostId>
</Error>

For example [2] fails but for example [3] succeeds.

This seems to be repeatable by file across browsers and with my
co-workers, so I don't think it's just a client issue.

Thanks,
James Coleman

1: https://apt-archive.postgresql.org/pub/repos/apt/pool/main/p/postgresql-14/index.html
2: https://apt-archive.postgresql.org/pub/repos/apt/pool/main/p/postgresql-14/libpq-dev_14.7-1.pgdg110%2B1_amd64.deb
3.
https://apt-archive.postgresql.org/pub/repos/apt/pool/main/p/postgresql-14/postgresql-14_14.1-1.pgdg20.04%2B1_arm64.deb



Re: Download failures on apt archive

От
Bradford Boyle
Дата:
Hi James,

apt-archive.p.o is hosted via an AWS S3 bucket which doesn't include/generate
index lists (e.g, index.html) which is why [1] doesn't work. If you are
familiar with the structure of Debian repos, you can view the repo's index by
browsing the appropriate files directly. For example,

https://apt-archive.postgresql.org/pub/repos/apt/dists/${DISTRIBUTION}-pgdg-archive/InRelease

where distribution is bullseye, focal, etc. The "InRelease" file will contain
paths for various "Packages" files. You can view the list of packages by
replacing "InRelease" with the appropraite package path. For example

https://apt-archive.postgresql.org/pub/repos/apt/dists/${DISTRIBUTION}-pgdg-archive/main/binary-amd64/Packages

Using this, we can see why [2] isn't working for you -- there is no libpq-dev
in bullseye-pgdg-archive, the package is libpq5-dev.

Hope this helps somewhat.

--Bradford



Re: Download failures on apt archive

От
Christoph Berg
Дата:
Re: James Coleman
> Hello,
> 
> We've been trying to do direct downloads from the apt archive (e.g.
> [1]) and many of the files fail to download with an AccessDenied error
> with XML from S3 like:
> 
> <Error>
> <Code>AccessDenied</Code>

Hmm, admittedly I've never tried to recheck if all the package links
worked after the initial setup, so it's well possible something went
wrong in the years since then.

I'll do some QA there tomorrow. Thanks for reporting!

Re: Bradford Boyle
> Hi James,
> 
> apt-archive.p.o is hosted via an AWS S3 bucket which doesn't include/generate
> index lists (e.g, index.html) which is why [1] doesn't work.

There are manually uploaded index.html files generated from the
database mentioned earlier.

> Using this, we can see why [2] isn't working for you -- there is no libpq-dev
> in bullseye-pgdg-archive, the package is libpq5-dev.

libpq-dev is correct, the 5 is only in libpq5.deb.

Christoph



Re: Download failures on apt archive

От
Bradford Boyle
Дата:
> There are manually uploaded index.html files generated from the
> database mentioned earlier.

Yep, I see that now. I should have read the code for
generate-archive-lists more closely before replying.

> libpq-dev is correct, the 5 is only in libpq5.deb.

Right again -- I don't know what I was doing wrong with my grep commands
this morning but I'll blame it on a lack of sleep and coffee.

It looks like the links on [1] are incorrect -- it gives [2] as the link
for libpq-dev_14.7-1.pgdg110+1_amd64.deb but looking at the Package
file, I think the correct link is [3] (i.e., pool/main vs pool/14).

I think the issue has to do with set_package_component in
generate-pgdg-source moving lib packages to a different component when
the source postgresql version is older than PG_MAIN_VERSION; it looks
generate-archive-lists assumes component is always main though.

--Bradford


[1]: https://apt-archive.postgresql.org/pub/repos/apt/pool/main/p/postgresql-14/index.html
[2]: https://apt-archive.postgresql.org/pub/repos/apt/pool/main/p/postgresql-14/libpq-dev_14.7-1.pgdg110%2B1_amd64.deb
[3]: https://apt-archive.postgresql.org/pub/repos/apt/pool/14/p/postgresql-14/libpq-dev_14.7-1.pgdg110%2B1_amd64.deb



Re: Download failures on apt archive

От
Christoph Berg
Дата:
Re: Bradford Boyle
> It looks like the links on [1] are incorrect -- it gives [2] as the link
> for libpq-dev_14.7-1.pgdg110+1_amd64.deb but looking at the Package
> file, I think the correct link is [3] (i.e., pool/main vs pool/14).

Ah right, thanks for spotting that, I had missed that when trying to
debugging the issue earlier today. I remember now that I had omitted
the extra pool locations, but I missed that they are still used.

So in summary, the files are all there, but the path in the
referencing web page is wrong. Will fix.

> I think the issue has to do with set_package_component in
> generate-pgdg-source moving lib packages to a different component when
> the source postgresql version is older than PG_MAIN_VERSION; it looks
> generate-archive-lists assumes component is always main though.

Yeah I had the idea that for the archive, I could just ignore the
changed component and shovel everything into "main", but of course in
that case the files would have to be uploaded to that location.

Christoph



Re: Download failures on apt archive

От
Christoph Berg
Дата:
Re: To Bradford Boyle
> So in summary, the files are all there, but the path in the
> referencing web page is wrong. Will fix.

I managed to fix the problems, the index listings in package
directories like
https://apt-archive.postgresql.org/pub/repos/apt/pool/main/p/postgresql-14/index.html
will now also reference files located in other directories like pool/14/p/...

James, Bradford: Thanks for spotting and debugging!

Christoph



Re: Download failures on apt archive

От
James Coleman
Дата:
On Fri, May 19, 2023 at 11:31 AM Christoph Berg <myon@debian.org> wrote:
>
> Re: To Bradford Boyle
> > So in summary, the files are all there, but the path in the
> > referencing web page is wrong. Will fix.
>
> I managed to fix the problems, the index listings in package
> directories like
> https://apt-archive.postgresql.org/pub/repos/apt/pool/main/p/postgresql-14/index.html
> will now also reference files located in other directories like pool/14/p/...
>
> James, Bradford: Thanks for spotting and debugging!

Thanks for fixing this!

James