Hi,
Marc G. Fournier wrote:
>> Yes, now it does ~5 pages per second, thus the whole mirroring process
>> will probably complete in under an hour.
>
> Does the mirroring process mirror *all* files, or just those that have
> changed? Most of the files on the site *should* be static, no? What
> consititutes the bulk of the pages? I would think the various versions
> docs, no? Which are static ...
All files except docs, since there are dynamic parts like survey and
particularly gborg stats on all pages. But there are not very many pages here.
As for the docs, there is version without user comments which will not need to
be mirrored unless the design changes or the docs for the newer version replace
the older docs. The version with user comments will need to be mirrored every
time, obviously. Though we may consider removing comment form from docs for the
older versions and ease this requirement.
Of course one can add Last-Modified header and 304 response to the pages, but
there still will be a request to make.
> BTW, does this script you are working on have any access controls, or
> does it allow *everyone* to mirror the site?
If you mean whether anyone will be able to access the script via web then the
answer is "no", it will be run from console. If the question is different then I
don't get it.