Обсуждение: CI speed improvements for FreeBSD

Поиск
Список
Период
Сортировка

CI speed improvements for FreeBSD

От
Thomas Munro
Дата:
Hi,

Here are a couple of changes that got FreeBSD down to 4:29 total, 2:40
in test_world in my last run (over 2x speedup), using a RAM disk
backed by a swap partition, and more CPUs.  It's still a regular UFS
file system but FreeBSD is not as good at avoiding I/O around short
lived files and directories as Linux: it can get hung up on a bunch of
synchronous I/O, and also flushes disk caches for those writes,
without an off switch.

I don't know about Windows, but I suspect the same applies there, ie
synchronous I/O blocking system calls around our blizzard of file
creations and unlinks.  Anyone know how to try it?
Вложения

Re: CI speed improvements for FreeBSD

От
Thomas Munro
Дата:
And after adding this to the commitfest, here's the first cfbot run.
The gain was due to "test_world" which shows a greater-than-2x speedup
(~4:30 -> ~2:08) from 2x CPUs.  That is nice for humans who want the
answer as soon as possible, but note that the resource usage cost
might go up because of the non-parallel parts now wasting more idle
CPUs: git clone, meson configure etc (as they do on every platform).

https://cirrus-ci.com/build/6060109692928000

Re: CI speed improvements for FreeBSD

От
Maxim Orlov
Дата:
Hi!

I looked at the changes and I liked them.  Here are my thoughts:

0001:
1. I think, this is a good idea to use RAM.  Since, it's still a UFS, and we lose nothing in terms of testing, but win in speed significantly.
2. Change from "swapoff -a || true" to "swapoff -a" is legit in my view, since it's better to explicitly fail than silent any possible problem.
3. Man says that lowercase suffixes should be used for the mdconfig. But in fact, you can use either lowercase or an appercase. Yep, it's in
   the mdconfig.c: "else if (*p == 'g' || *p == 'G')".

0002:
1. The resource usage should be a bit higher, this is for sure.  But, if I'm not missing something, not drastically. Anyway, I do not know
   how to measure this increase to get concrete values.
2. And think of a potential benefits of increasing the number of test jobs: more concurrent processes, more interactions, better test coverage.

Here are my runs:
FreeBSD @master
https://cirrus-ci.com/task/4934701194936320
Run test_world 05:56

FreeBSD @master + 0001
https://cirrus-ci.com/task/5921385306914816
Run test_world 05:06

FreeBSD @master + 0001, + 0002
https://cirrus-ci.com/task/5635288945393664
Run test_world 02:20

For comparison
Debian @master
https://cirrus-ci.com/task/5143705577848832
Run test_world 02:38

In the overall, I consider this changes useful.  CI run faster, with better test coverage in exchange for presumably slight increase 
in resource usage, but I don't think this increase should be significant.

--
Best regards,
Maxim Orlov.

Re: CI speed improvements for FreeBSD

От
Thomas Munro
Дата:
On Wed, Mar 13, 2024 at 4:50 AM Maxim Orlov <orlovmg@gmail.com> wrote:
> I looked at the changes and I liked them.  Here are my thoughts:

Thanks for looking!  Pushed.