Re: performance-test farm
От | Kevin Grittner |
---|---|
Тема | Re: performance-test farm |
Дата | |
Msg-id | 4DCAC593020000250003D5A9@gw.wicourts.gov обсуждение исходный текст |
Ответ на | Re: performance-test farm (Tomas Vondra <tv@fuzzy.cz>) |
Ответы |
Re: performance-test farm
(Tomas Vondra <tv@fuzzy.cz>)
Re: performance-test farm (Andrew Dunstan <andrew@dunslane.net>) |
Список | pgsql-hackers |
Tomas Vondra <tv@fuzzy.cz> wrote: > Dne 11.5.2011 23:41, Kevin Grittner napsal(a): >> Tomas Vondra <tv@fuzzy.cz> wrote: >> >>> 1) Is there something that might serve as a model? >> >> I've been assuming that we would use the PostgreSQL Buildfarm as >> a model. >> >> http://buildfarm.postgresql.org/ > > Yes, I was thinking about that too, but > > 1) A buildfarm used for regular building / unit testing IMHO may > not be the right place to do performance testing (not sure how > isolated the benchmarks can be etc.). I'm not saying that we should use the existing buildfarm, or expect current buildfarm machines to support this; just that the pattern of people volunteering hardware in a similar way would be good. > 2) Not sure how open this might be for the developers (if they > could issue their own builds etc.). I haven't done it, but I understand that you can create a "local" buildfarm instance which isn't reporting its results. Again, something similar might be good. > 3) If this should be part of the current buildfarm, then I'm > afraid I can't do much about it. Not part of the current buildfarm; just using a similar overall pattern. Others may have different ideas; I'm just speaking for myself here about what seems like a good idea to me. >>> 2) How would you use it? What procedure would you expect? >> >> People who had suitable test environments could sign up to >> periodically build and performance test using the predetermined >> test suite, and report results back for a consolidated status >> display. That would spot regressions. > > So it would be a 'distributed farm'? Not sure it that's a good > idea, as to get reliable benchmark results you need a proper > environment (not influenced by other jobs, changes of hw etc.). Yeah, accurate benchmarking is not easy. We would have to make sure people understood that the machine should be dedicated to the benchmark while it is running, which is not a requirement for the buildfarm. Maybe provide some way to annotate HW or OS changes? So if one machine goes to a new kernel and performance changes radically, but other machines which didn't change their kernel continue on a level graph, we'd know to suspect the kernel rather than some change in PostgreSQL code. -Kevin
В списке pgsql-hackers по дате отправления: