On Mon, Feb 23, 2015 at 5:59 AM, Andrew Gierth
<andrew@tao11.riddles.org.uk> wrote:
>>>>>> "Tomas" == Tomas Vondra <tomas.vondra@2ndquadrant.com> writes:
> Tomas> Interesting, but I think Gavin was asking about how much
> Tomas> variation was there for each tested case (e.g. query executed on
> Tomas> the same code / dataset). And in those cases the padding /
> Tomas> alignment won't change, and thus the effects you describe won't
> Tomas> influence the results, no?
>
> My point is exactly the fact that since the result is not affected, this
> variation between runs of the same code is of no real relevance to the
> question of whether a given change in performance can properly be
> attributed to a patch.
>
> Put it this way: suppose I have a test that when run repeatedly with no
> code changes takes 6.10s (s=0.025s), and I apply a patch that changes
> that to 6.26s (s=0.025s). Did the patch have an impact on performance?
>
> Now suppose that instead of applying the patch I insert random amounts
> of padding in an unused function and find that my same test now takes a
> mean of 6.20s (s=0.058s) when I take the best timing for each padding
> size and calculate stats across sizes. Now it looks obvious that the
> actual code of the patch probably wasn't responsible for any change...
>
> The numbers used here aren't theoretical; they are obtained by testing a
> single query - "select * from d_flt order by v offset 10000000" where
> d_flt contains 5 million float8 values - over 990 times with 33
> different random padding sizes (uniform in 0-32767). Here's a scatter
> plot, with 3 runs of each padding size so you can see the repeatability:
> http://tinyurl.com/op9qg8a
Neat chart.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company