On 22/02/15 22:59, Andrew Gierth wrote:
>>>>>> "Gavin" == Gavin Flower <GavinFlower@archidevsys.co.nz> writes:
> Gavin> What are the standard deviations?
>
> Gavin> Do the arithmetic means change much if you exclude the 2 fastest
> Gavin> & 2 slowest?
>
> Gavin> How do the arithmetic means compare to their respective medians?
>
> Gavin> Essentially, how consistent are the results, or how great is the
> Gavin> noise? There may be better indicators than the ones I've
> Gavin> suggested above.
>
> This is all rather missing the point.
>
> The relevant metric is not how much noise is introduced between runs of
> the same code, but rather how much noise is introduced as a result of
> non-consequential changes to the code.
>
> I can get variations of several percent - easily more than three sigmas
> of the timing of repeated runs of unchanged code - in the time taken to
> sort a float8 column simply from introducing varying amounts of padding
> into the body of a function which is never called in the test. Clearly,
> the only possible effect here is that the changed memory addresses of
> functions must be resulting in different patterns of cache misses /
> cache replacements, or TLB misses, or similar low-level effects which
> have nothing to do with the code as such.
>
> (That this is a low-level alignment effect is supported by the fact that
> the performance changes are not monotonic in the size of the padding;
> adding more padding may cause either speedups or slowdowns.)
>
What is you have pointed out is 'obvious', in retrospect - but still,
far more extreme than I would have ever expected!
There well may be analogues of the story where a DP manager was
convinced to get faster hard drives, 50% faster than the old ones, to
speed up an overnight batch run. The program then took about 50% longer
to run. Turns out that previously the mainframe processed one block in
a little less time than the old hard drive took to rotate once. So the
new hard drives ended up doing two rotations per block read, so took
longer, despite being faster! (A modern low end Smart Phone, probably
has at least a thousand times more processing power than that poor old
Mainframe!)
Cheers,
Gavin