On Mon, Feb 10, 2003 at 06:48:17PM -0500, Christopher Browne wrote:
> Unfortuantely, a lot of the older TPC specs have proven susceptible to
> "hacks" where the data proves to be almost totally non-interdependent,
> so that by throwing extra CPUs and extra disks at the benchmarks, you
> can get very nearly linear scalability. Based on history, I'm
> skeptical that the "hacks" for more recent benchmarks just haven't
> been found yet...
Sure. And isn't that what every database admin looks to do the
moment performance starts to crawl? If everyone can use the same
hacks, then you just find out who works the best witht he hacks.
Do the TPC tests tell us anything about how the DBMS will work with
application x? No. But they are a clear, well-understood standard
with well-known deficiencies and advantages.
The question is not whether a test reveals actual utility of the
system for its intended use. The question is merely whether good
performance in the set of tests is a predictor for good performance
in other areas. TPC-B and TPC-C appear to be fairly consistent
predictors of good OLTP systems. They don't give you any kind of
realistic idea of how the system will actually perform, though. (In
this respect, the TPC tests are no different from the
LSAT/GRE/MCAT/&c. exams. Does the LSAT test things necessary for law
school? Nobody knows, and the LSAC doesn't care. The test is merely
a good predictor of high scores at the time of law school
graduation.)
A
--
----
Andrew Sullivan 204-4141 Yonge Street
Liberty RMS Toronto, Ontario Canada
<andrew@libertyrms.info> M2P 2A8
+1 416 646 3304 x110