Обсуждение: [9.4 CF] Free VMs for Reviewers & Testers
Reviewer, Testers: As part of an ongoing effort to encourage patch review for the PostgreSQL project, we will be funding cloud servers for patch reviewers and testers who need them for CommitFests. That is, if you want to help with reviewing or testing a patch for a CommitFest, and don't have your own test server to use, the PostgreSQL project (via our funds at Software In the Public Interest) will pay for you to use a cloud server for the duration of your review/testing. So, if "I don't have anywhere to test it" was your excuse for not reviewing in the past, time to stop making excuses and start reviewing! Since these are cloud servers, they won't work well for performance testing. However, they will be excellent for testing replication. If you need one of these, please contact me to allocate a VM for you. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
On 07/09/2013 08:35 AM, Josh Berkus wrote: > Since these are cloud servers, they won't work well for performance > testing. I did some work on that a while ago, and found that I was able to get _astonishingly_ stable performance results out of EC2 EBS instances using provisioned IOPS volumes. Running them as "EBS Optimized" didn't seem to be required for the workloads I was testing on. What isn't made very clear in the EBS provisioned I/O docs is that for PIOPS volumes the provisioned I/O count is both a minimum and a limit; it's a *target*, not just a guaranteed minimum. This is really handy for testing. For a 500 iops volume I consistently got 500 +- 5 in pg_test_fsync. Real world performance was a bit more variable - my pgbench runs were stable within 5-10% over 5 minute runs with a forced CHECKPOINT first. They settled considerably over longer runs and bigger checkpoint_segments, though, and I see the same kind of short-term jitter in pgbench on real hardware. So - don't write cloud hosts off for benchmarking. With proper QoS they're actually really very good. While they're somewhat costly, being able to spawn them for a day's runs and destroy them at the end is quite handy, as is being able to easily automate their setup. For testing of logical changeset streaming replication / BDR performance I was using scripted sets of EC2 c1.medium instances with 500 piops 'io1' EBS volumes and found it to be an exceedingly useful way of measuring relative performance. Results were also pretty consistent across multiple launches of the same VM, so multiple people should be able to compare results obtained with different launches of the same VM type and configuration. These VMs aren't well suited to vertical scaling performance tests and pushing extremes, but they're really, really good for "what impact does this patch have on regular real-world workloads". Just remember to run pgbench from a different vm within the same availability zone (with appropriate security group setup) if you try to use EC2 for this sort of thing. Use of instance store or regular EBS volumes will get you performance that's absolutely all over the place, of course; only the provisioned I/O feature seems to be any good for perf testing. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
On Mon, Jul 8, 2013 at 7:25 PM, Craig Ringer <craig@2ndquadrant.com> wrote: > On 07/09/2013 08:35 AM, Josh Berkus wrote: >> Since these are cloud servers, they won't work well for performance >> testing. > > I did some work on that a while ago, and found that I was able to get > _astonishingly_ stable performance results out of EC2 EBS instances > using provisioned IOPS volumes. Running them as "EBS Optimized" didn't > seem to be required for the workloads I was testing on. My colleague, Greg Burek, has done similar measurements and has assessed an overall similar conclusion: the EBS PIOPS product delivers exactly what it says on the tin...even under random access. They can be striped with software-RAID. > These VMs aren't well suited to vertical scaling performance tests and > pushing extremes, but they're really, really good for "what impact does > this patch have on regular real-world workloads". The really, really big ones are useful even for pushing limits, such as cr1.8xlarge, with 32 CPUs and 244GiB memory. Current spot instance price (the heavily discounted "can die at any time" one) is $0.343/hr. Otherwise, it's 3.500/hr. Another instance offering that -- unlike the former -- I have yet to experience in any way at all is the high-storage ones, also the hs1.8xlarge, with 16CPU, 117GB RAM, and 24 instance-local rotational media of 2TB apiece. This is enough to deliver sequential reads measured in a couple of GB/s. I think this is a workhorse behind the AWS Redshift data warehousing offering. I don't think this one has spot pricing either: my guess is availability is low.
The really, really big ones are useful even for pushing limits, such
as cr1.8xlarge, with 32 CPUs and 244GiB memory. Current spot instance
price (the heavily discounted "can die at any time" one) is $0.343/hr.
Otherwise, it's 3.500/hr.
Just to keep in mind cpus are similar throttled:
One EC2 Compute Unit provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor. This is also the equivalent to an early-2006 1.7 GHz Xeon processor referenced in our original documentation.
Who knows what that does to memory bandwidth / context switches etc.
Jesper
On Tue, Jul 9, 2013 at 12:24 AM, Jesper Krogh <jesper@krogh.cc> wrote: > > The really, really big ones are useful even for pushing limits, such > as cr1.8xlarge, with 32 CPUs and 244GiB memory. Current spot instance > price (the heavily discounted "can die at any time" one) is $0.343/hr. > Otherwise, it's 3.500/hr. > > > Just to keep in mind cpus are similar throttled: > > One EC2 Compute Unit provides the equivalent CPU capacity of a 1.0-1.2 GHz > 2007 Opteron or 2007 Xeon processor. This is also the equivalent to an > early-2006 1.7 GHz Xeon processor referenced in our original documentation. This is only a statement of measurement (notably, it also is a metric that is SMP-processor-count-oblivious), for lack of a more sensible metric (certainly not clock cycles nor bogomips) in common use. > Who knows what that does to memory bandwidth / context switches etc. Virtualization adds complexity, that is true, but so does a new version of Linux or comparing across microprocessors, motherboards, or operating systems. I don't see a good reason to be off-put in the common cases, especially since I can't think of another way to produce such large machines on a short-term obligation basis. The advantages are probably diminished (except for testing virtualization overhead on common platforms) for smaller machines that can be located sans-virtualization more routinely.
On 07/08/2013 10:49 PM, Daniel Farina wrote: > On Mon, Jul 8, 2013 at 7:25 PM, Craig Ringer <craig@2ndquadrant.com> wrote: >> I did some work on that a while ago, and found that I was able to get >> _astonishingly_ stable performance results out of EC2 EBS instances >> using provisioned IOPS volumes. Running them as "EBS Optimized" didn't >> seem to be required for the workloads I was testing on. > The really, really big ones are useful even for pushing limits, such > as cr1.8xlarge, with 32 CPUs and 244GiB memory. Current spot instance > price (the heavily discounted "can die at any time" one) is $0.343/hr. > Otherwise, it's 3.500/hr. Yes, and for that reason, they aren't realistic for offering to reviewers for free, funded by the community. For real performance testing, we already have servers hosted in Portland which are real, non-shared servers. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com