URL for Industry Standard Benchmarks

Audience: All

Date: February 4, 2003

Here are several URL's containing industry standard benchmarks.

ISV Benchmarks

CPU Intensive

Data Warehouse

Transaction Database

Java

Graphics

Web

Current and Historic pSeries & RS/6000

Relative Hardware Performance by Vendor

I've found that you have to be careful when using benchmarks. Here are a few guidelines that I use:

1. The best benchmark is running your application on the same server you intend to run in production.
2. In the absence of doing your own benchmark, select a benchmark that reflects your workload. Don't assume that a fast CPU means fast I/O.
3. Benchmarks are better used for relative comparisons, not absolute sizings. There are several reasons for this. First, benchmarks are so highly tuned that you probably never achieve the benchmark results in production. Second, the motivation for many ISV benchmarks is to show the maximum number of users. As a consequence, ISV benchmark workloads tend to be a "light" relative to a production workloads. Third, every company customizes their software differently. I've seen the same application on the same server with the same number of users behave totally differently at different companies due to customization.
4. For sizings, use ISV recommendations. Use references when possible.
5. Finally, in many cases, it may be cheaper to just oversize the server instead of running a benchmark. The cost of doing a benchmark may be more than the cost of the hardware. For example, the last benchmark I ran cost $200,000. Also, from a business perspective, if the application you intend to run saves $1M per month, its almost a certainty that it is cheaper to oversize, rather than delay the rollout to run the benchmark. You can recover excess capacity with either WLM or partitioning.