gcocco software Home!
gcocco software, inc
Benchmark: SPEC®cpu2000

FAQ:
  1. What is SPECcpu2000 anyway?
  2. What does SPECcpu2000 measure?
  3. Why is SPECcpu2000 important?
  4. Which metric should I use, "base" or "peak?
  5. Should I buy a machine based on SPECcpu2000 numbers alone?

Machine Tested:

A.   Sun Microsystems Enterprise 420R

  1. SPECint2000 Results generated with "Generally Available" Compilers
  2. SPECfp2000 Results generated with "Generally Available" Compilers
  3. Tricks/Tips/Cautions
  4. Net/Net - The Performance you should expect!



Questions and Answers:
  1. What is SPECcpu2000 anyway?
    SPECcpu2000 is a set of 26 applications taken from different disciplines intended to provide a cross section of the basic computing capabilities of a vendors hardware and optimizing compiler. There are 12 that are fixed point arithmetic, most similar to commercial workloads like databases, webserving, etc.

    There are 14 floating point applications where the CPU time is heavily influenced by floating point calculations which are indicative of what you might find in the solver portion of an application like NASTRAN or ANSYS.

    The CINT2000 and CFP2000 benchmark suites are usually run separately. The metrics for CINT2000 are SPECint_base2000 and SPECint2000. The metrics for CFP2000 are SPECfp_base2000 and SPECfp2000.


    Back to FAQ

  2. What does SPECcpu2000 measure?
    SPECcpu2000 attempts to measure the "raw" CPU power of the Central Electronic Complex (CEC). It doesn't involve disk or network I/O. All data is held in memory or in some cases, entirely in cache.

    This is analogous to the horsepower rating for your automobile engine. Your engine may have a 500hp rating, while mine has a paltry 250hp rating. However, this doesn't mean your car will go twice as fast as mine or even 0 to 60 in half the time! What if your "car" weighs 5 times what mine weighs, for instance. Using any one metric is very dangerous.

    Besides, out on the road, the top speed of both cars is regulated by lots of other influences, like speed limits, the police, other traffic, road conditions, etc. Just having the "horsepower" to go 200+ mph, doesn't mean you will ever be able to get anywhere close to that speed, no less sustain that speed!

    The same holds true for CPU "horsepower" ratings. They are important, but only a small part of the story. For the rest of the story, you need to look to "higher level" benchmarks like SPECjbb2000, SPECweb99, TPC-C, and many, many others. With higher level benchmarks you start bringing in more components of the whole computer system, and start to get a true sense of how the machines will perform. That is why SPEC has so many different "application" level benchmarks in the first place.


    Back to FAQ

  3. Why is SPECcpu2000 important to you?
    Without the "raw" horsepower rating of a machine, how would we know who has the fastest processor? Does it matter? It does if the rest of the system takes advantage of it. It doesn't if some other system bottleneck causes the equivalent of a "regulator" which the CPU's performance can not overcome.


    Back to FAQ

  4. Which metric should I use, "base" or "peak?
    First let me digress about: Feedback Directed Optimization(FDO).
    FDO is a highly sophisticated compiler technology which is generally used only by software vendors who develop large server applications (like a database, webserver, etc). The computer code is compiled, and automatically augmented with trace points, allowing for the capture of dynamic logic flow through the application.

    The code is then re-compiled with the benefit of the "logic flow analysis", branch history and locality of reference information. It is optimally restructured to handle data similar to the data it was "trained" with. The performance gains from this procedure are quite good, but it is not an easy thing to setup, and most application programmers would not go through the trouble. Most programmers don't even turn "basic" optimization on!

    Both "base" and "peak" SPECcpu metrics allow for the use of feedback directed optimization. It was not always allowed and was somewhat controversial when adopted. The "pro" side is that you should let your compiler do all that it can and the "con" side is that the "normal user" will never use it, so it is not "representative" enough. Both of these answers are correct, to some degree.

    base: is defined as a set of 4 compiler options (at most 4) that can be applied equally across all 26 runs of the same type (C, C++, FORTRAN). This would imply these options are rather robust and couldn't be "tailored" or "patched" just for this benchmark.

    peak: allows any and all of the compiler options that you have at your disposal to be used to squeak out the last few milli-seconds from each run. It's really difficult to get the last few milli-seconds without the help of and the home phone numbers of the vendor's compiler group!

    For the record, I only use SPECint_base2000 and SPECfp_base2000 numbers for my modeling and forecasting work!

    Back to FAQ

  5. Should I buy a machine based on SPECcpu2000 numbers alone?
    Of course not! This is clearly an "IQ" Question. There are a whole set of other SPEC Benchmarks (as well as other industry standard benchmarks) to look at. Find one close to your "application" workload and try and use that!


    Back to FAQ | Back to top

A. Machine Tested: Sun Microsystems Enterprise 420R

System Sun Enterprise 420R
CPU (4) - UltraSPARC II
Clock Speed 450 MHz
Memory 4 GB

All Results quoted or displayed in this section have been
reviewed and accepted
by the OSG CPU Subcommittee of the
Standard Performance Evaluation Corporation (SPEC®).

A hyper-link is provided on each result to the
"SPEC full disclosure report" which is located on the
OSG CPU2000 Web-site.


Back to Topics


a. SPECint2000 Results generated with "Generally Available" Compilers

Metric Mem CPU Accepted Result Report
i1 SPECint_base2000 4 GB 1 27-Mar-2001 205 SPEC full disclosure
i2 SPECint2000 4 GB 1 27-Mar-2001 214 SPEC full disclosure

Table 1.

"Generally Available" code is that which has the full faith and support of the vendor company. This code is production ready and recommended by the vendor. For full details of the levels of code run, please see the "SPEC full disclosure" for each run.


Back to Topics


b. SPECfp2000 Results generated with "Generally Available" Compilers

Metric Mem CPU Accepted Result Report
f1 SPECfp_base2000 4 GB 1 27-Mar-2001 252 SPEC full disclosure
f2 SPECfp2000 4 GB 1 27-Mar-2001 265 SPEC full disclosure

Table 2.

"Generally Available" code is that which has the full faith and support of the vendor company. This code is production ready and recommended by the vendor. For full details of the levels of code run, please see the "SPEC full disclosure" for each run.


Back to Topics


c. Tricks/Tips/Cautions:
  • Benchmark tests run for a long time.
    Just in case you thought you were going to knock out a CPU2000 integer and floating point run (both base and peak) pretty quickly, think again! The four measurements described above, after they were completely tuned and working on the E420R, took about 36 wall clock hours to run! Yeah...that is what I said! One set of runs!
  • No I/O during Benchmark
    Absolutely no I/O here. This is a CPU benchmark!
  • Rate Tests
    If you have an n-way CPU, run n copies of the benchmark on the machine. Assuming there isn't something major wrong with the machine, you will get close to "n" times the one-way number. I may be missing something, but I don't have a real fondness for the "rate" benchmark.


Back to Topics


d. Net/Net - The performance you should Expect!

Think of this as a "best case performance" for your comparison. I would only use the "Base" number as my metric, and be a bit cautious with that since SPEC allows feedback directed optimization (FDO). If you are not going to compile with Feedback Directed Optimization, expect lower numbers!

I use the SPECint_base2000 numbers in my personal models which project SPECweb99 and SPECjbb2000 performance. Along with some other "fudge" factors, my models are pretty accurate, so CPU measurements are VERY important, but must be used carefully.

Back to Topics | Back to top

gcocco home
webmaster at gcocco dot com
Copyright © 1998 - 2006 gcocco software, inc. All Rights Reserved.
Page generated on: Saturday 30 September 2006 at 12:12:41 PM
URL: http://www.gcocco.com