Importance of Benchmarking Performance

We use electronics.  They light up, whir, hiss, swipe, load, clock cycles, and produce values based on input values.

In one of my earlier entries, I spoke about the different types of Technical Marketing and what they entail, and the importance of each type including what information and insight they offer.  One of the types I mentioned included performance benchmarking.  What is performance benchmarking, what does it entail, and why is it necessary?

These are all really basic questions for someone who wouldn't necessarily know about benchmarking, what it is, and the importance behind it.  It may even perhaps, sound foreign.  What does it have to do with measurements and data, and how can it be used in Marketing Claims?

First, let's talk a little bit about what benchmarking is and what its purpose is.  Benchmarks are a measurement tool (usually consisting of several workloads) focusing on and meant to measure a specific key product indicator under specific circumstances across platforms.  Platforms include technology from different vendors, specifications, and hardware/software/firmware setup.  Benchmarks are owned by consortiums, independent from but possibly in loose partnerships with vendors, designers, and manufacturers of the very platforms they measure.  This allows a chance for a fair measurement across these platforms.

Quantifying real-world use cases
Benchmarks provide an opportunity to quantify the performance in web browsing, raw compute, compute in MS Windows, Android-based workloads, and Graphics-intense workloads.

These quantifications are used to provide comparisons and values to support claims.

Substantiating Claims
In releasing a product, hardware, or platform for either consumer use, gaming, or productivity, it is easy to say ‘our new product is great.’  Supporting that claim with numbers consumers trust adds an extra element.

In addition to utilizing the data to create claims, how does one know that claim is valid?  The same benchmark used to measure can be used by a third party to validate whether or not that number is accurate and understand the confirmation used to achieve that performance data.  For example, in attaining a compute performance score using Spec CPU INT Rate using a standard memory configuration to see what the data is to determine if a higher memory size may have been used to boost the performance numbers, etc. the configuration information would presumably be presented in detail reports.  However, in a quick claim, it may be in small print or in a caveat included somewhere else.

Challenging Claims
A competitor can also use benchmarks to support data to challenge claims.  If one hardware designer claims their architecture is the best.  A competitor has an opportunity to challenge that claim and back up the challenge using a credible benchmark.

Finding hidden performance facts
Many benchmarks are available for platforms supporting different Operating systems.  Using
benchmarks that are valid for that use case but not initially considered can provide additional insight on performance.

An example would be using wanting to use a Benchmark on Chromebook but realizing that a lot of productivity is web-based also and adding a Web-heavy benchmark as a supplement as well.

Making clear, valid points
I mentioned earlier that benchmarks are a way to quantify performance claims.  Utilizing data from benchmarks takes the subjectivity out of the equation somewhat and provides a more objective basis for making performance claims.

Backing each marketing and performance claim can be a way to provide short-term and long-term credibility.  This is a great way to build and maintain trust with the customer.