discussion GPU Benchmarking Methods Investigated: The Comment Thread

[sswilson]

Great job SKYMTL!!!! I can only imagine how much work went into putting this together.

Personally what I'm taking away from it is that there is always going to be a need for a subjective opinion (doesn't have to be from reviewers, could be from trusted forum members)... no single test (or groups of tests) is ever going to be able to tell the whole story.

As to the companion article on our current benchmark files goes.... three thumbs up!!!! Can't be much more transparent than allowing individuals to dl the files and run them themselves. The only question I'd ask is if there is going to be a sticky on the forums with the latest version(s).

 

[SKYMTL]

Nice article. I also registered to the forums just because of it.

I found the frame rate graphs particularly enlightening. They show how meaningless average / min / max FPS can be. The Just Cause 2 one in particular shows how frame rate can vary significantly, with NVIDIA and AMD trading places, and there's a specific stretch where the AMD card just suffers. Though the AMD card gets a higher maximum, it's quite meaningless as it's just a blip.

This means that even this length of time doesn't tell the entire story. Is the dip for AMD representative of most gameplay, or is the (larger, in this sample) section where the cards show reasonable parity more representative? And if a card dips under 30 fps for one second out of 2 minutes, is that really significant for gameplay? Because it's clearly impossible to tell how often these dips happen just from the min/avg figures.

It's be interesting to consider whether different statistics could give a better feeling to how the cards handle the games.

I agree. Like I said, there is no perfect yet semi time-conscious way to benchmark a game unfortunately. Any graphs that are put up only show a small slice of the whole story but I would rather a 2-4 minute slice rather than a 15-30 second sample. :thumb:

 

[gingerbee]

this is why we come here thanks HWC :thumb:

 

[jcmaz]

You should send this article to the "2 dozen" or so site and ask for feedback

 

[FreeKnight]

Great article Sky, I'm with gingerbee in that it's the extra mile that you guys go that keeps me coming to HWC first for reviews and information. When I'm looking for a specific product I'll read every review I can find on it but 9/10 it's your, AkG's, etc articles that I'm using as the best guides for the info I want.

Keep up the good work

 

[rjbarker]

Nice Sky.....personally when I make a GPU Upgrade I run my new Cards througha small "Series" of Benches to check improved performance over old and can relate to the time-consuming part of it......always appreciated

It's a "noble" cause to try to clean up everybody's mess, start from scratch and give us the right information.

I find that, opposed to a few years back, there is now too many sites doing reviews. Now, you can find reviews that tell you "what you want to hear" basically. You cannot really make your mind by reading a lot of reviews, unless you go crazy and buy the most expensive piece of kit.

It reinforces the "brand loyalty", because nobody will risk to buy anything else but the "XFX or EVGA FLAGSHIPS, Corsair HX/RAM" or insert your "safe buy". Only the best gear, reaching max performance, max reliability and max overclockability keeps some sort of universal support, the rest of the crowd is just a grey zone...it's getting to the point where more and more systems look the same.

One site will tell you that one piece of hardware is the next big thing after sliced bread, and the other site talks about major flaws in design, performance or reliability of such hardware.

Same thing for Graphics card's performance...too many sites, to many different (and opposite) conclusions.

A real mess...really.

Thanks for going above and beyond to feed us the right information.

^^^ Thats a tad blown out of proportion........as soon as the nVidia Fermi cards were released I personally found all the available Reviews fairly accurate as to performance of the latest n greatest GPU offerings, obviously HWC was the best ...for Performance it still stands

1. 480
2. 5870
3. 470
4. 5850

^^ Pretty straight forward really.......throw priceerformance ratio into the mix and its up to the purchaser to make a decision based upon their own expectations and budget obviously.....

 

[stoanee]

Nice discourse on the whole vid card testing routine. This will probably not be enough to appease the rabid fanboys though. It doesn't help that we are perceived as a pro Nvdia site. (sarcasm intended)

 

[chrisk]

Good work and appreciated. I too wonder how long some sites bench a particular game as it is not always available. I agree with Stoanee though that it will not detract the fan-boys, but for those with a open mind this article should be required reading.

 

[SKYMTL]

The whole NVIDIA versus ATI debate doesn't really factor into this in any way IMO. I made no mention of either card in the conclusion and only a few vague references throughout the article. The whole point was to show what SHOULD be benchmarked and what should be avoided at all costs.

Unfortunately, we all know that this will do nadda when it comes to "cleaning up" the vast majority of GPU reviews out there. We still won't see details about benchmarking processes, etc just because it isn't in the interest of some sites to be transparent and admit they are using the wrong tools.

 

[MpG]

Excellent article, worth the wait, and the effort that went into it is much appreciated! While this isn't the first I've seen of canned benchmarks failing to match up to in-game experience, it's nice to see the numbers laid out to get an idea of exactly how much discrepancy there actually is.

My own two cents is that I REALLY like the full framerate graphs. As good as it is to get average and minimums (and know the exact methodology that made them), I find seeing the FPS-spread just provides a whole extra depth of information to use when making a purchasing decision.