discussion GPU Benchmarking Methods Investigated: The Comment Thread

[SKYMTL]

GPU performance has been a hotly debated topic recently and Hardware Canucks has decided to jump face-first into the murky world of games, timedemos, in-game rolling benchmarks and more. In this article we take an in-depth look at various benchmarking methods used throughout the industry and put their accuracy (or lack thereof) to the ultimate test.

[ Read full article ]

 

[_dangtx_]

lets take a look on your industry here

 

[Delavan]

Good work SKYMTL and HWC team!

It's a "noble" cause to try to clean up everybody's mess, start from scratch and give us the right information.

I find that, opposed to a few years back, there is now too many sites doing reviews. Now, you can find reviews that tell you "what you want to hear" basically. You cannot really make your mind by reading a lot of reviews, unless you go crazy and buy the most expensive piece of kit.

It reinforces the "brand loyalty", because nobody will risk to buy anything else but the "XFX or EVGA FLAGSHIPS, Corsair HX/RAM" or insert your "safe buy". Only the best gear, reaching max performance, max reliability and max overclockability keeps some sort of universal support, the rest of the crowd is just a grey zone...it's getting to the point where more and more systems look the same.

One site will tell you that one piece of hardware is the next big thing after sliced bread, and the other site talks about major flaws in design, performance or reliability of such hardware.

Same thing for Graphics card's performance...too many sites, to many different (and opposite) conclusions.

A real mess...really.

Thanks for going above and beyond to feed us the right information.

 

[theDarkHorse]

I thought I'd register just to say that this was an excellent and informative article!

Something that came to mind while reading the article: how do driver revisions factor into the benchmark process? I understand that different websites have different policies regarding which version of display driver they use in their testing, whether it be using the latest stable drivers, or using beta drivers. Obviously results are going to vary between driver releases, so how does (or how should) that play into benchmarking?

The situation I'm thinking of is when comparing a product against its previous generation counterpart, perhaps a GTX470 versus a GTX260, and the reviewer cannot be bothered to retest older cards. It seems like a situation where differing driver versions might mislead the reader into thinking some product performs far better than its predecessor when in fact the difference is nowhere near as great.

 

[SKYMTL]

I thought I'd register just to say that this was an excellent and informative article!

Something that came to mind while reading the article: how do driver revisions factor into the benchmark process? I understand that different websites have different policies regarding which version of display driver they use in their testing, whether it be using the latest stable drivers, or using beta drivers. Obviously results are going to vary between driver releases, so how does (or how should) that play into benchmarking?

The situation I'm thinking of is when comparing a product against its previous generation counterpart, perhaps a GTX470 versus a GTX260, and the reviewer cannot be bothered to retest older cards. It seems like a situation where differing driver versions might mislead the reader into thinking some product performs far better than its predecessor when in fact the difference is nowhere near as great.

Thanks for the compliments.

I didn't want to wade into the driver revision debate just yet since those deserve an article all on their own.

Here's where we stand currently: reviewers usually bend over backwards to test with the latest driver revisions and it consumes a massive amount of time and effort. In my opinion and from testing dozens of revisions now, when it comes to single card performance in existing games, performance increases are next to nil in most cases. Granted, every now and then we will get a massive update that pushes things to new levels but for the most part I have found that developer updates / patches affect performance more than any driver update can.

This wasn't always the case but now that NVIDIA and ATI both have good developer relations, they are more prepared with optimizations when a game is released rather than improving performance months down the road.

 

[burebista]

Thanks SKY. This is one of the best article I've read in 13 years of reading the net. It should be a reference. :thumb:

 

[_dangtx_]

the game patches should be an eye opener, to say the least.

 

[ET3D]

Nice article. I also registered to the forums just because of it.

I found the frame rate graphs particularly enlightening. They show how meaningless average / min / max FPS can be. The Just Cause 2 one in particular shows how frame rate can vary significantly, with NVIDIA and AMD trading places, and there's a specific stretch where the AMD card just suffers. Though the AMD card gets a higher maximum, it's quite meaningless as it's just a blip.

This means that even this length of time doesn't tell the entire story. Is the dip for AMD representative of most gameplay, or is the (larger, in this sample) section where the cards show reasonable parity more representative? And if a card dips under 30 fps for one second out of 2 minutes, is that really significant for gameplay? Because it's clearly impossible to tell how often these dips happen just from the min/avg figures.

It's be interesting to consider whether different statistics could give a better feeling to how the cards handle the games.

 

[W1zzard]

nice article, good job

 

[Jiraiya]

Thank you team HWC