discussion GPU Benchmarking Methods Investigated: The Comment Thread

[geokilla]

Not in a time demo or in-game benchmark. In open game play , yes absolutely it can. That's why you do multiple runs, and over a longer time frame, so that your average FPS is closer to what it actually should be. In a short clip or 20 second play through, you could have events that spike or drop your framerate for a couple seconds. That will have more of an impact on the average framerate in a short run, while the effect would tend to be minimized in a longer run.

So unless my understanding of this is completely wrong, time demos are the built in benchmark feature and in-game benchmark are clips of the actual gameplay correct? Then what's stand-alone benchmark as seen here?

http://images.hardwarecanucks.com/image//skymtl/GPU/BENCHMARKS/BENCHMARK-6.jpg

In short, I'm basically confused with what XXX benchmark or whatever is associated with when it comes to reviewing games. The actual game sequence where you're on a mission? The built-in benchmark that the game has? A clip recorded using FRAPS or whatever than replayed multiple times using the same and different GPUs? That kinda thing.

 

[Arinoth]

Very well done, way to keep us proud

 

[SKYMTL]

So unless my understanding of this is completely wrong, time demos are the built in benchmark feature and in-game benchmark are clips of the actual gameplay correct? Then what's stand-alone benchmark as seen here?

In short, I'm basically confused with what XXX benchmark or whatever is associated with when it comes to reviewing games. The actual game sequence where you're on a mission? The built-in benchmark that the game has? A clip recorded using FRAPS or whatever than replayed multiple times using the same and different GPUs? That kinda thing.

Here we go:

A Timedemo: This allows you to record a gameplay sequence AS YOU PLAY and then play it back as needed often using the in-game console commands. You can use FRAPS to record a MOVIE but a timedemo is played back IN-GAME using the game engine.

Built-in-Benchmark: These are rolling sequences which are either flythroughs or actual in-game sequences. A flythrough usually means no characters or interaction as you would find in the game. The in-game sequence is like a timedemo but is predefined rather than user-defined. The main issue with this is that GPU manufacturers can sometimes optimize performance for ONLY the benchmark and not the actual game. The same can be said for the next item...

Stand-Alone: A stand-along benchmark is just that: a separate program that installs all of the game files in a separate folder from the game itself and allows you benchmark but no interaction. As stated, they don't get patches so performance will be affected.

 

[johnjohn]

NO Crysis? Any game with an average of 30fps is already playable, any game over 60fps has no noticeable difference, that is why benchmark like Crysis are important, they can make any card fall to their knees and keep them there. Meaning if its possible to buy a card that can game Crysis at 60fps with everything max-out, no any other game will ever bother you with choppy images. Which is also what is lacking here.

 

[rjbarker]

Both JC2 and Metro33 are the "new" Crysis.....no need for Crysis anymore.

 

[SKYMTL]

NO Crysis? Any game with an average of 30fps is already playable, any game over 60fps has no noticeable difference, that is why benchmark like Crysis are important, they can make any card fall to their knees and keep them there. Meaning if its possible to buy a card that can game Crysis at 60fps with everything max-out, no any other game will ever bother you with choppy images. Which is also what is lacking here.

Crysis was pointless for this comparison IMO. It is over with the Crysis commentary because it has been trumped by Metro 2033.

 

[martin_metal_88]

Thx alot sky! Probably one of the most usefull article recently done! It's nice to know more about methodology and other thing like that. Carry one with good work!!!

 

[Appat]

Wow, very nice article ! --- I also registered to the forums just because of it :thumb:

I am somehow not surprised by the results. Of course the best way to benchmark a demo is to run it manually... but I am glad that you compared this to the timedemos, usually we assume the result would be the same (and I'm glad it's similar). The added worst case scenario bench is really a plus!

It's just too bad that the timedemo feature is so rare. I guess it shows the priority of the QA and the team resources.

I am curious, how much time did it took to make all the benchmarks ?? :funky:
​

 

[SKYMTL]

I am curious, how much time did it took to make all the benchmarks ?? :funky:

The research for this article was done slowly over the course of 4 months. THe testing was done over the course of 3 weeks so in total I would say I burned about 30 hours on it.

 

[geokilla]

Here we go:

A Timedemo: This allows you to record a gameplay sequence AS YOU PLAY and then play it back as needed often using the in-game console commands. You can use FRAPS to record a MOVIE but a timedemo is played back IN-GAME using the game engine.

Built-in-Benchmark: These are rolling sequences which are either flythroughs or actual in-game sequences. A flythrough usually means no characters or interaction as you would find in the game. The in-game sequence is like a timedemo but is predefined rather than user-defined. The main issue with this is that GPU manufacturers can sometimes optimize performance for ONLY the benchmark and not the actual game. The same can be said for the next item...

Stand-Alone: A stand-along benchmark is just that: a separate program that installs all of the game files in a separate folder from the game itself and allows you benchmark but no interaction. As stated, they don't get patches so performance will be affected.

Hm I think this answers all my questions. Thanks