Gadgetory

our first in-depth foray into the VR benchmarking scene is focused on testing the AMD r7 1700 and the Intel i7 7700 K CPUs on both the HTC vive and the oculus rift headset we're plotting delivered frame times dropped and synthesized frames warp misses frame rate overall and plane back Hardware capture of the benchmarks in this set of tests VR benchmarking is still new and challenging to perform so we've limited the tests to five games and four configurations but even just running those 20 total configs take significantly longer than benchmarking an entire GPU suite and must be preceded with an explanation of VR testing before getting set this coverage is brought to you by the 1080 SC which has a new MSRP that is it lowered with the launch of the 1080i series at 1080 and 1060 SD cards come with for honor or Ghost Recon wildlands which you can choose at checkout learn more at the link in the description below interpreting the data in these charts today isn't necessarily obvious this is a different type of chart and the data has a different significance than our normal test with traditional benchmarking so part of this is going to be explaining what we're actually looking at the first game benchmark that shows up on the screen will explain what some of the numbers are on the lines of the bars and all that and then going forward we'll be dumping a lot of the charts from the games following the first one which is dirt rally just because if we put every single chart in this video there would be 50 charts it'd take an hour to go through them one minute at a time so instead we're going to put only the most pertinent data sets in this video the rest as always will be linked in the article in the description below which will have pretty much every chart you could possibly want including individual breakouts of the two CPUs of the TCPS overclocked hardware and software monitoring and then the comparative analysis which is what we're mostly focusing on today in the video as an intro to VR benchmarking our setup is as follows we're using Hardware capture of the headsets to intercept the footage and send it to another machine this is done in a way that does not impact performance of the benchmarking system and that's important since it's just splitting the data a splitter box sits between the capture machine and the gaming machine and that feeds into a $2,000 SC hd4 card from vision which is capable of accepting the high bandwidth from VR headsets and also splitting things we then use virtualdub to capture the playback in the headset and on the gaming machine and run color overlays that bake into the output from the FCAT VR suite this can be later extracted to analyze delivered and drops frames on a hardware level and note also that you need a very fast SSD or r8 SSDs in order to keep up with the data because one minute of testing can easily equal a 50 gigabyte file we finally feed that file into our own compression script which is creating the files that you see in this video and those are compressed down to a level of hundreds of megabytes rather than tens of gigabytes on the gaming machine we use FCAT VR to intercept frame time delivered frame rate drop frames and warp miss data of note for today frame times represent the time in milliseconds between each frame delivery just like always VR headsets at range from roughly 11 to 13 milliseconds for the time required to deliver a frame before encountering some sort of drop frame or warp miss or other unpleasant action and the extra 2 milliseconds there at the end is really just an approximation because the risk can do some funny things at runtime to help lengthen the amount of time before the frame is required which is generally 11 milliseconds if at 90 Hertz refresh but there is some stretching room after that the captured files look something like what's on screen now at which point we feed these into a spreadsheet and into FCAT VR filter the data and output the data you'll see in the test later there are thousands of individual data points for each test which is one minute on and then the top row is several very good variables long as well so there's plenty of data there to analyze and interpret the hard part is figuring out what to do with all that data once we've captured it we have previous video explaining drop frames and warp misses and what those are if you're curious about those to specific names as it pertains to VR testing and in that video we also talked about the VR pipeline so that goes through this 11 millisecond window where do you hit the run time when do you need to have the frame ready and dispatched and things like that so that's been a previous content piece as for the benchmarking itself for today the our test execution is a big challenge so VR testing in general is brand new so we've been working on this for a few months with FCAT VR and early iterations but even with that experience it's not a perfect setup so a few things here to note first of all there's a major human element with VR testing that cannot really be easily resolved so it's not only do we have to as usually execute test passes which are fairly identical more or less from one path to the next but we have to do that while in VR with a headset where you've got head movement so the level of accuracy diminishes compared to a standard benchmark where you're just using a keyboard and mouse and that's something we acknowledge and basically use error bars in the graphs and the bar graphs to show a margin of error which right now is a bit wider than I want it to be we're working on it's sort of tightening the variance between tests but there's only so much we can do before you run into issues with just VR in general not being a very friendly platform to benchmark compared to normal benchmarking so another example of this outside of the human element there's also a randomized element with the games a lot of the VR games right now are king of the hill style stand in the middle you attack enemies that spawn around you those enemies are normally randomly generated and depending on your performance one run to the next it could be that you see more enemies or fewer enemies or whatever then in the previous pass that's not a challenge now fortunately we have enough data from again the months of working on this to know what the variance is and what the margin of error is one pass the next with the game and so we have that margin of error bar on the charts it's not too wide it's plus or minus 1.5 percent more or less but that will give you an idea of where the numbers fall or where they could fall one pass the next the game is benchmarks for today include the oculus rift version of dirt rally oculus rift version of elite dangerous and then we have the HTC vive running raw data Arizona sunshine and Everest if you're curious about the test platform used and the specs of the machine you can check the article linked in the description below but the most important element is the GPU and that was our gtx 980ti hybrid model that we built ourselves for the first test we're starting with the oculus rift and then moving on to the vibe our rift games include again dirt and a weed we're testing dirt rally configured to high settings with the event the blending option enabled this first set of charts will contain all the data we have while subsequent games will only contain a head-to-head data so they will be simplified and faster to get through for all of it if you want it for all the games check the article below let's start with the r7 1700 CPU at stock settings only the left axis on this chart shows frame times in milliseconds lower is better here and we have a rough 11 millisecond window to deliver the frame and the rift sometimes it can go up a bit maybe around 13 of jamak the magenta line represents the hardware capture while the red line represents the software capture the harbour capture cannot see what's going on at a software level and so only validates findings by illustrating dropped frames never delivered to the headset the software line is more of what we're interested in the lower third of the chart meanwhile is an interval plot this one else has visualized delivered synthesized frames dropped frames and delivered at new frames with everything explained now we can start the data analysis for this chart with the 1700 stock we can see that the 1700 encounters a few dropped frames or frames that were synthesized by updating a head dragon in position without a full update to the scene right now we're playing a video playback of the hardware capture for this run the experience overall as seen in the playback is smooth and the drops and synthetic frames go unnoticed during use we don't have enough to detect as a human here that we can detect it with tools and we'll talk about this more going forward because again as a reminder if you're at sixty seconds for a test 90 Hertz that means you have 5,400 intervals so a couple drop frames is not a big deal will show average frame times an unconstrained FPS at the end of this charted section next the r7 17 overclocked shows mostly the same performance with one pretty bad drop at the end of the capture but overall nothing too critical the time to deliver frames is now shorter with the 3.9 gigahertz overclock but there's no major difference in user experience this next chart shows the 1700 stock and overclocked at the same time where we can observe that the red line representing the values is consistently faster in frame delivery than the yellow line or the stock 1700 we are generally around 10 milliseconds for both devices but we'll show that value more explicitly in a moment and now here's the intel's i7 7700 kc view with stock settings Intel CPU has a few drops but just like the r7 chip these are not appreciable to the end-user frame times are closer to the eight to nine millisecond mark with the Hardware frame time spikes validating the software measurements exactly at the same moment we're playing some gameplay of the Intel i7 7700 K benchmark now where you'll notice that user experience is smooth and without any significant or noticeable hitches both CPUs can deliver a smooth experience which is subjectively to the human eye the same but let's look at how they compare objectively here's a chart showing the seventeen hundred and seventy seven hundred K heads ahead in stock frequencies because we have standardized this benchmark with the same head movement we can see that the frame time spikes generally aligned between both CPUs Intel's faster overall in frame time delivering this test with each CPU dropping fewer than 20 frames throughout the entire run it's not until drop frames are significantly greater on one config than the other that we'd actually notice them so for all intents and purposes the experience on each CPU is the same that said Intel it's faster in its frame times on this particular title and experiences shorter spikes when the going gets tough but let's look at a few more charts first one more frame time chart and then we'll look at the bar graphs this one shows the two CPUs overclocked until at four point nine and a MDF 3.9 gigahertz both overclocks are achievable on the majority of the respective chips the drop frames are similar again with the experience again being effectively equivalent and these closer to the 11 to 13 volt second cutoff window but still within bounds and so the experience is the same interestingly and tell loses ground and overclocking compared to the stock benchmarks and we'll see that this trying to repeat throughout the tests let's get a bar graph on the screen for better illustration this chart plots delivered FPS to the headset which is the most important metric then drop frames of the second most important metric and then unconstrained FPS as a calculation unconstrained FPS is an improved prediction of how many frames would have been delivered per second if the hmd did not have a fixed update interval of 90 Hertz since they are fixed at 90 Hertz really the most important item is delivered at PS the unconstrained value is calculated by taking $1000 seconds and dividing it by the average frame time which is done in the new FCAT VR tool automatically the two hard metrics are again delivered FPS which we can validate with an effectively infallible hardware capture setup and drop frames also validated by hardware capture drop frames are an absolute measure in total frame count over the test period which is 60 seconds at 90 Hertz a 60-second pass will produce 5,400 refresh intervals on the headset final note we currently have a test variance of roughly plus or minus 1.5% in this chart for dirt rally we immediately see that both the seventeen hundred and seventy 700k are capable delivering ninety FPS to the headset that's what we want and we next see that the drop frames have a range of eight going from three to eleven drop frames per test pass as we saw in our previous charge the drop frames are not clustered tightly enough to be noticeable by the user in the absolute worst case there are seven 1,700 encounters eleven drop frames over it's 5,400 refresh intervals to put that into perspective that yields a 0.2% drop frames for the task period a user would not notice this particularly when the drop frames are spaced out over the entire period we next see that unconstrained FPS lands around 135 to 137 for the 7700 K while the are 770 100 is in the 105 to 115 FPS range again this is an extrapolation we're still seeing 90 on either device through the HMD and let's now move to the average frame time chart here we see where that number is calculated this charts scale is set to 12 milliseconds at which point you'll probably start encountering that drops wort misses or reprojection issues the 7,700 case stock and overclocked cpus perform effectively equally at seven point three to seven point four milliseconds the r7 1700 shows a bigger gain from the overclock just outside tolerances landing at eight point seven milliseconds from nine point five four milliseconds comparatively the 7700 K stock experiences a 22 percent reduction in average over the 1700 stock and about a 14.8 percent reduction versus the overclocked r7 1700 let's move on to next game finally the next hosted game is elite dangerous which has Hardware capture footage on the screen now we're going to simplify the frame time charts here and only show comparative data if you want all of the charts check the article linked below elite has some issues with the vive so we're using the rift this was configured with VR high settings in the game and played in the VR training level this first chart shows the 1700 versus the 7700 K both had stock frequency is the r7 1700 times to run slower an average fan time delivery sometimes running against the limit before we start encountering the rift 11 to 13 most I can refresh interval again note that oculus does some things and run time to stretch that 11 mils I can refresh out a little bit hence the extra buffer at the end there as illustrated in the interval plot at the bottom the r7 1700 and the i7 7200 K are dropping and sympathising a similar amount of frames you can see that with the green yellow and red colors on the lower two charts neither of these is appreciably worse than the other as a user again this experience is equal within the confines of human perception and with what the heads of the Wow that said the difference is statistically significant you could make an argument that the extra Headroom is valuable but we have not yet found a scenario where we begin encountering noticeable jarring or stuttering on either the 1700 or the 7700 K this chart shows that overclocking the r7 1700 it tends to close the gap versus Intel which is also now overclocked on the frame time and interval plot graph that's the same as we saw in dirt rally it appears that over clocks don't benefit Intel quite as much as they're benefiting AMD here likely because of the r7 s lower starting frequency we're seeing a range of 6 for the drop frames from 11 to 17 in the best and worst cases and the i7 7700 KOC and 7700 K is stock performed equally when looking at the bar graphs that's within variance and further illustrates that VR benchmarking is not yet repeatable enough to analyze with tight margins and the hard statements as to what data were seen even in the worst case of the 7 and drop frames we're still at 0.3% of all frames delivered as drop whereas 11 drop frames would be 0.2 percent completely imperceptible to the user the dropped frames are also dispersed enough to not matter with regards to actual delivered frames we're at 90 FPS for all four tests intel holds a lead in unconstrained FPS as it has a shorter frame Layton sees overall as shown in the next chart that we're on now for average at frame times R at roughly 8.6 milliseconds between the two Intel tests and at nine point nine and ten point four for the AMD test ten point four is starting to push the limit of what we're comfortable with but still delivers a smooth experience in this game the overclocked keeps us reasonably distance from the 11 to 13 most I can mark and again there's no perceptible difference between the excuse given that they all hit 90 FPS without any significant drop frame counts that are noticeable on the screen now our test passes for the next game raw data we're moving on to the HTC vive using raw data as a VR specific title that's shown some promise this is another king of the hill title as you can see from the footage we've got on the screen and these first results are with the game configured to high settings with zero mr-s this frame time plot shows the 1700 versus the 7700 K both stock neither CPU struggles here with the 1700 generally being faster in frame delivery that said both are below the 6 millisecond line on average and the interval plot below shows that Intel encountered 0 drops or synthesized frames and the encounters a few but they're not significant enough to be perceived by the wearer this seems to be a trend between these two CPUs because they're both pretty good at what they're doing here these two for all intents and purposes are again effectively equivalent in this game here's the overclocked data same story here Intel drops no frames but tends to run average frame times a bit over the 1,700 let's just move straight to the bar graphs delivered frame rate post 90 FPS for all devices with AMD dropping 6 frames in each pass and Intel dropping 0 this is again not really noticeable but it's worth plotting because we can measure it Intel CPU is again showing limited gains from overclocking with both line items performing effectively identically an unconstrained same rage and that's both 7700 KS the same is true for AMD this time we're overclocking doesn't really change the fact that we're around 229 unconstrained fps looking at average frame times we see that the 1700 CPU stick around four point three six milliseconds with Intel around four point nine six milliseconds looking back to the video capture of the game which we can play on the screen we think that both of these are so distant from an eleven millisecond refresh interval as to be effectively equivalent in visual output you could buy either CPU and experience an identical experience in the headset for this raw data pass we did on high based on our initial testing more charts in the article if you want to see the rest of raw data data we're now testing it using Arizona sunshine which is another kind of the hill title on the vive and the rift we're using the vibe here and testing with advanced CPU features enabled and very high texture quality as you've seen oil introducing this game is a zombie shooter game and this test is conducted during the first swarm wave here's the first chart both CPUs perform at an equal framerate particularly when considering the certainty of test variants in this game there are seven 1,700 encounters more sins and drop frames this time but we've still not encountered a warp miss which is important we have not seen a single warp miss on any of these tests without any warp misses between these drop frames or without heavier drop frame saturation in the interval plot it remains fair to say that the experience between the two CPUs is really not that much different visually objectively Intel does have fewer drop frames in this title we're just basing a question of when does that become noticeable to the end-user and since VR testing is still new it's kind of hard to say we need more experience in the game to really make a definitive statement on that overclocking it helps the r7 1700 I've seen here and post an effect of zero change with a 7700 K again and then finally let's move on to the next chart here's the FPS output our seven 1700 ops 100 frames over 5400 intervals or 1.85 percent that's the most out of everything we've seen so far but still not a big deal and that's reflected in the delivered frame bar as well where we're seeing 87.5 FPS average versus ninety FPS delivered frames to the headset the overclock gets us down to 38 dropped frames but the gap between 57 fps and 262 FPS on our seven CPUs shows that again this test is in perfect this is precisely why we added those error bars 87 verses 89 FPS is also imperceptibly different and the unconstrained framerate does not post a statistically significant difference either though objectively Intel is better in this title at least with regard to drop frames very quickly here's the average frame time chart that supports that statement again no statistical significance in the differences between frame times with this test configuration they're basically all the same finally Everest is more of a tech demo than the game that's the one we're showing now it's got some high-quality visuals that are stitched together with photogrammetry lastly spoke to the Everest ebbs the demo used more than 30,000 photos of the actual mountain to build its environment we're using can blue Icefall for the test course the settings configured so that the bars are 2 ticks down dramatic settings and so that they're all equal lengths in the graphics menu which is really not a great graphics menu here's the frame time chart between the 1700 and 7700 Kate stock it's also the most boring chart we've looked at thus far the interval plot shows really no activity other than mostly successful frame deliveries frame times look about the same looking at the FPS chart quickly we see the same thing these numbers are more or less equal average frame time chart finally blasting through these reinforces that everything is running around 6 milliseconds to CP is producing equal experience in this game with neither superior to the other for the most part there's no real significant appreciable difference between these two CPUs in these games and normally that would be kind of a boring thing but because VR testing is so new this data is good to have because we can actually start building an understanding as a community as to what is a good experience in VR and as we move toward testing things like r5 and i-5 will probably start seeing more drop frames and limitations in VR with CPUs which of course VR being such a high resolution thing is almost natively a GPU bottleneck but by controlling the settings in the games and lowering them down to things like high with a 1080i hybrid we're able to eliminate a good amount of that and still show some of the CPU limitations it's just that the r7 and the i7 do fine in this they don't really have an issue in general for the 2 rift games we tested Intel tended to do better but it was not in a way that really ultimately and the same was true when Andy did better in the one or two games where it posted superior frame times whether that was a big or not Swain the difference to a user was the same so it's an interesting challenge to test these things the heads-up Sarlacc 290 Hertz your frame rates lock 290 FPS and unconstrained that metrics are really interesting and cool to have be but it's more for GPU testing where you might have six GPS on the chart seeing on constrained FPS in theory helps tell you what the GPU is capable of in a more familiar metric beyond the ninety FPS limit which is effectively locked vsync setting still it's an imperfect metric because it's trying to extrapolate something that you're not going to see now unconstrained FPS does kind of tell you hey this GPU or this CPU can prepare more frames for delivery so in theory going forward it might indicate that that particular product performs better but it's really too early to say if that's how it's going to play out we'll see so overall first VR tests let us know you think we have some more plans probably for the r5 maybe the i-5 as always you can go to patreon.com/scishow solves that directly with this in-depth testing thank you for watching subscribe for more gamers Nexus not net for the article version of this or if you're a fan store doc gamers Nexus net for the shirts I guess the natural progression of a youtube channel is that the end slowly widens as you add more and more things to plug so that's it for this one thanks for watching I'll see you all next time you