Gadgetory

welcome back to harbor unbox today we are talking about a topic that seems to just keep popping up no matter how many times we put it down as soon as the g-force 20 series was announced too many of you started requesting that we monitor in videos driver development to make sure they didn't start intentionally slowing down the previous generation Pascal GPUs an effort to make chewing look a bit more appealing it seems the team over at ufd tech we're also seeing similar requests and this time last week they decided to look into it I've actually been doing the same thing so in their video popped up a week ago I was interested to see if they'd been seeing the same thing as me which was diddly-squat Wright began the video by suggesting that they had caught in video route so right away my interest was piqued about 6 minutes and 30 seconds later though the testing was all wrapped up and in my opinion there were just a few surprising results most looked to be sort of margin of error type stuff certainly nothing alarming but Bret seemed pretty convinced that that was evidence of gimping however after wrapping up the testing he went on to say they just got their hands on the new 4 11.7 T driver and that restored performance the same level seen previously with the 399 24 driver which was the last publicly available driver before Nvidia released the RT X series the driver that they found the issues with was drive immersion for 1163 and that was the first publicly available driver to support the RT X series so he concluded that driver probably just had a bug that nerfed performance on the Pascal GPUs so all very interesting I had a few thoughts about the video but never intended to make any kind of reply at least not directly like I am with this video which is why this video ended up coming a week later since ufd tech published their video we've received dozens upon dozens of requests for us to look into this both in the comments section of unrelated videos that we published last week and on our various social media platforms that we use a viewer on twitter reached out to science studio paul's hardware JS to sense and harbor unboxed to see if we could look into the video published by ufd tech though claiming that in video was enduring the g-force ten series and give our opinion I simply replied with did you watch the entire video Paula and Jay also weighed into the conversation with their own thoughts now before I continue with I suppose my take on this subject I just like to make it very clear that I'm not calling you FD Tech out in this video I respect the work that Brett and his team do and I'd ask anyone watching this video do the same they've done nothing wrong here at least nothing really wrong and I think I can add some value to the conversation given that I have spent thousands possibly upon thousands of hours benchmarking I will just say though as a viewer and a fan of ufd tech I wasn't wasn't that wrapped with the format of the video I felt that it would have been much better if they just show the results for all three drivers from the start of the video to avoid misleading any viewers because we all know the second someone sees results that they want to see they're taking a screenshot and running over to Reddit claiming that Nvidia is the devil and well I'm sure you guys know how that goes I'm can understand how annoying it would have been for them having done all that testing thinking that they're onto something they probably should have double-checked the numbers but anyway they did a lot testing so good on them and then everybody it goes and releases a new driver they install that and find that out hang on we're back to where we started so that's a very frustrating when that happens we deal with that sort of stuff a lot so I can certainly understand why they didn't want to go back and completely restructure the video but ultimately that's probably what they should have done if we look at the top comments on Bret's video we see stuff like nothing surprises me with ingredient video is aware that people are gobbling up those ten series cards on the use market myself included we need AMD to save us now nvidia is so scummy nowadays Lowell greedy Nvidia trying to scam their customers any way possible they need to make 1080i bad so they can sell 20/80 nice bathroom hmm okay I think that last one is unrelated at least to this subject possibly not to the video but anyway I did like the comment made by Johnny six for one come on guys you're ready to call out and video for gimping cards over to FPS difference input this is well within a margin of error I'm all for calling out companies for shitty practices but this is a bit ridiculous by your own test you guys struggled to tell the difference between sixty Hertz and 240 Hertz so how in the hell would two FPS even be perceivable off topic there is a world of difference between sixty Hertz and 240 Hertz that's like a night and day difference but suppose I'm getting a bit off topic there he is right with the first part 1 to 2 FPS is well within the margin of error for testing and I will show exactly that in a moment there won't be many benchmarks in this video as there is there's honestly nothing to report I have done a whole heap of testing I've spent over a day checking a whole heap of games and yeah nothing at all to report but I will talk a bit about benchmarking practices and explain why I think they saw some of the results that they did all up I believe they tested about 11 titles which is certainly a good number and good on them for investing so much time and effort I can certainly appreciate that the biggest performance drop was seen in ashes of the singularity with the GTX 1082 I using the higher quality preset at 4k and I assume they were using the DirectX 12 API but that wasn't specified here the 1080 t I lost 6 FPS on average dropping down from 98 to 92 fps Metro last light dropped to 2 FPS which at the low frame rates that they were seeing at 4k meant it was about a 5 percent drop in performance at Deus Ex mankind Nevada dropped just a single frame for a 3 percent reduction and rise of the Tomb Raider lost 4% for a 2 FPS drop shadow of the Tomb Raider was 5 percent slower dropping 3 fps and Final Fantasy 15 was just 4 percent slower dropping just 2 fps so with the exception of ashes of the singularity some very minor reductions they're certainly within the margin of error and Bret commented that all the Ubisoft titles saw no change and it was a bit baffled as to why that was Ghost Recon wildlands Assassin's Creed origins and far cry 5 all showed no change the same was also true for middle-earth shadow of war and hit man with the GTX 1072 at 1440p Bret reported no change in performance at all so the same results with either driver and then the gtx 1060 at 1080p saw slightly mixed results though all work within the margin of error I myself have been testing the Pascal gpus with the four 1163 driver and i've done so in a shipload of games and i haven't found any noticeable difference between it and the previous 399 24 driver I hadn't tested with Asha the singularity as that's not really a great GPU benchmark so a bad one but I typically reserved it for CPU and sort of system memory type testing stuff anyway given their findings and pretty much all of you guys seem to want us to dig into this and do some testing I thought why not have a look at ashes of the singularity so using the DirectX 12 API and the higher-quality preset at 4k weak the GTX 1080i I began testing for this test I'm using the built in benchmark and I'm not going to show any data using a third-party tool like a cat for example to report performance I did check the frame time results though the margins were identical to what I'm showing here and they mostly focused on their average frame rates cuz that's where they saw some big changes so we'll do the same so that being the case I am just going to keep it simple and show the average results for the normal and heavy batches but again I have dug into a lot more titles looked at frame time performance and there is absolutely nothing to report outside of the margin of error I'm also going to show the results for each individual run here however unboxed all our data is based on an average of three runs and I assume ufd tech are doing the same and not just reporting a single run so the first run using the four 11.7 driver we saw an average of 112 FPS for the normal tests and 94 FPS for the heavy tests I then ran at the display driver uninstall or better known as DD you to remove the driver and install a fresh version of for 11.6 t3 and then I ran the tests again this time I got an average of 115 FPS for the normal batch and the heavy batch spent at a result of 96 FPS on average so that is a two to three percent improvement with the driver that ufd tech claimed the bug with I then ran ddu again installed driver 399 24 I ran all the tests again and this time I got 114 FPS for the normal test and 96 FPS for the heavy tests so that is a 2 percent increase over the for 1170 driver and I would certainly say that 2 percent is within the margin of error so for me the latest driver actually provided the worst result to confirm that Iran DD you once again reinstalled the 411 dot 70 drivers and then rerun all the tests once again this time of the normal batch test averaged 113 fps and the heavy batch 95 fps this meant I clawed back just a single frame for each test and now the four 11.7 a driver was within 2% of the 4:11 dot 63 driver which again is very much in the margin of error it's worth noting that ash to the singularity even at 4k is very much a system benchmark when using the higher quality preset with the GT X 1080 Ti any change in memory or CPU speed will impact the results not massively but certainly by a few percent and that's not something you will see when using the crazy preset to illustrate this point I went back and reroute all the tests again but this time with the crazy preset which makes the game extremely GPU bound now all three drivers provided the exact same heavy batch result of 59 FPS with no more than a 1% deviation for the normal batch test had Brett's team tested ash of the singularity using the crazy preset I'm pretty certain they would have found the same results as I did and to tie him with that as I noted earlier Brett was surprised that the Ubisoft game showed no difference in performance between the two drivers and I think I know the reason for this and we're talking about games such as Ghost Recon wildlands Assassin's Creed origins and Far Cry 5 and the reason is all three titles are extremely GPU bound at the 4k resolution even with mild quality settings this leads me to believe that there was possibly something going on with our test system perhaps something was running in the background they did only see unexpected results in tests that were heavily GPU bound and this is what I'd expect to see if there was something else running in the background of course I can't be sure but that's how I would interpret their data and we're only talking about one game here really ashes the singularity the rest of them it was just margin of error stuff 1 or 2 FPS isn't really a difference worth noting if they had of you know run DD you got rid of the driver done a fresh install and then taken I'll compared the data over multiple runs 3des maybe even six runs and taking the average you'd probably find that the FPS margins were at one at most possible zero I should just follow up a note that yes I'm aware that Assassin's Creed origin hammers CPUs but I assume they were probably using an 8700 K so that means that the game would have been 100 percent GPU limited at the 4k resolution you could probably down clock the 87 or okay in fact I have done this to four gigahertz from five gigahertz and you don't see any change in framerate even in the big cities for those of you wondering I have tested all the games just mentioned plus many more with all three drivers and there's no noteworthy changes into performance so some people will be upset that I haven't shown that data but it's really boring you just see the gtx 1080i delivering the same performance so i'm not going to show you a holly / graphs that tell you no information it's just complete waste of your time and my time making the graphs that tell you nothing now if nvidia did nerf or gimp the geforce 10 series whatever you want to call it even if it was by mistake that's still a seriously big deal and something we would take very seriously and it's a reason why I've kept in Ireland I don't expect them to do it and I'll talk about that in a moment but at this point all the big names have done all the big names in their industry you know you Linus tech tips and all those guys have done their testing and they've done it with the 399 dot 24 driver so they don't really gain anything by doing this with the first public release driver and it's also just a really really dumb thing for Nvidia to do and therefore they'll never do it deliberately gimping GPU performance via the drivers just it's not a thing guys it's a myth made up by fanboys it's just it's not gonna happen it's certainly worth keeping an eye on for sure if it's only to spot a bug but it's seriously not going to happen and there are multiple reasons as for why it's never going to happen one reason being and I'm not saying this is the most important reason or the most obvious reason but it's certainly one reason is that it would be bloody obvious and easy to spot if they did do this all you need is a previous driver which is extremely easy to get in video even provides them on their website alternatively there are countless online resources where you can get any Nvidia driver that's been previously released throughout the years and then you can compare them on a single GPU in an appropriate title and you will find your answer the newer drivers should never be slower outside of the margin of error I see these comments all the time and you FD Tech obviously get them a lot because they decided to do the testing and make their video and I believe they had a lot of requests because we had a lot of requests but I just thought it doesn't make sense and it's certainly something a video could do but they're just not going to do it as I said what they could do and probably do understandably so if they do do what I'm about to suggest is not invest the time optimizing for all the hardware hardware that's more than a few generations old take Kepler for example Nvidia and the game developer as well because the games also have to be optimized for these older architectures they're not going to spend a lot of time optimizing old hardware they'll certainly ensure that Kepler works and there's no major bugs stopping it from working as it should but they're not going to invest loads and loads of time fine shooting drivers and even the game itself just so four year old hardware can run as well as possible so if you start to see I don't know half decade old GPUs performing worse than expected and the latest and greatest tiles that's probably what's going on Nvidia and AMD are deliberately gimping this old hardware they just don't want to invest the time money it takes to get it to its peak level of performance and they still want to be beating the other team in the graphs when we do our big you know 50 GPU tests they don't want to see the GTX 780 right down the bottom getting smashed by the r9 390 that's not particularly a good look not great for business so they will want to make sure that it's operating close to where it should be they just as I said don't want to invest too much time or money on this old hardware now let's talk a bit more about benchmarking that easy thing that anyone with no experience can do because they just need to click the benchmark a button or have MSI Afterburner open while they spot check the FPS counter every now and then hey ma I'm benchmarking I'm basically gamers Nexus seriously though benchmarking it really isn't that hard benchmarking accurately though and comparing multiple data sets whether it's different drivers or different hardware that can be a bit of a chore as it does take quite a bit of time and experience to do well I'm not suggesting the guys at UF D Tech don't know how to benchmark I'm not doing that at all but I would just like to explain why some of the results may get a little cattywampus at times for this particular testing step one would be to create a fresh install of Windows 10 to make sure there's nothing influencing the results there because you really do want a clean and sterile system for driver testing not necessarily for other things but for this test it certainly makes sense you want to remove any other factors as much as possible so fresh install of Windows 10 then if it installs a default driver which it often does you want to then reboot in a safe mode used edu to uninstall that driver and then boot back into Windows and install a pre downloaded version of the driver that you're going to be testing with once you're done with that driver don't be tempted to just install the newer driver or the older driver that can leave some old or newer driver elements behind so reboot back in a safe mode used edu to completely clean up the driver that you're currently using then you can boot back into Windows and install the new driver so yeah very important step for this kind of testing especially if you want to gather accurate data other things that can skew the results ever so slightly include air bloody air sometimes it's cool other times it's less cool ambient air temp can affect the results by a few percent generally fans will spin up to offset a higher ambient temperature but that's not always a fix they might already be spinning at or close to full speed and any increase in temperature will force the hardware to begin throttling temperatures can also affect CPU performance and you'll notice this in sort of system benchmarks like ashes of the singularity the GeForce 10 series also makes use of GPU boost 3.0 which again can skew the results by a few percent nothing major but if you're looking for small performance changes this can cause them ideally you want to run the game for around 10 minutes before you begin testing as this allows the card to find its maximum operating temperature and this sees that run at a sustained operating frequency for sure to test where the card hasn't fully heated up the operating frequency can spike quite a bit higher than the sustained frequency and this will lead to a few extra frames another big issue is updates games regularly update in some we don't even notice it which makes retesting quite important of course there are also Windows updates to contend with and even simple stuff like BIOS updates though those are a little easier to spot because they are carried out manually but you have to be aware that you've done it put that in the back of the old noggin to make sure all the settings are back the way they meant to be again I'm not suggesting that you FD tech unaware of all of this stuff and that they made any mistakes here and Bret did say that they did all the testing within one day so it's unlikely that they got caught out with any kinds of updates as far as I'm aware there's been no update to ash of the singularity during that period now the basic things include how many runs you make is the data based on an average of three runs for example pretty much all the tech sites and channels that I'm aware of base their numbers on at least three runs and most of them make this known but yeah make sure that the data is based on an average of multiple runs and not just a single run as I noted earlier any small changes to the BIOS can impact performance so you need to ensure that the configuration remains exactly the same throughout all your testing I usually check CPU and memory performance before benchmarking games with software such as Ida 64 that's really good for doing all that kind of stuff and yeah it just allows me to make sure the system is where it should be you also want to make sure software running in the background such as Steam for example isn't downloading or installing any updates as this again can impact performance by a few percentage points also it's good benchmark practice to start over from scratch when you find strange results I only did this about half a dozen times when I was testing horizon for the first time fun stuff that was that's also a good idea to ask others in the industry what they're seeing it's my experience that a lot of the guys in this industry are very helpful and if you have some odd or strange results reach out to them tell them what you're seeing and they will often check it in a timely manner for you so yeah that helps avoid publishing anything that may be incorrect anyway it is good to see guys like Bret and his team working to keep hardware company such as in video on their toes so definitely don't want to discourage that we just have to be a bit more careful about how we deliver the message most people won't make it to the end of a video to catch the punchline and on that note I'm going to end this one as I'm probably only talking to about 50% of the viewers that we started with enjoy the video be sure at the like button subscribe for more content and if you appreciate the we're doing our own box then consider supporting us on patreon thank you for watching I am your host Steve and I will see you next time