Gadgetory

silicon quality and the so-called silicon lottery are often discussed in the industry but it's rare for anyone to have enough sample size to actually demonstrate what those phrases mean in practice aside from well maybe Silicon Lottery the website we asked gigabyte to loan us as many of a single model of video card as they could so that we could demonstrate the frequency variance card to card at stock the variations in overclocking Headroom and actual gaming performance differences from one stock card to the next this helps to more definitively strike out the question of how much silicon quality can impact the GPS performance particularly again when stock and also looks at memory overclocking and range of FPS and gaming benchmarks with a highly controlled bench and a ton of test passes per device finally we can see the theory of how much one reviewers GPU might vary from another's or from your own when running an initial review testing before that this video is brought to you by Thermal Grizzlies conduct a not liquid metal conductor not as what we've used in all of our liquid metal and delayed thermal tests capable of dropping TV thermals significantly and replacing the stock thermal interface over CPU thermals don't just allow better over clocks but also lower noise levels because the transfer efficiency is increased the mix of gallium and indium makes for a thermal conductivity of 73 watts per meter Kelvin outclassing traditional pastes significantly learn more at the link in the description below so this was a pretty exciting video to work on because it's not common that you can get so many of one device at once we borrowed these to make that clear so a gigabyte has loaned these to us for this research because this is not to our knowledge not something that's been done for GPUs at any meaningful scale and seven is it's not as many as we wanted but it's as many as we could get still pretty good sample size though and all we're gonna look at is things like average frequency out of the box without any overclocking and see where does that cap out because card to card each one will have a bit of a different VF revolt frequency curve and that's determined by largely by silicon quality and then from there we can look at overclocking Headroom and the actual impact on gaming performance so where this has been done a lot for CPUs and looking at CPU overclock in frequency Headroom this is a at least to our knowledge one of the first opportunities to see this for GPUs in a third-party environment so pretty fun stuff the topic idea actually arose from two things one of them was sort of silly it was a comment on one of our Xbox benchmarks ages ago where they it was a console player who hadn't really seen our content nor knew who we were and said that we only had a sample size of one so the tests were invalid which is I mean it's just that's not really how it works with the console but the thought arose of well okay so with GPUs if you have a sample size of one how reasonably can you draw a conclusion of a stock card just ignoring the overclock inside of it that's a bit more variable so we're looking at that partly and then investigating some of the overclocking Headroom and largely debunking some of the concerns about again sample size of one not being enough for a stock conclusion or even an overclocked one so anyway a testing method it's pretty concrete here we'll define them more in the article below but this is our previous test bench from before the super launch not that it really matters much it's the same Banshee the way and using the older games we did more test passes than typically so we typically do a minimum of four here in some of the tests we did up to 15 like with some of the 3dmark testing we did we also had a minimum duration of 30 minutes for each of the frequency tests to allowed the cards to - so can heat up and reach a target temperature level and then we also for overclocking we raised all the fan speeds to 100% we got the cards - about the same temperature plus or minus one degree Celsius or so and overclocked it until it reached instability and then we ensured that the overclock would hold for a couple of hour for about four hours or so through an overnight test or during the day test so it's more strict than our review guidelines where we just we needed to hold for the duration of the review so it's bit stricter in that regard and that gives us a more realistic look at what a user might sustain for an overclock rather than something you just need for testing and further we've gotten the standard deviation here run to run in most of these games down to about 0.1 FPS average the under sent loads which are always the most variable or in most cases down to about 1.1 FPS average in the 0.1% deviation so there's still some margin of error here baked in of course there always is run to run variance but a lot of the time and we'll talk about this more as we get into the numbers the variance is less than 1 FPS like over an average of the benchmark so we'll see numbers like 1 13.5 8 to 1 13.7 7 for example for a range of of results across multiple passes also you'll see the last few digits of the serial number and some of the charts that was just for us to internally identify which card is which when we were doing the testing so let's get into it we'll start with some of the frequency stuff that'll be the most interesting we'll have some game benchmarks at stock so you can see the differences if I guess the idea here is if you buy one or if you're watching multiple reviewers videos assuming none of us get a car that's just straight defective which happened here by the way we have one then how much range can you expect from one card to the next stock so let's get started doing an overtime frequency chart with this many GPUs would be impossible to read as you can see in this quick example because GPU boost on Nvidia cards means that there's almost never a flatline it happens but it's rare it's typically tied to an unlocked b bios and a very cool GPU like in the 40s to 60s degrees celsius typically close to the 40s for the most part each card will fluctuate based upon thermal power and voltage limiters and so we end up with this mess when we plot so many devices instead we're resorting to a simpler bar chart with averages across 1,500 cells of an identical load scenario we get a much more readable format with this chart the out-of-the-box performance establishes a stock frequency range of about 45 mega Hertz from top to bottom we control the ambient temperature and other variables so this comes down to actual silicon quality and variance of the GPU itself for each GPU a vote frequency table is generated that is used to establish the frequency stepping under Nvidia's boost parameters it'll run out very slightly different voltages and frequencies and clocks can't dial in down to say 1 megahertz so you typically see a range about 10 to 12 megahertz steps the best GPUs on this list in terms of the out of the box stock frequencies are GP is 4 5 and on the worst is GPU - although will later show that GPU 3 is actually defective making it the one that's really the worst something you'll see in the game benchmarks but not in the frequency testing so this is a matter where they all technically have the same spec on the side of the box or on the website so they're all hitting the boost spec the base spec but this is additional boosting Headroom based upon again thermal power and voltage and then silicon quality being the one we're looking at today note that the best stock GPU doesn't mean the best overclocker and the best overclock iran air doesn't mean the best overclocker on liquid nitrogen as kingpin of EVGA has told us in the past the next chart shows the maximum overclock for each card beginning to establish the start of the silicon lottery demonstration the maximum stable GPU frequency was on GPU 5 running at an impressive 20 72 megahertz for the 1070 TI the next best was GPU 4 at 2048 megahertz than GPU 7 at 20 19 megahertz note that GPU one has a higher stock frequency than GP 7 but GPU 7 has a higher overclocked frequency also note that this is the stable frequency after a few hours of burnin so it's an aggressive bin rather than a bin we would do just for a review duration GPU 3 has some serious issues but they're deeper than the frequency you're seeing here GPU one would be disappointing in the face of GPU 5 as 70 megahertz is actually beginning to be significant for benchmark performance perceptually in terms of gameplay you won't notice much if anything between the two but it would show up in the chart finally for maximum memory overclocking we ended up stuck around the same as 600 megahertz offset for each set of VRAM GP is one two and three hit twenty three or four megahertz with GP is five and six getting a bit higher landing at 23 52 megahertz for those gb of four manage 24 hundred megahertz impressively and GPU seven got stuck at a measly 2194 megahertz a definitive worst and a few notes here - you'd have to multiply this by 4 to get the effective speed that might be the number you're more used to we're showing the actual speeds and separately these are the frequencies after we did validation to make sure they weren't tanking the zero point low performance with memory errors that you don't see by just looking at if it's quote unquote stable in times by for example so these are also longer burn in numbers the frequency range tells most of the story so far for this comparison but games will help illustrate the rest knowing now that our stronger performers in the chart are GP is four and five and sometimes one in terms of average frequency over time we can see if any of those differences correlate with meaningful benefits that exceed test variation for the stock performance and note again these are from our old GP test bench results as they were attested before our TX super or Navi came in so you're not looking at the newest data but it's just for the 1070 TI is that we're testing on mass today so it doesn't really matter for GTA 5 at 1080p we measured the average results to be within a range of 1.3 fps of each other once subtracting out the single outlier GPU number three is an RMA unit and it seems there's a good reason that it wasn't a harm a unit someone actually had a problem with this one it looks like this one has a defect in the GPU despite being from haribol and clock speed we retested this one three full times and camp with the same results every time stripping the outlier the FPS range is close enough that it's within test error as for whether the better performing GPU is correlated with the higher FPS GP is 1 4 & 5 especially 4 & 5 had the highest stock clock speeds and these GPS also ended up with the highest framerate in this game the data consistency here is excellent to see and our test to test variance has low standard deviations so accuracy is good even though 0.1% lows are within test error and standard deviation with about 5 test passes per GPU per resolution for these cards we can see that there's slight deviation from card to card based on those earlier clock speeds but it doesn't amount to much one stock at 1440p the stack remains very similar GPU 5 and 4 tie for first differentiated only by error margins and 1% loads and GPU one follows these with GPU 6 next on the list our top three performers for clock speed remain the top three for stock fps predictably and GPU 3 is again shown to be a defective outlier our standard deviation is 0.1 FPS average across thirty-five test passes 0.5 FPS average for 1% lows 1.0 5 FPS for these 0.1% lowest standard deviation and then we ran these tests at the minimum of two resolutions sometimes 4k depending on the game for f1 2018 at 1080p our results range is 2.7 FPS average with standard deviation at about 0.3 FPS for the average 1.6 FPS for 1% lows and 0.6 FPS with 0.1% lows performance at lens GP is 4 and 5 again at the top with the next three all roughly tied one for sentence 0.1% lows are all within test variation with only GPU 3 standing as an outlier but at least it's consistent the top to bottom range here is 2.6 percent improvement from GB 1 to GPU for and with an average FPS deviation of 0.1 FPS thats dead-on for each card 2.6 percent isn't a huge deal from a consumer standpoint but it can be a big issue for reviewers especially if you test one card for your initial review and maybe you change it to a different one for a later revisit it could mess up the data if you're trying to compare new data versus old data because the stock frequency could be different enough that there's a bit of a percentage change baked in there that might be all you would see from a driver change anyway so if you're trying to test drivers using two different GPUs test it across maybe a couple months apart the difference is from those factors alone could be enough to overwhelm the difference from a driver change for example so same reason it's best to control GPU speed as best as you can for tests where it needs to be fixed so that's something to pay attention to for reference GPU 1 scored between one hundred and thirteen point four and one hundred thirteen point eight FPS average and all tests extremely consistent GP for scored one sixteen point five six two one sixteen point seven seven and all tests also very consistent and that's the same for just about all of these GPU three again being the outlier because it's defective additionally GPU one was one of the better three overclockers in 3dmark but that performance doesn't necessarily carry one-to-one to every type of workload as you can see here at 1440p the test results for f1 2018 show a range of 2.7 FPS average or a maximum percent increase from bottom to top of 3% consistently GP is four and five are at the top of the list although GPU Wan remains a lower than we might have expected based on the previous results far cry 5 at 1080p is next for this one our range is 105 point six to seven point eight FPS average ignoring the obviously broken outlier bottom the top that's an increase of two percent or a range of roughly two FPS average results are all consistent still for GP is four and five which are tied and within even the smaller error of this test they're actually almost identical and GP is six and two are within our standard error for these tests as well 1440p shows almost the same stack except GP is seven and one trade places and GP is four and five trade places this time shadow of the Tomb Raider is the last actual game we'll look at for this at 1080p the range of performance is 1.8 FPS average total with GP is 4 and 5 remaining consistent in placement throughout all of these tests we can see that there's at least some direct real-world correlation to the higher average clock speed in the 3d mark test earlier but it's not enough of one that you'd ever really notice without a large sample size and a highly controlled set of tests you might start to notice for overclocking as we showed earlier where the range is closer to something like 70 megahertz and some of the GPU to GPU comparisons the only difference on this chart that a user might actually observe would be the defective GPU like unit number 3 the rest are all within average test variance and less controlled environments than our own especially at the end of the day against our sample size here it wouldn't much matter one card to the next as long as you don't get a total limit like GPU number three which clearly has a defect and shows good reason for its RMA even if reviewers got golden samples so to speak which we have more evidence contrary to than in support of minimally the stock performance what would be with an error of what anyone else would get and overclocking performance has some range of course but that's not news to anyone what is interesting is seeing the range Illustrated on the same model card where we scale from 2070 to megahertz at the high end all the way down to 1954 megahertz which is a massive range and again same exact model on these not just the GPU but the video card itself the skew is identical so 100 megahertz an overclocking head room can have an impact on framerate and definitely on benchmark scores but if you fall within the mean say closer to 20 30 megahertz here it'd be hard to differentiate any actual perceptual differences versus the top-end clock even though if you're benchmarking competitively some terrible competitive benchmark frequency but if you were doing it you would see the difference is there it's just in-game you would never realize those as a human player the memory overclock or the 1070 T is also spanned about 300 megahertz range although this will behave differently with G DDR 6 and HB m too clearly so with regard to whether gold samples exist we already knew that answer the answer is yes silicon just works that way some pins better than others and that's all there is to it the further question of to what extent do they exist we have a better insight on their their silicon from this generation that overclocks much higher than these as a further example but this gives you an idea of the range and also things like the cane bin card or the lightning cards wouldn't exist if they couldn't push the higher frequencies on those natively so now we can illustrate to what degree the clocks change even when stock which is important for reviewers because if you're a reviewer and you want to run multiple video cards for a non GPU test bench then it's important to control that variable too because the stock the stock range here if you're looking at 45 megahertz or so will definitely start to chip away at some of the percentage differences in the scores and that matters you could have up to a 2% impact just from changing between these cards not even counting some of the others where we've seen up to 5% so that will mostly wrap it this isn't really a discussion on review samples although that is an interesting aspect of it it's more it meant to look at the consumer higher sample size although not as much as we want but still pretty damn good a higher sample size of cards of what's the range and as far as the review samples go while we can tell you that at least for the most recent two launches from both a in the end Nvidia and the sent out for Radeon seven they sent out the chips that were more on the like sort of the 50 percentile so not great overclockers they were actually on the overclockers which is only a in the-- straight-up told us and that was not because they were trying to send one way or the other it's because it was early silicon nvidia for the super launch didn't check the cards either and we know that because some people got dead cards in the mail so they weren't even necessarily turned on although that much is normally done so anyway that's it for this one pretty cool information if you want to see us do more of this type of content with a higher sample size of stuff let us know what the thin is and if you have some ideas of what you'd like us to look at because we can probably talk to the manufacturers and and get a loaned amount of that device and and send it back eventually but it'd be enough to do the tests thank you for watching subscribe for more go to store documents nexus net to support us directly like by buying one of our mod mats or our new tool kits which are in stock and you can also get a patreon.com slash gamers nexus that you were watching i'll see you all next time