Gadgetory

do I look fresh and ready to go because I just stepped off a plane from Intel's event in New York and man the tech media was abuzz and the final hours they're about principled technologies you may have heard of them Intel commissioned a company called principal technologies to independently benchmark its 900k CPUs and it did so against other CPUs including a and these so it's competitive benchmarking not just within the same new generation for example so this got then published to the official Intel newsroom thereby effectively endorsing the testing that was conducted Intel seems like it trusts the data and published it and put its name on it so these these documents that were published are effectively Intel's documents we can put the possessive s on there because Intel has taken possession of the documents and has published the data and references the data in its own news room so the reason I'm pointing this out and it's important is because in a situation of gross incompetence by a third party like principal technologies which we'll be talking about in great detail today the company that commissioned the work can then hide behind that incompetent company principal technologies and say we didn't do it we didn't do the testing I don't know I don't know man they did it push the blame over to them but in this case there's a certain level of responsibility that you must take when you are branding something with your name and publishing it on your official newsroom website while you have a room full of media from all over the world so in today's content then we'll be walking through every single point of Fault in the document painstakingly focusing almost entirely on the flawed test methodology we don't even have to validate the tests although we have data for most of them anyway but we can speak to the flawed testing from the get-go so let's do that before that this video is brought to you by us and the gamers Nexus store at store doc gamers nexus net or you can grab one of our other products like the Raglan 2-tone hoodie which we have in stock or the full gamers Nexus logo t-shirt that we just restocked in cotton and tri-blend flirtin more at store gamer's nexus net quick update before getting into this video we had it all done but Intel called us and provided a statement and so that's important to provide before we get into all the reasons why this testing that was bad anyway the statement reads quote we are deeply appreciative of the work of the reviewer community and expect that over the coming weeks additional testing will continue to show that the ninth gen Intel Core TM 9900 K is the world's best gaming processor principal technologies conducted this initial testing using systems running inspect configured to show CPU performance and has published the configurations used the data is consistent with what we have seen in our labs and we look forward to seeing the results from additional third parties in the coming weeks now it's still inconsistent with what we've seen and also all of the other problems with the content still remains so we'll go over that anyway in this video but Intel provided that statement after we had rendered in everything so we thought might as well include it and rerender it so that Intel has their side in here as well and another note here just before you really get going in this we found out principal technologies is about 25 minutes away from our office so at this point we'll already be driving over there we're just gonna show up and see if they let us in and see if we can talk to them about their testing configuration so you're on some of our own tests look at their data stuff like that so we'll see if they let us in probably not but this video will go live hopefully just around when we're arriving so they should have any reason to shoo us away just you head back to the original content first of all it's it's unfortunate that this is even happening right now because the 9900 Kay launched what otherwise smoothly I don't really have any problems with it as an event which is really rare so the launch was pretty smooth and you know the company's stuck on an old process they're stuck on 49 I mean they're plus plus plus plus plus plus plus but they've still made improvements that consumers wanted they want with solder they've they've listened to people and they've done things to advance what they have to work with and the solder yes it probably helps them to get another one or two hundred megahertz in a technology where they otherwise can't but it's still progress and to then have all this soured by the principled technologies publication which and I'll tell you why why this annoys us it's exceedingly deceptive two points of to the point that we have to question all of the data in the document because you know just look at their ashes of the singularity results they're like no one's ever seen testing with those kinds of results in ashes the results for the rise in CPU are mind-blowing Lilo and this is coming from the guy who was pitchforked for his 1,800 X review so before one post in the comments that we're biased towards AMD or something I'll remind you of that incident this is not anything short of calling out poor testing when we see it that's what this is ashes of the singularity it's not really a game it's a synthetic benchmark and Intel does pretty well in games they do well in gaming benchmarks Intel doesn't need to fudge numbers to look good in gaming scenarios they already win whether or not you agree with the margin to which they win or whether it's a relevant amount of frames per second you get extra that's irrelevant right now all that matters is that at core and thread parity with a frequency advantage which they have Intel will outperform AMD in an unconstrained scenario so then why why fudge the numbers ashes of the singularity and maybe they didn't maybe it was just completely incompetent testing which actually will go through that soon but Ashley the singularity should post pretty damn good performance on Rison and it's because it's a synthetic benchmark that's programmed to use all of the threads it's like the only game that does that so it's it's odd that this particular game jumped out so much but it's not the only one so anyway the point was that this was an unfortunate thing to have happen considering the otherwise smooth launch of all this and Intel's got its name on this so they even if it's a third company they they are ultimately the one that this report goes through intel has QA teams they could have validated this before publishing it but let's let's go through it so let's tear it apart and grow through the bench methodology principal technologies tested eight CPUs here including the 9900 K 8700 K 886 k99 ATX II which isn't due out for another month the 2700 X and more it also selected 19 games for its testing as you can see on the screen on page one of the report principal technologies demonstrates its complete lack of understanding for how to test games by stating the following we use the in-game benchmarks where possible otherwise we used fraps to record frames per second FPS to state that they're mixing software to record framerate and that they are using in-game benchmark reporting is already flawed in that most in-game benchmark FPS trackers are not all that accurate and sometimes average differently from fraps here's what's also confusing principal technologies also takes the median of three runs rather than an average which is a very odd choice when considering the sample size is three and that the benchmarks all have variants there's a reason that we have standard deviation markers and our charts so to then pick the middle point of multi sample testing is completely eliminating the purpose of multi sample testing which is that you can average the data and eliminate outliers if you see any and then rerun the test if you need to not to pick the middle result that makes no sense next part for Civilization six testing and this is one that's close to us principal technologies demonstrated supreme levels of not knowing what they're doing by running the graphics benchmark to their credit they also ran the AI benchmark which is the only useful benchmark for CPU testing and sub six but we previously published data illustrating why the graphics benchmark is more or less entirely irrelevant for CPU testing and it's because it's civics you know what the graphics benchmark does in sip six on the slower CPUs it will get to the next frame struggle to process the turn on the CPU because it's CPU bound and stare at that frame for seconds on end what's the GPU doing on this time it's spewing out frames like crazy cuz nothing's changing on the screen so depending on how they tested this and I don't know how long they allowed it to run I don't know if they let it run the entire duration or what they did for the graphics benchmark it probably did depending on how they tested it though especially if they set a specific test duration and then cut it off at some point the actual result will be that slower CPUs will perform better than they should in any other benchmark because the GPU is a frame rate or while the GPUs ability to spew out frames is what I should be saying while the CP is trying to process the next turn is going to counteract any performance deficit so that is a tremendous fall on the testing methodology and this is a a genuinely bound to the CPU game for the most part but if you're on the graphics test that does change quite a few things so if I don't know what they did here but the next one so further know that principal technologies and this is a weird one routinely presents data down to even two significant figures which is absolutely hilarious now I'll admit we've done this in the past but the reason we've done it is because I forgot to remove one of the significant figures when we're putting out the data the way they're doing it is seems like it's based on which ones use an in-game benchmark and which ones don't so it's you can kind of probably looking at the significant figures they have on each one you can kind of figure out their method of what they used to log the data because we know which which games do and don't present to that level of accuracy and we know that it's not an averaging thing because they say that they don't average it they use the median so like that's the other thing at least at least when we've done multiple significant figures that actually we're not that significant we averaged a lot of data so at least it makes some sense because you've got math going on where you end up with weird splits of numbers but in this instance it's three test runs supposedly cut down from three to the median so you cut out one then you cut out two and that leaves you with one piece of test data and so how do you with my stomach comfort why is it significant so it's it's a it's a very weird way to present the data let's go over some BIOS settings then maybe that'll be sobering wake us up a bit for BIOS principal technologies used different memory frequencies across all of the platforms they use 226 66 megahertz for Intel and 29 33 for AMD despite using all the same kits although this sounds better for AMD the company made sure to negate any frequency advantage by absolutely butchering the timings which are chiefly important to all benchmarks particularly on and the platforms another quick thing that Intel noted and this is again Steve from the future at this point being edited in is that actually do CP was enabled for the AMD product it just wasn't listed anywhere the testing methodology so they still step down the frequency without really any explanation whatsoever and then they did enable the OCP which is a functional XMP but all the other problems with memory timings and motherboard variants between memory kits and motherboards still remain so that's still a concern even though it is perhaps less of a concern than it would be otherwise back to you Steve from the past and we've even we've demonstrated 30% scaling by using the same kit of memory with XMP and nothing else just XMP just like they're kind of doing here on some platforms moving it from one board to another and the reason that happens is because boards are all over the place with memory times that's why you standardize and you try to pick your boards carefully and make sure that the timings are pretty fair because if you have TR efi refresh interval tier fe if you have that one board it could be it could be way higher than i mean you're talking times a 20 percent refresh cycle so TR FC that's another good one RFC we've seen the same kit of memory sitting at 300 RFC on one z 370 board or whatever actually sounding board doesn't matter you put in another one and it's 700 that's a big difference in terms of forints that's pretty huge for gaming so to not control for timings in any way whatsoever is really amateurish is is the nicest way I can put it so very strange decision or one which is completely uninformed which is more likely and then next by the way if you want to see some of that data for how timings can impact things motherboard to motherboard check out our what our RAM timings video for primary x part one principal technologies I would recommend you watch it you might learn something memory timings are extremely inconsistent there and that was a big problem for us so moving on from every timings and on to things like thermals which we also know decently well I would say we have more concerns all of the testing was done inside of a thermal take case not one of the better ones it was one that more or less resembles a an r6 for example so it's got poor air flow and this might not matter except that the AMD CPU the 2700 X was for some reason tested with the stock box cooler which is okay as a cooler I guess but the all the other platforms supposedly we're tested with the noctua Nhu 14s which is one of the best air cooler we've tested recently so what happens here is the eighth and ninth gen products end up on a really good air cooling solution versus a stock solution that's completely unfair like okay so the intel parts didn't come with a box cooler why is that an excuse you're the if you're like here's okay so here's the thing if there are argument is the AMD came with this one so we should use it in our testing then the counter would be that the intel parts didn't come with one and so we shouldn't use one for testing either stick to the same thing use what's in the box i eat nothing or use the same cooler for all of them it's completely unfair yes it's crazy so what's another way it's it's it's almost like I don't know some of this stuff so the company fails to mention control of fan speed at all it's kind of another really important thing which leads us to question whether the fans were plugged into the cases fan controller which would be an accidental blessing and that they would be probably somewhat controlled with these in the bounds or into the motherboard in which case who knows who knows what it does depends on the motherboard and they might be in fast mode if they're plugged into the case controller or something if they're lucky it seems like the only way this company would do anything at this point competent for testing would be by accident and so we will allow for this possibility that maybe the fans were controlled properly by accident but otherwise it is not mentioned at all and the 2700 x ends up without the Noctua cooler there's no there's no mention of what the threader for CPUs used that we saw anyway unless you missed it but if they strapped a 10 hu 14s on there that one is not specifically made for thread Ripper unless they use the tr4 model and that one doesn't the non gr4 model doesn't cover the whole IHS anyway you can make it work certainly we've done it but if you yeah that was pretty funny when we didn't do it though but if you do that it doesn't cover the whole IHS there's a big difference then performs thread rivers so very very odd decisions and again air cooling for the CPU so the fan for the CPU coolers also left unmentioned we don't know if that was Auto or what it was next up eight different systems were used with eight different sets of hardware including GPUs so even though it's technically the same the same well we know it's a 1080i for the GPU theoretically it's a gigabyte 1080i who the knows well who knows what the hell that means we don't know which model gigabyte card it is just a gigabyte even if let's just say they use all the poorest gaming whatever three fan car they use the same exact model for all eight systems that's still flawed anyone who's tested things knows this so how do they not and the reason that's still flawed we've tested this we have multiple things like multiple FTW three is multiple SC twos and you know at one point when they came out like a year ago we said hey what if we take these multiple cards and we try to get simultaneous benchmarks going for CPUs and the takeaway was yes we can do that but only in completely 100% for sure cpu-bound tests ie blender ie Cinebench maybe something like potentially ashes but anything that even for a second dips out of 100% CPU load and goes to GPU load you start running into issues of Pascal and silicon and die quality to a point where on the same card and ji3 census keep saying ie J's $0.02 taught in that since literally J's - sentence J published content on this a while ago and we've seen a - where between - same model cards and Pascal you can have up to a 5% difference in performance pretty massive and so to have that maybe between two benches you can kind of reconcile this with proper testing and we've done that too but between eight what what's your variance then one might be 5% faster what if one is 5% slower than baseline you might have a massive Delta there and for no good reason the principal technology seems to be missing this specific core principle but there's more still for memory kits of for by 16 gigabytes were used in every system which is completely insane 64 gigabytes of RAM with a 2700 X and an 80 700 K really really so they either whatever the reason 64 gigabytes of RAM that's fine but here's the reason it's actually not fine the reason it's actually not fine is because as we said in the memory it's you like they're just completely chaotic sometimes there's XMP sometimes they're not sometimes they're dropping frequency sometimes it dropping it to a different number than the other number and so the reason we kind of came up with for what's the motive why would you do that short of just genuinely having no idea what they're doing is because that but also they try to make 64 gigabytes of RAM work and if it's not validated for the board or it's just because 64 gigabytes of RAM it might be kind of hard to get just straight XMP sometimes the whole and we've seen it I mean it's 64 it's a lot to ask from a system so it's possible that they were having issues with the higher frequencies and or the higher timings and defaulted to garbage and ran that instead which is an invalid test running properly dual or quad on applicable systems like a GT platforms we don't know if they did that we don't know if they use the right slot and while these things seem extremely basic and one would assume that everyone does it I mean when you read or watch reviews from any of the major outlets in the space you kind of assume they have the memory in the right slots but given the level of genuine incompetence here you have to start questioning everything everything they did it seems like they've never done this before so let's get into software next that seems fun so first off the Intel Tara obvious application was installed these when possible rise and master was installed and set to game mode the company used the these things are not terrible by the way but the company is the exceptionally extraordinarily flawed benchmark of final fantasy 15 which we have railed against time and again and discovered originally that was us the problems with it of not kollene objects and when we saw that with ansel we then went in and validated it with render doc which shows that it's constantly rendering basically every single object in the game ever even stuff that you're never going to see that's that's literally you have to like fly through mountains to get to it and it's not an issue with Ansel rendering stuff and this is going back to old content render doc showed it to and to prove a point Square Enix came out more or less indirectly apologized for their terribly flawed benchmark and updated it and they didn't fix everything but they fixed the hair works LOD scaling issue and they admitted it was a problem and NVIDIA knew it was a problem but everything else remains including the lack of culling objects that are out of view the lack of calling objects you'll never see the poppin issues the LOD issues on things that are really friggin far away so the the f-of-x me benchmark is not a useful benchmark it's not it is trustworthy whatsoever and also it is GPU bound almost every time but here they've use it so they didn't get the memo I guess but there's also the company's ashes of the singularity results and this is where hardware and box Steve alerted us to these via Paul I was with Paul and Kyle and we all started talking about this and they are on PC worlds videos talking about it as well and the results trash to the singularity as I mentioned earlier are just completely wrong everything about them is wrong we already talked about it earlier in this video but Andy does a lot better than is presented in this data if the environment is controlled properly so maybe these are genuinely the numbers that principal technologies got that doesn't mean they're right they'll be right for that set up but nothing else so there's more for GTA 5 although the company explicitly specifies every other setting used for every other game and setting presets for those games they do not specify anywhere what they use for GTA 5 other than 1080p and ovc and GTA 5 doesn't have a preset you have to go through every setting and set it manually and also GTA 5 will change those settings with a hardware change that's important to know because if you don't know it's doing that it can change something that you might not even realize changed not like resolute it could be grass quality and that has a big impact on performance so we have to question whether these are valid results because they don't specify if they even know that they have to do this and like I said normally with let's just take heartburn box for example or I don't know an intact or someone like that if one of those two sites or YouTube channels runs tests and something like GTA 5 I can trust with pretty good certainty that they did it properly and that they set the right graphic settings that they know they might change and that they're controlling those things I can trust that with relative certainty that's because those outlets have gained that trust principle Technologies has Illustrated that there is no reason to trust them and so we must call everything into question including the GTA 5 testing finally there's a big disclaimer at the bottom that reads the following disclaimer of warranties limitation of Liability principle Technologies Incorporated has made reasonable efforts to ensure the accuracy and validity of its testing however principle technology specifically disclaimed any warranty expressed or implied relating to the test results and analysis their accuracy completeness or quality including any implied warranty of fitness for any particular purpose all persons or entities relying on the results for any testing do so at their own risk and agree that principle technologies its employees and its subcontractors shall have no liability whatsoever from any claim or loss or loss or damage on account of any alleged error or defect in any testing procedure or result parenthetically and I'm adding this of which there are many and then continues to say in no event shall principal technologies that be liable for indirect special incidental or consequential damages in connection with its testing even F advised of the possibility of such damages and no event shall principal technology's liability including for direct damages exceed the amounts paid in connection with principal Technologies testing basically a refund customers sole and exclusive remedies are set forth herein and also it says elsewhere and this may have been in the Intel counter documents that was included but there was another statement that said software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors Wow so is that it nope there's one more one more there's another and this isn't too uncommon it's just the absurdity of everything else leads us to it Intel there's a quote Intel be marketing the Intel registered core TM i $9.99 hundred K with the tagline quotes performance unleashed close quote in certain jurisdictions including PRC and Vietnam Intel will be marketing the Intel Core 900 k with the tagline quotes Intel's best gaming desktop processor end quote in certain jurisdictions including Argentina Belarus Belize Egypt El Salvador Guatemala Honduras Italy Japan Panama Peru Saudi Arabia and Turkey if you are media or an influencer from these countries or other communicating directly to residents in these countries eg on local language social media please only refer to the tagline Intel will be using in that country and LU of the claim on this slide or document which only I said this this particular thing isn't malicious we've seen it before all the companies do it it's some weird legal mystical marketing thing that apparently is necessary and no one actually has to follow it's just no one wants to be told what to do either and you're gonna have one to haggle on in one country I don't know I don't know what what the deal is but it's just you know in light of everything else it does seem oddly it seems odd seems weird so that's that sounds like the whole situation is just weird and it's very disappointing because I'm pretty excited about testing some of the stuff coming up not gonna say the products not really excited for the product directly but for testing a lot of it there's some cool stuff Intel's done and it deserves being looked at by someone who knows what they're doing not these people not not these people so I why why did you do it Intel why did you do this to yourself it went so well the event was was properly organized you didn't have Jensen on stage talking for three hours you didn't delay meeting outside for two hours in what the publican first you had like useful demos and people on site to talk to and everything was ready to the point and we got the information we needed we got it quickly for the most part except for well whatever we'll go over that later but then then I I leave the event go back to the room prepare to leave and I'm presented with this thing let's just saw this that was just whatever why why should do it anyway we'll be reviewing the 900k probably I think they sent it they did because they're not going to now but I don't know about that this is Intel's been pretty decent as hangout samples but the point is jokes aside we'll be reviewing it and it will be reviewed in a vacuum as it should be for purposes of testing and seeing if it's any good checking the value all that stuff um I could consider this mess when we review it but of course it's important to present the mess to you so that you're aware of what the companies are doing because sometimes it's pretty screwy so an Intel for what it's worth did not conduct these tests but did Commission them and then find them valid enough to publish and stand behind to the extent of writing documents with sub notes referencing these tests and saying that you can reference the document this document for that data so intel's got its name on it they've taken responsibility insofar as publishing the data on their own website or at least referencing the data on their own website so you can't hide behind I didn't do it it wasn't me anyway you like yeah yeah I could present I can't think of any nonviolent analogies here so I'm just gonna keep them to myself but that's it for this one subscribe for more you'll get a lot of it you'll get our testing too it'll be more or less in a vacuum not a Dyson one they don't like us and go to start out here insects design has become one of our mod mats one of our shirts if you'd like to support its directly because it helps a lot what would we do stuff like this and patreon.com/scishow sacks is for that Avenue as well also a quick note I was in a PC world video on their YouTube channel you should check it out it was a lot of fun and thanks to Gordon for that but anyway come back for more I'll see you all next time