Gadgetory

welcome back to harbor unbox today I am again discussing the principle technologies benchmark study it was commissioned by Intel and was accessible to the public after Intel's recent event on the 8th now in a previous video I taught the paid benchmark study apart exposing a number of glaring issues with the results I thought this was something that would very much interest our subscriber base and it certainly caused a lot of fuss with our patreon members in our private discord chat in fact it was a few of our longtime members who requested that we dig into the results to try and replicate them or better yet correct them so that's what I ended up doing so the plan initially was pretty simple we were to dig into this for our patreon members and provide them with the results our results to see how they compared to the results from PT so I did this and what we found was quite shocking so much so that the patreon members demanded that a video be created on the main channel and it didn't take too much to convince me I was pretty much already on it as I said I felt this was something that our subscribers would be interested in and after all it is a very relevant topic to this channel we're quite well known for our in-depth CPU and GPU benchmarks what I didn't expect was for this to plop up quite like it did and since then we've seen a lot of other media outlets pick up on it and some have done their own digging Steve from gamers Nexus has done a lot of good work on this topic and we suspect that GN Steve would have probably been just as quick to the punch as we were if they weren't on a plane returning from the Intel event where they first saw the suspect results that made Steve's hair stand on end at least that's what Paul from Paul's heart which tells me anyway so I'm going to assume you've seen the first video that we did and that's why you're watching this video having said that though let's just quickly summarize a few of the main issues that I raised in that previous video I was upset that Intel went ahead and published their own testing done suspiciously through a third-party ten days before the real reviews would be alive effectively blocking reviews from refuting the 9900 K data this was my main issue with all of this not so much that the PT results looked off but the Intel was publishing them well ahead of independent reviews after the numbers I concluded that it was quite obvious the principle technologies arise and results did not align with our own findings and I wish to point out that our own findings do align with the results of the majority of the tech channels and tech websites out there I went on to say that I didn't have too much of an issue with Intel commissioning the report itself and I noted that the principal technologies report was very transparent as they clear standard how they conducted their testing again my main gripe was with Intel and how they were exploiting this situation we believed to fool consumers and booster preorder sales of the ninth Gen Core processors as I said since then a number of media outlets have picked up on the story and others have even done their own investigations and added to it PT has since responded as as Intel Intel's response was typically corporate while PT are defending their work and credibility which is understandable and it sounds like they are attempting to correct their wrongs so it seems at least on their end this could just be a series of honest mistakes made by Principal technologies very the more we learn about them the less likely that seem as possible in spite of that though I am willing to give them a chance to sort the numbers out and make this right having said that though that probably wouldn't be probably wouldn't be as much of a big deal as it is if this benchmark study hadn't been made public ahead of the release of the ninth Gen Core Series now we mostly just wouldn't have cared anyway let's just assume everything is above board here which may very well be or at least was intended to be possibly a big stretch on my behalf but for argument's sake let's just go with that for now Intel paid PT to conduct the independent benchmark study that could be used to show partners and media how much better the core o nine ninety nine hundred K was than competing products Intel could of course do their own benchmarking and provide media with the review guide as I often do but most of us just ignore those results and use them as a rough guide which is what we should do so Intel pays PT because they have roughly 16 years of experience and are trusted by their clients to do a good job Intel would have outlined that they wanted an in-depth benchmark using stock CPU or memory frequencies something along those lines PT then goes away ducks the benchmarks puts together a detailed PDF that they then submit to Intel for approval again something along those lines intel approves the PDF and provides it to media at their event at which point it's no longer under any kind of embargo and can be distributed freely just as we saw now in that scenario I have no issue with what PT has done they were commissioned to carry out a benchmark study by Intel and they did just that nothing wrong there in our opinion the study was deeply flawed but let's ignore that for now assuming my hypothetical scenario is accurate or roughly accurate PT has been thrown in some pretty hot water shortly after their study entered the public domain and irate Australian tore it apart it was quite obvious there were some serious mistakes made when testing the horizon 720 700 X a key CPU in the comparison now didn't L know the 2,700 X was well down on where it should be I can't believe for a second that Intel didn't know exactly what their competition was capable of and how the horizon 720 700 perform of course it's not in Intel's best interest to correct PT I suspect these results were probably seen as a gift as they're basically a best-case scenario Intel aside for a moment this as I said puts P T in some seriously hot water as their integrity is called into question either that or their competence there's just too many basic but very serious mistakes made here and Steve from gamers Nexus has done a good job of highlighting them as have our viewers those of you not to speed issues included using different memory settings for AMD and Intel platforms though that now appears to have just been a mistake in the documentation the memory configuration though still wasn't optimal horizon the use of canned benchmarks almost exclusively for CPU testing is an ideal they also used an inferior cooler to test the horizon 720 700 X and they didn't report the average of multiple runs they used questionable quality settings and the biggest one of all and the main one that sticks out at us was the fact they disabled half the cause in the 2,700 X essentially turning it into a quad-core and therefore placing it at an unbelievable disadvantage now in my initial video the bulk of my frustration and criticism was directed at Intel for creating this situation I felt that the PT testing was seriously flawed and did not come close to reflecting how gamers would configure their eyes in seven systems and well we now know that to be true but my focus and frustration was primarily on Intel frankly I'm not really that interested in taking PT to task they got paid to do what they do and if they made a series of honest mistakes then fine it's well worth noting that if P T's results were released after real independent reviews were alive and we really wouldn't even be talking about this it might be something that Tim chuckled about for a minute and his news corner segment on a Friday but really that's about it Intel commissioning misleading benchmarks certainly isn't great intentional or otherwise but it's reasonably harmless once the internet has been flooded with accurate information so P teammates and mistakes and they're now responding by looking into them co-founder of principled technologies bill Catchings reached out to us with an official response under the responses to inquiries header they said the following we have received a number of inquiries regarding the testing methodology we used and the potential for bias in favor of Intel we're providing additional information as transparent as possible and help allay these concerns they are investigating them all key issues a three of which include the use of game mode the choice to use the AMD box cooler and then the memory speeds they plan to re test with all cores enabled and will update the report with those results so that is good news I hope they do retest with the knocked or Nhu 14s at this point they have justified why they use the Box cooler but that doesn't really make any sense but it's certainly far less of an issue than disabling the cause they also justified the choice to use for sixteen gigabyte modules on all configurations not an ideal setup for me if the XMP settings weren't loaded though they now claimed that they were so I don't have too much of an issue with this the other issues such as only using a single resolution and medium to high type quality settings well I guess ok not everyone can test a wide range of games at multiple resolutions and quality settings so let's keep it real anyway bill wrapped up he's a response by saying but our goal is always to do the right thing and get the answers right we're currently doing additional testing we will share that data and we'll certainly call that if something is significantly different to what we've already published we are confident in our test methodology and results okay not to drag bill and Petey over the coals again but the testing methods used don't come close to meeting the standards used by respectable media outlets in 2018 not to mention they don't come close to representing the configurations used by the vast majority of gamers that said I look forward to seeing their updated results because they had better believe that enabling the other 50% of the 2700 acts will lead to significantly different results significantly better results as for Intel well I don't have anything more to add from the previous video but as I've said I'm mostly blaming Intel for creating the situation and not principal technologies and that said their credibility has been called into question and rightfully so over on the Intel website we find a page titled performance benchmark test disclosure as a seriously long page but right at the bottom we find this principal technologies benchmark disclosure Intel as a sponsor and member of the benchmark X PRT development community and was the major developer of the X PRT family of benchmarks principal technologies is the publisher of the X PRT family of benchmarks you should consult other information and performance tests to assist you and fully evaluating your contemplated purchases basically on that page it stated that PT is used to provide performance claims for a number of Intel 7th and 8th gen core products biased performance claims then Intel paid for and to cover their backside legally they are forced to disclose that at the bottom of the page it seems as we suspected PT were acting in accordance with guidelines provided by Intel I'm starting to suspect that they were told to use the game mode with the 2,700 X what they do or don't do next will be very telling in any case Intel's clearly been forced to state that this isn't an independent or reliable source of information and as such consumers should look elsewhere to receive purchasing advice moving on to wrap all this up half the main motive behind exposing this wasn't for views as bill the co-founder of PT suggested in his interview with gamers Nexus a rather it was to alert consumers that the 50% performance uplift claim that was made by PC gamers N and a news article that was unrealistic on that note though I would like to just quickly point out that PC gamers and were quick to act and they corrected their news story shortly after reviewing my findings so kudos to them in the words of some famous organizations are you part of the solution or part of the problem my suggestion for solving this and avoiding any risk of confusion the future is to keep any results of a commission test under embargo until independent reviews go live simultaneously if you're going to use PT to create a review guide or help prepare internal marketing and sure do that but what was done here was unacceptable and anti-consumer if you're interested in learning more about how the core i7 87 arcane rise in 720 700x really compare we have a number of videos covering this subject in extreme detail you can find benchmarks covering well over 30 games at 3 resolutions comparing both the average frame rate as well as frame time performance based on an average of 3 runs mostly focused on ingame custom benchmark passes you can also get a strong sense of how much disabling caused in the 2700 x impacts gaming performance by looking at our in-depth video titled is rise a more future proof for gamers there's loads of great information and results in that video and it'll give you a good sense of just how unfair the testing conducted by PT really is I've heard it claims that these big companies will start to do more of this in an effort to control the narrative and as a result mate reviewers such as myself redundant that's obviously not going to happen for a few reasons most notably of which is you the viewer and we work hard each and every day to earn your trust and as a trusted resource it makes sense that you'd seek out our benchmarks and not those commissioned by a corporation on that note I hope some lessons have been learned here I successfully shut down I'm I suppose you can even say destroyed any attempts to mislead consumers with the PT benchmark results so will we see a repeat of this from Intel with their next big product launch or any other major company for that matter in this industry well if they're smart simply no because you can rest assured that will jump all over at once again along with everyone else who called the results into question anyway hopefully PT finds what they should with the rest of the 2,700 X turned on they amend the document and we don't have to discuss this again and I can get back to doing some real benchmarking if you did enjoy this video be sure to hit the like button subscribe for more content and if you appreciate the work with your Haren box then consider supporting us on patreon that kind of backing is very useful for when we have to buy processors which we may have to do in the near future thanks again for watching I'm your host Steve see you next time