Gadgetory

now some manufacturers can game the benchmarking system because their devices can detect when the app is testing the device and we'll talk about how they do that that's why we partnered up with Geekbench who created a stealth app that allows us to get a more authentic more genuine take on the devices capabilities and let's talk about what we found now for those of you who are not familiar with this we just got to get real basic and talk about what a benchmarking app actually does and in the crudest simplest of terms a benchmarking app measures a device's capabilities it's going to push its power its CPU and its RAM capabilities give it a score to measure that devices capability it doesn't necessarily correlate though to your everyday regular use of that device that's what hands-on reviews are for you see manufacturers can not only design their devices to detect when bench wearing apps are running but they can make sure that their device is seen in the best light better than it normally would in everyday usage and light of the fact that the Canadian junior team is killing it in hockey right now let's use a hockey analogy you see for example if I am a hockey coach I have the option of pulling my goalie and replacing him with an additional offensive player greatly enhancing my team's offensive capability and if you were watching my team at that moment you'd be saying chase your team is great on offense but is that a accurate and fair take on my team's overall general offensive capability no because I've given it an artificial temporary boost that I cannot sustain and that's precisely what devices or device manufacturers can do when they detect a benchmarking app testing or giving a take on their devices capabilities but in our best of 2018 testing we were able to work with our friends at Geekbench to configure a stealth Geekbench app this app at least in theory was able to reduce the chances that devices were able to detect the benchmarking app in progress not all devices on this list showed cheating behavior during both single core and multi core tests for example the HTC u12 plus and xiaomi me8 only show significant decreases during the multi core test the lowest device's identified beyond single noise was a three percent jump in scores but we found an up to 21 percent leap in two devices the huawei P 20 pro and the honor play the graphs here show regular Geekbench scores versus the stealth Geekbench scores from the phones that detected the app and then modified their behavior these results are the averages of five benchmarking runs all of which had slight percentage differences as you see in the mate 20 detail cheaters do best in the regular score in yellow and drop back when they don't recognize the benchmarking blue is the stealth result now this graph here demonstrates how much better these devices performed while cheating but to be fair we should know that not everyone cheats during the best of Android 2018 we tested 30 of the most powerful and modern Android devices the devices we talked about above cheated but that still leaves 24 devices that fought fair and square besides our reference device the mate 20 and the mate 20 Pro the list includes the samsung galaxy note 9 sony xperia x - the vivo x21 the LGG seven thin queue the google pixel 3xl the oneplus 60 and the xiaomi me a - to name a few now I want to bring up the fact that people will say look Jase is this really cheating because this kind of thing is done in many other industries if we go back to the car analogy and you're talking about mileage when you want to buy a car there's the mileage right on the sticker on the car and you buy that car and you don't get quite the the same mileage in real-world usage and that's because that number that they came to is under ideal conditions under almost no wind on a flat surface a smooth surface you know all that kind of stuff but of course in the real world it doesn't work that way I I would submit to you that the idea or the spirit of the benchmarking system is that we get some sort of hint or indicator about real-world usage and I think that's implied and when you gain that system and you don't make it known that you're gaming that system that that is inherently dishonest that bothers me but what about you what do you think about it let me know in the comments below or on Twitter I would like to hear it you know what software starts do things it's not supposed to do who solves those problems it's the unsung heroes our developer teams in fact it's called incident management and our beloved sponsor today ops genie is just for them opportunity helps us minimize downtime which is critical for our customer experience ops genie empowers devs and ops teams to plan for service disruptions and stay in control during incidents it helps notify all the right people through a smart combination of scheduling an escalation paths that take into account things like time zones and holidays it allows for deep flexibility and how and when and where alerts are deployed and supported by over 200 integrations with ops genie your next incident doesn't stand a chance go to ops genie comm and sign up to get a free company account and add up to five team members that's ops genie comm never miss a critical alert again with ops genie sounds like wow