Why and how do OEMs cheat on benchmarking? - Gary explains
Why and how do OEMs cheat on benchmarking? - Gary explains
2017-02-09
how they mind ghosts in from and royalty
now based marked eating has come up
again in the news this time it was one
plus has been accused of cheating the
benchmark scores on the one plus three T
they're not the first company to ever be
acute of benchmark cheating so the
question is why do OEM try to cheat on
the benchmark and how do they do it
well let me explain so first a bit about
the types of people that buy smart
phones I suppose broadly speaking there
are three groups of people that buy
smart phones the first group of person
doesn't really care much about the
specification or the speeds they'll go
into the shop to buy the phone that are
for sale the system they'll say yet the
good phone they like the price they'll
buy it at the end of a transaction done
deal
the second category of person are those
that have a bit of technology
understanding they might understand what
the specs mean they'll understand the
difference between four gigabytes of RAM
and two gigabytes of RAM 16 gigabytes
internal storage 32 gigabytes internal
storage they might have screen
resolution that kind of thing and
they'll make an educated decision based
on what they understand about each of
the phones they may also have read some
stuff all from the internet but it isn't
their area of expertise but they know
enough about technology to make an
educated purchase and I suppose the
third category of person is the one that
is basically the geeks or nerds that
knows everything about smartphones read
all the reviews which is all the videos
knows what's coming down the pipeline
knows what was the best phone and you
know they're the kind of person that yes
yes hi oh hi yeah I'm making it making a
video okay it'd be quick okay yep yeah
it's a good phone it's good phone it
what's the price Oh excellent price yes
great off-contract brilliant okay but
I've got to go okay so thank you bye bye
fight fight fight fight
sorry about that so they're the kind of
person that you ring when you want to
know what is a good phone to buy so the
question is how do we M actually cheat
well what actually happens is when a
device is made
the balance has to be struck between
performance and power usage now really
these two our enemies are at loggerheads
with each other maybe give an example on
your desktop you've probably got an
Intel or AMD CPU it may be giving out
between 50 a hundred maybe more watts of
thermal heat it is got big heatsink on
it it's got cooling fans leakages
running from the mains electricity now
for smart phones at the other end of
that scale they are run by batteries
they mustn't get too hot they've got no
fans on them and basically the more
performance that you want to get from a
mobile device the more heat it produces
and the more battery life it uses so
there's a balance between getting good
battery life and getting a good
performance
now when the device makers make a device
they make various decisions in the hard
way like the processor and the clocks
means the processor and the clock speed
of the GPU and other things like that
and they also make some decisions in the
software now the software can of course
be tweet dynamically while you're using
the phone the phone is actually always
changing the speed of the CPU the speed
of the GPU it's rotting is increasing
there's a whole technology which I've
covered in some of my other videos now
every app that runs has a unique ID and
those unique IDs are part of the
information that's put on the Google
Play Store and so the UID the unique ID
of every app remains the same so antutu
has the same UID today and tomorrow and
it had it last year and basically it's
not very hard for an OEM to put into its
firmware something that detects when a
particular app is running and if it is
running it can tweak the software to say
hey let's boost the CPU artificially
during these few next few minutes boost
the GPU now it's not the normal
circumstances the phone would use
because if it ran at that speed the
whole time the batteries go flat very
quickly the phone would heat up very
quickly but for a 90 second test for a 2
minute test it's not going to make any
long-term difference and so the way
they're cheating is basically when they
see antutu running or deep veins running
or some of the
the popular one basically the firm
waited hey let's push up the cpos push
up the GPU and then it gives out
different results than it would be if it
was using its normal settings its normal
performance settings for when it's
running a normal program now of course
there is this question I'll throw it out
there I'll say something I think it is
cheating but let me judge this is it
really cheating because actually it's
not like the firmware has somehow
managed to send messages or inject data
into the benchmark and actually get it
to display a full score that was the
score that that device produced that is
what that device is capable of doing but
it's not what the device does day to day
because to reach that capability
it actually has to run the CPU much
harder than it would under normal
circumstances but let me look at it this
way what about the speed things the car
subject it will do nought to 60 in you
know 4 seconds or something like that
well how many people actually pull away
from traffic lights to go north to 60 in
four seconds well you don't just too
dangerous it's not what you do but
actually people will say wow look that's
the speed of that car so you know
they're artificial statistics that exist
in all industries and maybe the
benchmark are an artificial statistic in
this way in fact they are artificial in
the sense they don't really reflect the
workload that a normal app that does I
mean even pain quite complicated games
doesn't actually reflect what you get in
the benchmarks I've done some other
articles about this that shows you how
the benchmarks use the CPU and GPU the
graph sort of produce and they're very
different to what you get under normal
games so Bayes marks are artificial to
start with so if the smartphone is using
the mark officially well is that really
being issue I get it it I say it is
cheating because people expect the phone
to run that fast in normal usage and it
doesn't but it's an interesting question
I'd like to hear your thoughts in the
comments below and so there you have it
the OEMs are cheating on the benchmarks
now this is not the only industry that
is happening it's happening in the car
engine we've seen that with emissions
within that with fuel consumption
figures this is going to happen in all
industries because it's so cutthroat
that people the OEMs are trying so hard
to get
head so their product sells the most
doesn't means right but I'm just saying
it's a reality
my name's Gary Sims from Android source
and I hope you enjoyed this video if you
did please do give it a thumbs up don't
forget to subscribe to Android
authorities YouTube channel P tell me in
the comments below what you think about
Oh am cheating on benchmarks download
the end or throws the app and don't
forget to go over to Angela source calm
because we are your source for all
things Android
We are a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for us to earn fees by linking to Amazon.com and affiliated sites.