Gadgetory

no man's sky is the least stable game we've worked on in years normally when games are somehow unstable it's at least predictable or consistent in the behavior of that instability with no man's sky we're seeing frame drops that are spurious and unpredictable a fact that results in such severe stuttering that by extension the game is not only unplayable but also impossible to accurately benchmark and we've generated a few sets of benchmark data on the Titan XP the Pascal card thank you to Sam one of our viewers for that and also tested out a 980ti just to make sure it wasn't an architectural thing or a driver thing with the XP and then we tested again on an RX 480 to give a more mainstream and also a look at AMD performance so these three cards give us a pretty wide berth of coverage with no man's sky in terms of performance driver sets architectures three different architectures three different driver sets and we still had instability across the board this isn't an instance where we can just test anyway and produce the charts and see what it looks like because the framerate is so varied that you would actually end up with cards that really make no sense hierarchically it's definitely within reason that a Titan X Pascal could be outpaced by something like a nine eighty or ninety eighty Ti or whatever or that on our X 480 could be outpaced by an Rx four seven yeah why would that happen the reason is because of the way the frame drops work and you can watch our what is a 1% low video for that but because the way frame drops work they drag down the average so hard and they're unpredictable so it's not like this happens necessarily every single test run that the result is cards even between average two test passes will just really almost randomly and chaotically fall on the chart so there's no there's no valid reason the benchmark cards against one another if they're not actually going to be representative of performance and any measurable and repeatable fashion so before getting two examples of why this game is not reliable to test here's a quick note it looks like Hello Games just released an experimental branch of no man's sky that's supposed to resolve a few issues shader caching will with the game's intro at least I'm not sure if that will help with the gameplay itself but certainly the intro and among the stuttering issues we also had some issues launching the game on different hardware configurations tested on three different machines just to make sure and saw a varied behavior across the board all windows 10 by the way we've seen spurious mouse movements the FPS cap doesn't work and generally imposes a cap that is not equal to the game settings so 30fps doesn't equal 30 FPS in the game and you can't play the game with the monitor configured to a non-native resolution I know this is a really rare use case but that's a result of the render setting with rendering words within the engine so a lot of small issues and some pretty large issues here a few of these will be resolved by that experimental branch put out by hello games but we're not testing that yet because it doesn't fix the core issue which is severe stuttering in actual gameplay we'll wait for the that patch to be ratified until we test for that so that's the basics the first chart shows the Titan XP running one test execution and that's what the game configured to its lowest settings that's what you're looking at now here's a chart for the first test pass we did multiple this one is at 1080p it measured over a 30 second period and let's just stress that for a moment this was only 30 seconds of play and that's what is normally acceptable for testing but if you look at multiple passes these spikes are seen in almost all of them which means that you should expect to encounter one of these game interrupting stutters at least once every 30 seconds if not more in this instance we've actually got the Titan XP stalling for a full 100 milliseconds before the next frame is output and it gets worse from here by the way if you hang on a minute when we discussed frame times we normally talk about deviation from the mean latency as being the most jarring ie if you move 16 milliseconds from one latency to the next then that could be pretty noticeable in gameplay and produce a terror but in this instance it's blowing out the graphs limitations so to speak the result is an output of 138 FPS but 836 fps 0.1% low and a minimum effectively of zero for that 100 millisecond period which really is a pretty long amount of time and this next one shows the second test pass where we go from an average of six-month second frame time to 20 in the worst case scenario and that's certainly observable but not nearly as bad as the last time so a deviation is still exceeding that 8 to 16 millisecond range at times and that's where you will see some noticeable swings and performance our numbers are at least somewhat more consistent here though let's look at what happens if we configure the settings to what actually makes sense for a tighten XP just a reminder those were low settings here's a ultra with 1080p first we'll look at the rest later notice again that we've got a 100 millisecond spike drag in our 0.1% lows down to an impressively atrocious 31 fps at 4k we're looking at an output that spikes to 1,000 milliseconds that's one full second and that means that during gameplay while walking our fixed and looping and repeated benchmark course we had a stall where the game effectively froze for a fall second this dragged our 0.1% lows down to 18 fps though the minimum value was again 0 FPS for that one second period out of the full 30 second benchmark let's look at the 980ti a now before moving to the arts 480 just to make sure this isn't a driver or architecture-specific issue we tested the 980 I reference card with low and ultra settings looking at low settings we're seeing spikes hit 40 milliseconds at times in the 30 second test period resulting in a 2x deviation between the 0.1% lows and the average the second set of data perfectly illustrates why we've decided against benchmarking this game right now showing a dip in frame performance from the previous 122 fps down to 93 fps and that's in the second test pass of the same settings same graphics configuration and same part of our test course that's a huge swing 30 fps and these two test passes were conducted within one minute of each other in the same area we see 93 FPS average and 3 FPS 0.1% lows are produced by the outburst of a 4200 millisecond frame latency and again that's exactly why we use those little metrics though here they're really you need something worse than that you need actually a minimum for once because it's not an outlier and that's because a 93 FPS average looks pretty damn good aside from the fact that the game was unplayable from four seconds out of 30 so you four seconds out of the thirty second pass you can't move that's just it's effectively lag or stutter or I think some some people call it hitching whatever you want to call it it's not playable to further diversify architectures we looked at an msi our X 480 gaming X which is an Adam board Partner card and that's in addition to the two reference and video cards with low settings low at 1080p we hit one snag of 1500 milliseconds or 1.5 seconds and another of 1,100 milliseconds 1.1 second that was in the same test pass by the way with also a 380 millisecond snag and that's nearly three seconds of frozen game time in a 30 second period dispersed across about a 15-second range of testing so within 15 seconds you freeze three times for periods that are longer than in some cases one full second so that's about 10% of the entire 30-second test pass that you can't actually move in the next pass we saw better performance but not much snags we're at 120 milliseconds at least wasn't a thousand and several were clustered at 60 milliseconds resulting in a 0.1% low of 19 FPS things weren't much better at Ultra either with a 3 FPS 0.1% low in this test pass resultant of an 1,100 millisecond frame time followed in sequence by a 500 millisecond frame time then a 300 milliseconds frame time or latency between frames so this game even if you're able to sort of play it just by fighting your way through these things drop the settings enough that you're only experienced only only experiencing his 1 second hands every minute or so and I know some folks have reported they can play the game obviously this is hugely dependent on hardware things like that but we did validate on three systems 2 X 99 identical 100% identical platforms except for RAM RAM is different and that one z97 platform with a 4790k so we've got 250 930 K is a 4790k all three of those platforms validated the same performance more or less and we have three different video card architectures with the Titan X Pascal certainly at the very top end even that thing is having trouble with this a 980ti for a last gen architecture look just in case maybe driver maturity maybe the game developers built on last gen architecture certainly within reason and of course the polaris card rx 480 which was an admin board partner card so we really did given how much of a mess this game is right now we really did cover all the bases just to kind of make sure ok is this thin something we can benchmark properly and the answer was no just looking at those 3 cards with the three systems 9 total configurations we kind of backed away from it decided okay let's just let's see if they actually fix this thing first because right now as you saw in one of those test passes we had a Swain an average of something like 30 or 40 fps and sometimes it freezes that 0 fps so that's not really the here's the here's the thinking okay so because of those swings yes we could benchmark the game we could still perform maybe 5 passes even and average the numbers and all of that and say look it's really not performing that well right now but the problem is because the purpose of a GPU benchmark is to compare one video card to the next for its performance against a competitor or whatever an Adam board alternative because that is the purpose it doesn't make sense to do a benchmark right now no man's sky because it's really at that point not looking at video card performance looking at the games performance which is dismal and so to plant cards on a chart I pick them against one another and declare victors at different settings and resolutions and what-have-you is really unfair to the vendors GPUs all that because we're not even looking at isolated GPU performance anymore looking at totally random spurious FPS output again with fluctuations from 6 milliseconds to 4200 and frame latency the deviation is just it's not even enough it's not an outlier no it's not an outlier because it's happening every single test pass and sure sometimes it's a hundred milliseconds but at least four of these examples I gave you just now we're over a thousand so the short of it we are not benchmarking no man's sky today I really wanted to it's not gonna happen because it's not fair to the GPUs and it makes no sense to do so and we just be making stuff up at that point because you're looking at frame rates that don't make sense so hopefully they fix this there is a patch forthcoming the experimental update that's out there right now you can grab on Steam if you own the game does resolve a few issues like the shader caching things like that it does does lessen the workload in areas where the workload does not belong but it doesn't fix everything and it's experimental so I'm not showing the results today we'll look at it once the patch is sort of matured and been ratified into the game code officially and see if things have improved but for now on our hardware we tested it's not playable I would not recommend playing it I know some folks certainly many of you I'm sure have the game operational to some level I know I've seen reports of people saying you have to play it for an hour per session and then it will stabilize that's really not it's not acceptable to me but I'm not telling you don't buy the game but if you haven't bought it yet maybe consider waiting of it so that's all for this video as always pay challeng to post your video tell us how to directly subscribe we have some pretty big stuff coming up this week actually a lot of it and you don't want to miss that thanks for watching I'll see you all next time