Gadgetory

hey it's Josh regard from android authority here at Google i/o at the project tango booth and I'm here with one of the engineers of project tango Johnny and I wanted to get some to get some extra information because I know that tango in some ways might be somewhat of a mystery to some of you out there so I wanted to get it from the actual people themselves what project tango is from their word so when we get a bit of an explanation about our project tango is and we have seen gaming examples of it but what else can project tango do sure they at a high level the goal our project tango is too advanced 3d sensing and tracking on mobile devices so we've been working with the hardware ecosystem and the software our partners to advance both the sensors and the algorithm is necessary to make these devices track their position and capture models of their space so one of the things that we've done it a lot recently is what you can do it just the tracking information where we take that camera and control either a unity basic 3d in my environment or an unreal engine based 3d environment we and we add the data from the depth sensor this is where it starts to get into a new code and new algorithms that aren't quite ready to put in the SDK yet but also why we haven't had so many demos but we have a few demos here that essentially as you walk through a space you're actually able to capture the geometry of the floor and the walls and the furniture and outside of gaming well inside of gaming that allows you to have characters that sort of know how to navigate through your hallways or potentially play hide-and-seek in your in your house and because it knows where your closet is but industrial applications and professional applications include real estate where you're trying to estimate the square footage of your home we have apps from Trimble which essentially start to explore well how much better could you make room scanning because rim scanning has ends up being important for shopping for furniture or being able to figure out how much corporate do you need or how many tiles that you want to put on the and then there's an industrial inspection so the trumbull also has another very early prototype of an experience where we've taken a 3d model of the most funny center and once we recognize the position of the device in the Moscone Center you can essentially get x-ray vision of where the elevators where's the service stations for this particular amount panel and without having to walk around the building sort of looking at the signs on the walls you actually can hold up the tablet and see sort of through the walls with sorry with a one one question I just came to mine for for various applications I say consumers would be using would the algorithms or the 3d schematic switch let's say would they be made available for the various applications or would the consumers be able to create those schematics themselves so the amount of data that the device can generate from scratch is relatively modest it basically is just the position of the device as you were holding it and then rough geometry information sort of i can pick out sort of rough walls and floors and some furniture it since it has a 3d sensor to capture that geometry it does not automatically know about say where does the hot water pipe run through the wall that requires having the CAD model in advance or some other source so and chances are people don't won't have that available in most buildings but for professional and industrial commercial structures that data may be available by the Builder so that it's a 7-inch tablet that has a Tegra k1 inside and our custom designed sensor so we have a fisheye camera that's designed for motion tracking which is in the middle and if you think about human vision we have a tremendously wide peripheral vision that expands up and gives us enormous amount of context about the way the environment is moving and so what this does is gives us a wide field of view of the environment and then we have a set of sensors on here that are depth sensors these are sensors that basically give us information about the floor and the walls to tell us what the geometry is in this particular tablet we have structured light depth sensor from a company called mantis but we also have a prototype hear of a depth sensor from a company PMD tensions that has a different principle of operation which is time of flight and the time of flight has slightly different properties and that it doesn't require a large separation of sensors and potentially is more robust to sunlight but as we evolve the hardware will continue to sort of push the push the envelope in terms of the performance of those deficits is true so let me give you a quick tour of the software that actually runs on a project tango device this is our Diagnostics tool which just gives us a view of the sensor data coming in so on the left side of the screen you can see this fisheye lens in fact I can still see that my fingers just around the tablet and so this is a super wide camera that that allows us to understand the motion of the device the left points are being tracked yes so those green points are essentially the image processing doing optical flow so this allows us to understand while everything moved in the world in a way that makes sense to the way that device was help we also have the gyro and accelerometer data right below the image that is essentially time synchronized with the with the camera so this allows all the sensor data to come into the same time Orwell timestamp so we can make an estimate of where the device is but if I actually move the tablet left and right or actually make a circle you can actually see that it's actually tracing out a circle on this one over on the screen yeah so I'm going to just walk around here a little bit and you'll see the tracking showing up on I'll point at it over here yeah so Johnny's gonna go off camera and you'll see the tracking right here is actually taking where he is and there it goes you can see it's actually mapping the entire area that he is walking through using all of those sensors and now we have this sort of this kind of janky figure eight right here oh that's great yeah so this is tracing out my path so one of the things we can do is insert that path into a game environment so this is one of the unity applications that we have will be part of the SDK which is how do you actually use this data for end user experience and so what we've done is we've connected that tracking data to the unity camera and this is a very simple game where essentially I just pick up the cube and put it on the switch but to actually finish this game I you can see that I can't quite reach the cube from where I am now so actually have to go walk forward to solve anything is that using the data that you just tracked that's right all right so so that this ability to track the physical motion of the device is now being piped into unity so if I pick up this green switch buck sorry I can see that the green switch is actually over there on the booth you hit them so I'll come back alright and there you go so this so this basically introduces motion into mobile gaming so if I move forward in the real world I actually move forward in the virtual space as well so this allows game developers application developers to start thinking about well how would I actually use my physical room either my my living room or my office space and actually use this inside the game which is if you can imagine now that I understand where I am in my house or my environment and I actually have a little bit of geometry data I can sort of transform my environment into a fantasy world so again i'm just using the tablet as a camera controller i can look at the trees and the stones but i also see on the ground there's a small wizard so if we crouched if I crouch down I can get down to the Wizards level and interact with him so essentially if I have the ability to place these assets i can start transforming parts of my living room into a fantasy environment so I can't imagine you know my living room is fantasy world versus the my bedroom is another sort of safe haven or you know in Iceland and you can actually use physical space as part of the game if we had game a game that actually understood the shape of the Moscone Center you can imagine actually having multiple people walk around the space competing with each other for territory you think that would not be very dangerous though like I actually have a personal desire to see a game at some point that causes two people to actively tackle each other but we'll see that actually comes to comes to fruition here's an app from a university partner that we just got recently that actually combines the real-time data of the tracking with the depth sensing that we have I will see you this runs this was an app that we just got a few days ago so what this is doing right now is actually combining both the tracking data and the depth sensor data to allow us to create a 3d capture of the space oh it's like oh that's actually imaging the yeah so this is like well I'm painting the environment with my camera and I can back up when i'll scan you into this place okay that's supposed to be me right there yeah so this is a you know a very simple mesh that we're building in real time but as the software improves and as the hardware improves in terms of accuracy we can imagine that the quality of this data continues to get higher and higher fidelity if we actually stored the data and did more post-processing on it app developers could actually get much much higher quality but this is just what we could process in real time so these are sort of the building blocks you can imagine allowing developers to start building use new user experiences that use both motion as well as physical geometry to create new eps ok do you think that next year maybe for i 0 20 15 or 20 or 16 you'll be able to create that game using the Moscone Center schematic I think it would be a really neat target to aim for oh cool we'll have to see there's a lot of things that have to sort of improve on both the software side and the hardware side to make it happen I'm sure it's gonna get there I mean yeah some great minds working on end with the way the word is already i'm almost sounded this is really wonderful like it's great all right well thank you very much and yeah you know keep it tuned here we have project tango as it as it happens we're going to be covering it as it comes along it's a wonderful wonderful sweet of technology that we have here and Johnny one of the minds behind it thank you so much for for fielding our questions and for giving us a demo of it thank you right take care all right this android authority coming to you live from google i/o 2014