Gadgetory

so I think we should bring out our guest he's a brilliant guy the principal researcher at Microsoft he's making some frankly bizarre stuff and I'm looking forward to this conversation please welcome Andy Wilson from Microsoft all right thanks no pressure no pressure you have to one-up battleship oh you do that I'll try actually kind of low pressure ah so Andy these guys don't know you maybe they know you a little bit we did a video uh months ago like a year ago or something yeah then I thought was really came back yes I did yeah I came to visit your home away from home your lab yeah which you actually do a lot of fun you do live there right yeah pretty much tell me what you do where how did you how did you start doing this and what do you do at Microsoft and give me your give me a little your backstory so I'm a researcher at Microsoft Microsoft has a big research group of about 850 PhD level folks doing research on all sorts of things I'm mostly interested in natural user interaction and different ways that we can exploit sensing technologies and display technologies in in completely novel ways and we make the way that we relate with computers so a lot of the things you saw there are prototypes where we believe very firmly and building things and playing with them and to sort of finding our way through all these new things just by experimenting and trying as many things as we can and your work is I mean from what I saw and some of the stuff we're going to show today your work is really about I mean we're very used now to these kind of we have a screen or we have something in front of us we have some in our hand and we interact with that we're kind of like going into that interface your stuff is like taking that interface out into the world I mean a lot of what you do is is it's like projecting on to the world basically in making the world the interface where does that where does that instinct come from I mean what is that what is the end game for you with that so that's right so so I like to call it beyond the surface Sam so I worked on the first version of surface and the one that used a video camera is the large 30 inch tabletop and so for me it's always been about what can we do with sensing technologies and interesting ways and for me the work that we've done with depth cameras for example you know lifts all of the interaction we're getting off of touchscreens out into the real world and that's where we really live we don't live in the the flat plane of a display we live in the real world let's bring the sensing and display into our real world I mean you headed up the surface project that when it was a large table so I did the very first prototype and a couple of us a couple guys and I did the first efforts around surface and building the first product it was a lot of fun was it painful to watch microsoft introduced a tiny tablet called the surface to that don't know it was very flattered that they decided to reuse our name i thought that was a great choice but now you can't call the surface to surface that's right there that's that there's that ring that does that hurts doesn't it a little bit you can share not is no judgment here in this room okay so you also incorporate a lot of connect sort of like technology and into what you're doing is connect the first I mean that's you know there aren't many consumer products that are actually utilizing the physical around you not yet to be is that the first step do you see that it's just like that you know Oh point version 2.1 of this absolutely is it's the opening opening move and a thinking a great you know Renaissance and natural user interfaces you know people have been doing this research for a long time I was a graduate student at MIT Media Lab in the 90s and what we were doing a lot of this hand tracking kinds of things and interacting with with things on screen that now you can go out and buy it's really nice to be able to see that move from the lab gosh just taken a little while but now it's like it's now that its commodity at least in for certain kinds of applications let's think about other applications where that where this would actually be useful can is this the kind of thing that you would find in a in a conference room at a workplace or get to be excited when I hear that with would we find this in a conference room probably yeah I'm working is there anything more exciting than something in the conference room I mean I know that's up here that's like you're in the papers I know you're in bright orange on the conference room tech but like can we go to read what about for the home what about for the kitchen friend yeah kitchen is a nice one because you know you don't want to get your hands all dirty on the on the touch pad right so maybe you maybe you put that on the on your table top in the in your in your your kitchen counter and you actually lay out the recipe and the things that you need and we have a we've had a few demos over the years on Microsoft where you it would be really fun for kids you set out the recipe and the ingredients for the recipe and kids get that and then yeah sure can I pitch an idea to you go ahead of the kitchen let's hear it so I'm imagining something like you know these tentacles hanging down over your over your kitchen work surface and then you use some kind of motion tracking the tentacles are all full of different spices oh and use the motion tracking you put a chicken or something underneath and then you just kind of minority report the spice is all over it excellent do they get something you could work on or absolutely we actually have complete make that happen I just can do for me we actually have some video stuff that you that you couldn't bring with you yeah this thing is called a Bhima Tron can you explain this is the be mature on that's the Bema Tron and so with that the Bema tryna is a pan tilt motion platform kind of like what you see at rock concerts with lights except we replace the light with a projector and a depth camera and so what we're doing is projecting the car and the car responds to the real world just as as if it were a real model you know radio-controlled car and so by using the the pan tilt movement we can actually get all the way through the room we can project a graphic into this person's hand and move that that graphic from one hand to the other here's a shot another shot of the of a ramp she has a terrible driver yeah absolutely so so the idea really is to bring the the interaction out into your entire world so with one camera one projector in this this motion platform we can we can you know track these people there's we can project graphics on to this person and reacted appropriately she can actually pull graphics from the wall and send them from one place to the next and like a spice yeah sure for instance you do that you can track that the get the Korean yeah exactly paprika boom tentacle down onto the chicken why not uh and then you have another one but you have another one that we have video of which is called the hollow flector can you tell me a little bit about this this is like this freak this is insane so this is not done on post right this is actually video shot of the of a mirror and I'm standing in front of the mirror and then I'm interacting with this this graphics and this little bit like the peppers ghost illusion you might have seen it and you know a Disney theme park it's a little different but we can superimpose graphics on to it and so it's really augmented reality but in a very large form factor and we're incorporating some some concepts from the phone here's you know maybe I'm doing a little holographic phone calls is actually connect record at you that's you talking that's me holding yourself this is my little Maj to the princess land you know a little that's a little weird yeah uh so-so but you brought something today I did in the end and we should take a look at it let's let's check this out what is it called what did you bring this is the wearable multi-touch projector okay I'm gonna come up with a better name for that is a wearable multi-touch projector at least check this let's check this out yeah help this is handsome this is nice now now now uh what did you say this is called again we call this the wearable multi-touch projection we got to come up but once she's caught the predator the I mean it's kind of I don't know yeah a little bit looks a little bit it looks a little bit violin see this is kind of like a connect here it is basically like the kinect uses the same technology it's a short-throw version of connect right so it actually works in this sort of arm reach volume just a very in front of me and you think in the future everybody'll have like a lenovo laptop hanging off of there hanging off of their side is that is this the future i feel like we build prototypes right and i'm just starting the projector there's also a laser projectors it appears at a pico projector projector okay great and so this is our this is actually our second prototype the first one we built actually just connected to it so we have a desktop that was hanging off every way so it brings like a full name eat our meals and you have like a little assistant to carry next to your a second one of course if we were to consider product izing this i mean we would not have this this is but this is a really nice setup because we can actually you know right like really sloppy code and get that running on here and yeah and do all the experiments we wanted so you met your code is sloppy abscess fine nothing wrong with that re so show us what this thing ok what this thing does so when tell me why it does right so this is the idea is to bring some of the interaction into the world so we can actually co-opt surfaces in the world and I'm gonna lean over here and start this and it'll take a moment to start hopefully and just now what version of windows is this this is one of seven actually uh-huh so I'm sorry I've been upgraded yet huh nyan nyan interesting ok so we're using the depth camera to roaring that completely yeah exactly all right that's it so here I am so I'm gonna touch this the wall here you notice that I get a bit of a menu okay it's a little bit like our voices the new windows phone 8 I dressing about so and you notice that as I move back and forth hoops they're just disappeared that's all right we can start that again and as I move back and forth that actually stays the same size we're actually using the depth camera to rectify or make it appear as if the graphics are the same size are actually the real goal would be to stabilize it so it feel feels like there's actually painted on the wall so what I'm going to do is show you a couple things there's a phone icon we can actually touch this and there's this ubiquitous slide so Oh ubiquitous that's it that is a patent case right there Apple are you watching because you can have to you're going down you're done I even bother I would just I just find to take whatever money is in your bank account right now and said yeah right so we have a dial pad here so we can do the kind of expected thing you notice there's this sort of red red ball here and that's showing me where it thinks my my finger is an actual phone number well I gosh I hope not keep going yes okay let's call someone and I'm just dialing a random oh he's gonna call the number because this actually capable of dialing it is not that way so that's that's just something we've been playing around with but it gives you an idea of the sensing technology that we have so we can do a few other things oh by the way it's multi-touch right so in we know what if my finger got in here what do it happens this guy you might try a min if you come and you know yeah that's it is working yeah I thought you might erase this yeah yeah absolutely this is horrible what you've drawn it yeah I mean see if I can do me see if I can do something yeah so it works a little bit better if you come in from the side or a flat oh there we go so and that has to do with the way the algorithm that we've run let's give draw face but that's obliterated right ah you notice that I'm actually so I'm stopping each of the little demos when you by putting my hand over the cameras kind of a funny thing to do but it's a way for us to reset the system I'm gonna show you a couple more things okay great so we have yours here's your map and again as I move back and forth Apple might want to also take out of it and so it's multi touch as well and so I can scroll around and there we go so that's the map and not nothing surprising there and then let me show you guys see the Hoover Dam on that as well as he was intact so there's a couple of other things that we can do one is of the notion of being able to invoke an application just by drawing the space in the world that you want to see that see it and that's not be the space that you want yeah it's what you're saying yeah so can you tell me what that so tell me what the practical what you envision a practical application for this is what is it what are the typical yeah exactly something like that so Ashley its currency converter that's great yeah so that's it give me a price oh I'm walking a night in the future when these are just commonplace yeah we're give me a scenario where this is becoming you know I need this so a good one would be you know where we just met and we're you know maybe we met in the hallway and we're you know I want to show you something over here and i want to show you it in the large I don't want to like you know huddle around the tiny little phone right and actually throw this up on the wall and lamb ajan that's bigger and brighter and all these things projector technology is going to get better of course then we can interact with this as if it were a real you know large multi touch display right right so it to the guys like a guy just met this nerd with this thing on it now it now you wants to show me something exactly that's right but of course everybody will have that well imagine it's a tiny thing maybe it's actually fits in the end you know in a button on your shirt you the entire the entire thing and a button on your shirt yeah you think we can get there I think I think so in a matter of time yes and all of these things that I'm showing you here are really small when you sort of take them in the women I mean the pico projector is just a there's not even a lands on a pico projector right oh we can you show the audience turn around here I just want you guys to see you're seeing the back but do you see this whole setup here I mean the idea that this could get down to the size of a button seems frankly like a pipe dream are you on any dope right now but look at the size of the first cell phones right and look at what with her that's right now that's layer originally like this tall they'd have wheels on them I mean if you turn a thing Oh thing with a wire that came out you took a modern cell phone showed it to somebody from you know 30 years ago they'd be like what is this alien technology you're showing me right it does look it does look alien well this certainly does Yeah right let me let me show you one more thing okay and so we set this and then I just want to give you a little bit of an idea of the processing that we're actually doing so this is just a display that shows the the fingertip tracking and you notice there's it's multi-touch and then as I reach in it goes from hover state to a touch state and its we doing two at once and then finally on the back out it one more time and real quick I want to show you the nature of the processing that we're doing and this is this is basically just showing you the depth stream that we're getting from the Kinect camera Wow right I can I get in here you can get in there right and so you get an idea of some of the processing that we're doing do you want to yeah I don't know I don't know what this Liz I'm exactly but that's all right shows you in this out yeah that's right now it's gone yeah who's gonna get too far away it's too I was too funky for it which is a problem a lot of people feeling too funky and then and the sheets and the other thing that we've been studying is the idea that we can actually project onto anything not not just a whiteboard or a wall or something like that and and then we can we can actually track this you know this notepad for me I won't actually use that is why you tripping me out and so we can actually do slide to a lot of ubiquitous so i doing law you band then but and then we can even do it on your back so it was a turnaround on my back yeah yeah so freaking kinda like where this is headed potentially joining to take my pants off oh you know I get this to work okay right so yeah there we go so alright coops a little bit lower i have a little bit of back pain tonight yeah right there it's perfect there was a key press really hard on oh you'll slide them off yeah right is it so literally we can use any surface available in his slide to unlock supposed to make you sexually aroused is that saturday alright that's very cool get stuck very cool just turn around here let me just uh Andy Wilson everyone Andy Wilson and his and uh and we never we never did come up with it we never did come up with a name for this huh and I call it the sex machine that cool