Gadgetory

behind me right now is the biggest supercomputer in the country it will be serving researchers across Canada budding the human genome in bioinformatics particle physics materials research even Humanities Research it's called theater it costs the federal government through the Canadian Foundation for innovation over 16 million dollars and we get to be the first to unbox this savage jerky is created without the use of nitrates or preservatives use offer code LTT to save 10% at the link in the video description so see here is a big data Musti it takes up a quarter of the 5000 square foot data center it occupies meaning actually that there's room for it to grow but right now it has 27,000 Intel Xeon processing cores 190 terabyte of RAM 64 petabytes of storage 584 GPUs and a whole power draw of 560 thousand what though with that said its efficiency is a shocking 1.07 on the PUA scale where one would be perfect and a typical data center would be one and a half to two we get into how they did that a little bit later though so our tour starts right here behind me are what they call the high availability rack so everything back there has dual power supply for redundancy with a battery backup for that and a diesel generator backing up that everything back here is mission-critical things like networking login servers and management servers are all here and this is also where you'll find the bulk of Peters storage let's get in for a closer look at Peters connection to the outside world this networking appliance from Huawei has a street price of around a million dollars Wow and right here this is where it gets really bananas these guys are cedar is a dual 100 gigabit connection through Vancouver and then as if that wasn't enough these orange ones here are dual 40 gigabit connection through nearby furry just in case somebody puts a backhoe through one of these other fiber lines and they would have otherwise lost their internet connectivity I think that's their backup backup but Ethernet it's not really the way you want to connect high-performance computing those this this right here is the true networking heart of theater these are 48 port on the path switches and they're configured in what's called an island topology so the island is in almost all cases 32 compute nodes each of those compute nodes is connected to 32 ports on one of these switches in its rack then the remaining 16 ports come back to here that means that every island gets a dedicated line to each of the core switches giving you failover and massive bandwidth each one of these fiber links right here is capable of 100 gigabit per second so even though between islands we are let's say bottleneck by our 16 connections not only half the total theoretical speed within an island we're still talking over a hundred gigabyte per second so stops really an issue ok now let's move on to SFU and compute canada's versions of petabyte projects boiler alert there's a better in every conceivable way so in the five cabinets behind me we bought Cedars 50 petabytes IBM tape library system they have a 40 gigabit link to the rest of the supercomputer and each of the five thousand ten terabytes magnetic gates inside can be grabbed out of storage moved with like a robotic arm into a reader and the can be accessed when needed and this is done automatically cool right okay yeah but due to the slowness of that swapping process this is still what we would consider to be cold or archival storage next up here is general purpose storage land where any data that's being used for any current research project would be housed so here they're using off-the-shelf 5u rack each of which contains if we can crack one open here a total of two kind of trays here and 84 8 terabytes of public here enterprise capacity SAS drive from Seagate but there's actually more to this system than meets the eye every 4 of these storage node requires 2 nodes of what they're calling object storage servers these act as a high-speed cache with their SAS 10,000 rpm drives as well as kind of like a traffic cop for everything behind it so every single read or write to the hard drive actually goes through these nodes so right now general Ford land is 10 petabytes but in the mirror mid future it will be expanding when P and K now that DIY approach to storage is great for scaling up at a little cost but when it comes to performance they went for this data direct network storage appliance because it has got the real good now in the rack next to this brain will find a mere four petabytes of actual storage due to its higher cost but thanks to its proprietary hardware custom software and solid-state burst buffers this thing can handle up to 40 gigabytes per second of the same throughput making it perfect for data intensive applications that rely on humongous data set now let's get into compute there are about half a dozen different types of compute nodes all connected to the same high speed on these half network backbone that are optimized for different types of research we'll begin with the base compute no very whopping 576 of these each of these is a computer so there's actually four in a single to you shell each of which contains two xeon e5 26 83 16 core processors 128 gigs of ram and about a terabyte of raid 0 SSB storage for scratch so each rack here contains two islands so that's a total of 64 compute node giving us a whopping 2048 compute units per rep so these nodes are the basic workhorse of Peter handling everything from monte carlo simulations for material science to simulating dynamic processes in nature with a high degree of randomness like no fall or rainfall they would also be used in any highly parallelized workload because if you need you know 10,000 CPU cores for one job there aren't enough cores and any other class of servers to handle that kind of load moving right on up we've got the big memory nodes there are 48 of these and half of them are just like the basic node except with 512 gigs of ram while the other half of them these copies have one and a half terabyte of system memory these ones make up twice as much rack space though each of these one use is a single dual pocket system because you know what there just wasn't enough gocbar room for all 24 64 gig memory modules that are required for that much ram personal problems yes these guys are really special these are the aptly named three terabyte notes there are only a handful of them but these are quadrant socket machine beyond 48 o9b 4 4 of them way to pick those are only 8 core processors these don't even have more processing cores then those little tiny ones that pick up half of you what's the deal here well it turns out that some file informatics workload like genome sequencing don't actually scale very well with more processors they just read massive amounts of memory to hold the datasets that they need to work on so while the team here probably isn't super stoked on using up for use just so they can stuff more memory into the system until Intel octane reaches a higher level of maturity this is the only choice they have now finally we're getting to my favorite notes the most expensive notes these are the GPU note and while they're actually quite similar to the base note with respect to their CPU and RAM configurations what got the researchers in the fields of molecular dynamic AI and machine learning all amped up about these are the quad nvidia tesla p 100 graphics cards that they have crammed into each one i mean seriously with 1500 watts of power being consumed by each one of these is it an innocent engineering marvel that they've crammed enough power and cooling to make this whole thing work so actually now that you think about it how exactly did they do that so the key Knight among you might have already caught a couple of hints earlier in this video but the secret lies in the rear doors on the server rack look how thick this is yes my friends this entire door is a gigantic meat exchanger so their servers don't actually have water blocks that would be more expensive what they're doing is they've just got the front of the rack all sealed up so there's no back rack pressure and they've got normal air-cooled servers that pass the air from the front where they just grow in room temperature air in here and it comes out hot like 30-plus degrees and push it through the heat exchanger where it is actually cool to my skin that how efficient these are and that cooling system is massively expendable - you can actually see above me I am standing where we bought a blooming green cooling pipes connected to a whole bunch of quick-release fittings ready to add more racks right here but to see what they actually do with the heat we're actually going to have to go upstairs where we'll find the final and perhaps the coolest stuff in our tours here this is a mechanical room where the pumps and these freakin pipes take all the water from downstairs and dump it into three cooling towers outside the building out right now the weather is favorable to cooling the ambient temperature is quite low so it's just operating as gigantic radiators but get this when the conditions become less favorable in the summer they kick things into high gear with an automated system that sprays water onto the fins of the radiators in the cooling towers and if you watched our long cooling video which you can check out right here you'll be familiar with this concept already but this is called evaporative cooling and by these means even in ambient temperatures up to 30 degrees Celsius they can achieve the 17 degree coolant levels that they need to without employing the massive chiller unit that they have over on the other side of the room Squarespace is the way to build a website whether it's for your small business or for your you know local freaking book club it doesn't matter if you want a web presence affordably and quickly Squarespace gets it done for you it starts at just 12 bucks a month and you get a free domain if you by Squarespace for the year you just pick one of their templates and boom you upload some pictures you fill in some text it's all cloud-based and your website will simple as that look great on any device every website comes with a free online store and their cover pages feature allows you to set up a beautiful one-page online presence in just minutes so start a trial with no credit card required and start building your website today then when you decide to sign up for Squarespace don't forget head over to Squarespace comm /l tt and use offer code LPT to get 10% off your first purchase so a massive thank you to SFU and compute canada for allowing us to run amok in their data center thanks to you guys for watching if you dislike this video you know what to do but if you liked it hit that like button get subscribed maybe consider checking out where to buy the stuff we featured as a link in the video description also down there you'll find a link to our merch store which has blue shirts like this one in our community forum which you should totally join