Gadgetory

so I've got a problem when it comes to building a storage server there are two main options for your drives you can either go with SSDs which are compact and have amazing performance or you can go with traditional mechanical storage which has the benefit of lower prices and outstanding capacity with the drawback of being an order of magnitude or more slower especially in highly random workloads like if they were loaded up with a bunch of virtual machines or in our case if they're pulling network storage duty for a whole bunch of heavy users who are all trying to edit 8k videos at the same time if only there were some way to merge the benefits of both with the drawbacks of neither well there is it's called the Appel fusion drive and this episode is brought to you by iFixit the marlins screwdriver set from I fix it features five specialty precision screwdrivers check it out at ifixit.com forward slash linus at the link below the concept of tiered storage is nothing new it's been commonplace in the data center for many years and has even made its way down to the consumer level in the form of Intel's Rapid Storage Technology or rst which uses a small fast SSD to accelerate a larger slower mechanical drive and more recently Intel's octane memory technology which does pretty much the exact same thing but with even faster octane rather than NAND SSDs the problem that I'm running into here is that most of the tearing solutions out there have one of two problems they are either inordinately complicated to set up requiring extensive knowledge of not just Linux but even broader storage architecture or they are expensive because you are paying the folks who have the aforementioned knowledge to create a turnkey solution for you there are some exceptions to this however and I would like to explore a couple of them so the popular freebsd based FreeNAS software for my existence uses your systems RAM and its SSDs to cache frequently used data that's stored on your mechanical hard drives but the issue is that ZFS this particular implementation is geared more towards accelerating performance of large databases and doesn't look like it's gonna be great for our workload because the file sizes that we want cached that we're storing on our hard drives are so large like these are 8k red video files and there are practical limits to how much level to Ark that would be the SSD tier I mean there's also practical limits to how much RAM you can put in a given system so then the one we're gonna try today is the built-in tearing mode for storage spaces in Windows Server and what better way to test this out then to set up a tiered storage space on the network so I'm going to use this test server right here and then have our team of editors just switch over to it and see what happens and it's at this point then that we wondered whether or not obtained would make a difference for some of the heavy tasks that we deal with around the office no no it does not now our regular editing server uses 31 1.2 terabyte nvme drives and it was really expensive for us to build and they've been complaining that there's not enough space on it because we've only got about 24 terabytes of usable space after all the lost space to redundancy on it and they were like oh well you could just upgrade it I went yeah ok technically I do have another 17 bays that I could fill up but that's gonna cost me like 7 grand 8 grand even if I buy used drives on eBay so I was looking for another way so our test server here the config is pretty straightforward I've got 4 10 terabyte drives then because I want to speak exactly how much cash you need this is what really got me thinking about it Intel uses just 32 gigs of cache for large multi terabyte hard drives and we've seen real world huge increases in performance so I went okay let's try this so I took two of their 900 series op team PCI Express drives and then I've got a total of 40 terabytes of mechanical storage and I want to see if for a fraction of the price we can get good enough performance that the editors aren't going to run into a bottleneck so you can see I've got my four mechanical drives I've got my two SSDs allocation is all gonna be automatic so I go ahead and I try to create a pool okay I've created a pool now I try to create a virtual disk create storage tiers on this virtual disk it should automatically detect which ones are your fast drives and which ones are your capacity drives so we're gonna use mirroring because that way we're protected from a drive failure going to use fixed provisioning and now we get to specify the size of our faster and our standard tier we're gonna go ahead and select the maximum sizes failed to create virtual disks not supported so the issue is that the GUI version of storage spaces doesn't have some very important options like selecting how many columns you want so when you only have two drives and the default is three columns it spits out this stupid error how did nobody at Microsoft notice this for years anyway doesn't matter I called in a life line LED eater over on the forum is a guru when it comes to this stuff so I'm just gonna pull up his DM over there and follow his instructions to hopefully do this in PowerShell so this will be fun you guys are gonna be learning right along with me I haven't done this before the video so I can still use the GUI to create my storage pool apparently go ahead and do that then I need to fire up PowerShell and change the default number of columns okay well I don't know what I just did but I'm going to change it to mirror number of columns default one I think I think we did that let's find him so I think the command I just entered changed the mirror layout default to one column it did not okay so I'm using PowerShell ISE now and this seems to be working better mirror number of columns to fall one come on so much right now I just want to change a stupid English setting definitely called mirror it's definitely called storage pool one ok create these commands using their own thing we'll figure this out at some point the good news is there's a back-up plan swear I had this working at one point all right here we go so right now we've only got a 10 gig link I can put a faster card in here if it ends up being a networking bottleneck but for now what we need to do is grab a bunch of just like video projects that are actually current and copy them to our tiered storage drive here the idea is that I want to bring over more than would comfortably fit in the cache and then I want the editors during our test to open up first a whole set of completely different projects from each other then close those open up a new set go back open the other ones and evaluate the performance for themselves so we're back and we settled into a steady transfer speed of anywhere from 200 to 300 plus megabytes a second and all of our projects are over on here so I guess all that's left for me to do then is go interrupt the editing team so I just need to know if you guys all hit this and try to edit 8k footage at the same time will you notice slowdowns because I know some people are sensitive to slowdowns so here's the idea this is the network share so everyone kind of pick a project and open it and start doing video editor things whatever those are work the way you would normally work because full is going to be affected by like your system as well so the idea is that once it arrives it gets cached in the fast tier so I just wanted to get a feel for what this is like ed are you opening project Oh coding but you're encoding it Oh exporting probably isn't a very stressful test ok what are you doing Taryn can you try doing some more editorials like pretend to edit this moving stuff around I keep pretending pretend to do your job yes can you go back and play that sequence again and see if it's any better how fast would you expect it to be like it's just normal or okay just hold on a second I haven't I haven't given up on saving money yet so the thing I want to find out now is if you guys keep doing stuff will it get any better so I'm starting to come around to the realization that maybe tiered storage solutions aren't grab-and-go because in every environment you'd want your tears to behave in a completely different manner like if there was some programmability in storage spaces where I could say okay take all the newest files that come in and assume they are important until such time as they do not get accessed for a very long period or until such time as something new comes in and bumps it out then it would be less likely for those guys to open up a project and have it chug until storage spaces goes oh this is important and can promote it from the slow mechanical tear so it's possible that if I had a much much larger fast nvme tear we wouldn't run into this issue because this is really a very heavy load for for hard drives and two SSDs but the problem is that that would require me to basically build that entire server without even knowing that it would work so you are dropping I'm dropping neighbors in my friends can you rate your editing experience out of ten for me here seven out of ten sorry you're you're saying this is better than one all right everyone just go back to wanna con map the drive throw it all in the garbage so that's pretty much it I am coming to the the the acceptance stage in the in the stages of grief here where I'm realizing that if we want more capacity of our high performance network storage I'm just gonna have to pull out the wallet and make it happen thankfully to take some of the sting out of it is our sponsor for this video Squarespace with Squarespace you can make your own beautiful website without the hassle there all-in-one platform makes it easy to get up and running quickly and they've got all kinds of great features from their logo designer to their built in e-commerce to the ability to publish in apple news format so that your blog will be available immediately to millions of potential readers worldwide and they offer all kinds of support so you can build the best website you can build whether it's their seminars or contacting their 24/7 tech support via live chat and email so don't take my word for it go try out Squarespace at the link in the video description then when you decide to sign up for real go to squarespace.com slash LTT and you can get 10% off your first purchase so thanks for watching guys if you dislike this video you can hit that button but if you liked it hit like get subscribed or maybe consider checking out where to buy the stuff we featured at the link in the video description also down there is our merch store which has cool shirts it's winter like this one and our community forum which you should totally join