Storage options - RAID?

MusoD · Nov 7, 2017

Hi Folks,

I'm at the stage of needing another external hard drive to store photographs... which got me thinking about back up, and bigger storage solutions.

What are the thoughts on RAID systems, and how easy are these to configure and work with? It's totally new to me.

Any recommendations on what products are good and work well? I'm Mac based, so something that works nicely with Macs would be good.

Thanks for your thoughts!

pwp · Nov 7, 2017

I think what you're referring to is a NAS (Network Attached Storage).
Go straight to the Gold Standard, Synology. Do a search and read reviews.
https://www.synology.com/en-global Go to the Value Series.
I use an earlier version (DS1511) of the DiskStation DS1517, six years old and 100% stable.

-pw

MusoD · Nov 7, 2017

Thanks PW, that's great.

Forgive my ignorance, but do I just plug these into the computer and they essentially behave like a regular external HD?

LDS · Nov 7, 2017

MusoD said:
What are the thoughts on RAID systems, and how easy are these to configure and work with? It's totally new to me.

RAID are a first level of protection to avoid a catastrophic data loss, downtime, and the need to restore from a backup (if you have it). It can also be used to increase the size of a single volume, beyond the size of a single disk, and usually read speed (write speed depends on the configuration - they can be slower in some), because operations can happen in parallel.

They increase availability being able to survive one or two disks failures, depending on the level (but RAID0 or JBOD configurations that just increase the volume size, with no redundancy), at the expense of part of the disk space used to store redundant information. This allows to keep on working, and rebuild the array, although another failure will mean all data will be lost - a good backup procedure is still needed. The large disks of today unluckily increased the chance of another failure under the stress of rebuilding a large array, especially if disks are near the end of their life.

With most systems, and especially consumer NAS, configuring RAID is very simple. Just select the level you want, and the array will be built automatically. Once an array is created, it will appear and will be used like a single disk. You'll just need to monitor is health - and replace failed disks ASAP.

It is important to select the right type of disks. For spinning ones, there are desktop/laptop-oriented ones which are designed for speed, less fore reliability, and there are storage-oriented ones which are designed for reliability first. Some prefer to avoid disks from the same production batch, because it may increase the risk of failures at the same time.

You can use a RAID storage for primary storage, secondary or backups, but RAID itself is not a backup.

I have a primary storage which is a RAID-1 (mirror) array, a secondary one which is akin to RAID-5 (but built on ZFS, which implements a different kind of RAID), mostly because it allows to create larger volumes with a degree of safety, and then backups on other media. This configuration ensures I can keep on working if any disk fails, and a full copy of data anyway exists on other media if I need to rebuild an array.

LDS · Nov 7, 2017

MusoD said:
Forgive my ignorance, but do I just plug these into the computer and they essentially behave like a regular external HD?

A NAS is attached to the network, hence the name (Network Attached Storage) and you see it as a network share (although a volume can be mounted directly via iSCSI if supported, for example, but it is more complicated). There are also DAS systems (Direct Attached Storage) that are attached directly to a PC using Thunderbolt, USB or eSATA and will behave like regular external HD - plus the utility fo configure them.

Zeidora · Nov 7, 2017

I use RAID 1 (or RAID10) like an external hard-drive, so without the network aspect. I had some NAS, but the network controller has failed on some of them, so I am rather weary of them. Unless you need the network aspect, I would go with simpler external HD configuration.
Re which RAID flavor, for additional safety, I would only consider full mirroring solutions, so RAID1 with two discs (both disks have the very same information) or RAID 10 with multiple discs (with 4 disks, 2 pairs are joined to make one larger volume, the two pairs have the very same information). I would stay away from RAID5/6, because the once one volume fails, the strain put on the remaining drives increases during re-storing, and failure probability increases for a second drive. Then the data becomes unrecoverable. See wikipedia entries for details. IMHO, the additional storage space you get with RAID5/6 is not worth the risk, given how cheap HDs are.
I use LaCie RAID drives and am happy with them. Have 2 disc desktop, 2 disk portable, and 4 disk desktop versions.
Buffalo Tech NAS have had the network card failure. Been a while, so maybe they have gotten better.

MusoD · Nov 7, 2017

Thank you all for the comments so far, I appreciate your help.

As you say Zeidora, I'm not too worried about about the network or remote access side of things, really I guess the main things I want from it are increased storage for increasingly large photo libraries and a mirrored backup. So it sounds like some kind of RAID 01 or RAID 10 Direct Attached Storage is a good option for me.

So looking at the Synology range as PWP recommended, are there any thoughts on:

Synology DS216 two bay enclosure
Synology DS418 four bay enclosure

Even with just these two devices, there seems to be quite a few model variations, and that's before I add the drives!

Thanks again.

cayenne · Nov 7, 2017

Go ahead and pony up for the 4 bay enclosure.

I have one and it is GREAT!!

I plan to get either another couple of them, or maybe a larger unit.....

Depending on the RAID set up you use...you can increase your READ or WRITE times which helps if say you need lots of throughput for video editing, etc...

But I would vote with the others, I love my Synology NAS units.

cayenne

MusoD · Nov 7, 2017

cayenne said:
Go ahead and pony up for the 4 bay enclosure.

I have one and it is GREAT!!

Definitely thinking this is the way to go.

So if I get a 4 bay enclosure, could I for example add something like two 4TB drives now, and use that (mirroring one f the drives) and then add the other two later? And would it mater if the later drives were a higher capacity (assuming that I need more storage down the line and I'm guessing that larger drives are gradually getting cheaper), or do all the drives need to be the same? Sorry if that is a daft question.

Mikehit · Nov 7, 2017

MusoD said:
cayenne said:

Go ahead and pony up for the 4 bay enclosure.

I have one and it is GREAT!!

Click to expand...

Definitely thinking this is the way to go.

So if I get a 4 bay enclosure, could I for example add something like two 4TB drives now, and use that (mirroring one f the drives) and then add the other two later? And would it mater if the later drives were a higher capacity (assuming that I need more storage down the line and I'm guessing that larger drives are gradually getting cheaper), or do all the drives need to be the same? Sorry if that is a daft question.

Discs do not have to be the same size.
Also be careful with mirroring - in its simplest form, a copy is created in 'real time' and that means all edits and deletions so if you delete something it is deleted from the 'mirror'. Mirroring in this way is useful if you are concerned about your disc crashing in mid-operation but it is NOT a back-up in he traditional sense because deleted data is lost. IMO discs are so reliable now that I would not even consider a mirror.
But you do need a definite back-up strategy: multi-bay NAS are great for creating back-ups after you have finished for the day and you have the choice of incremental back-up (saving only the changes) and full back-ups (saving the whole drive anew).
Backup strategy is about risk containment and everyone has different ideas about what is sufficient.

privatebydesign · Nov 7, 2017

I recently went through this same process, buy yet more HDD's and daisy chain or consolidate and upgrade to ba more streamlined and efficient storage solution.

After answering the DAS/NAS question to yourself, and that is not straightforwards, then you can start to look at feature sets.

DAS or NAS? You need to answer these questions:
1/ Do you have one workstation that you do all your image editing on? If yes then get DAS.
2/ Do you need remote access to your files outside your home network? If yes then get NAS.

DAS is cheaper then NAS, it is simple to set up and generally they are much faster to access files than NAS systems. They can be configured for speed, storage redundancy and a mix of the two. I ran an expanding DAS system for years and liked it because it could grow very easily as my file sizes grew!

NAS is more flexible than DAS and the configuration options are bewildering. But none of it could be called intuitive even when you use the simplified 'comes in the box' utilities. The truth is if you don't need remote access NAS is not the best way to go, file access is slower, it takes a lot more looking after and configuration, it is considerably more vulnerable to hacking and corruption and I probably didn't do the smartest thing in getting it.

I got the QNAP 453A with 4x 4TB set up in a RAID 5 which gives me redundancy and 12TB of space. It is a great piece of equipment and QNAP offer more bang per buck than Synology. I have mine set up as a media server as well that streams to my network attached TV and sound system and it also streams to mobile devices too so I am getting some more value out of it, Plex and on the fly transcoding are great features if you have a use for them, if you don't save some money and get a nice DAS! I am surprised at how slow image files move to and from the NAS on my main workstation, to the point that I need to do a lot more work on network switches and configurations to get a faster throughput, big composite PS files are pretty slow to open.

Jopa · Nov 8, 2017

Unfortunately all proprietary solutions more or less suck. If something happens to the controller, firmware, data integrity - you're completely on your own. It's usually very hard to recover a proprietary file system. The "factory-built" NAS devices are usually small and lack proper cooling, which is also critical for mechanical hard drives. Ideally - build it yourself. The drives are cheap (large size but slow RPM), Linux is free. You need a good RAID controller with a battery backup - performance and recoverability will depend on it, and a decent size enclosure with a bunch of fans. Any motherboard, CPU and 16GB of RAM will work. For extra reliability you can add a UPS that could shut down your storage machine in case of a power outage. You can buy all parts on NewEgg, tax free.

scottkinfw · Nov 8, 2017

MusoD said:
Hi Folks,

I'm at the stage of needing another external hard drive to store photographs... which got me thinking about back up, and bigger storage solutions.

What are the thoughts on RAID systems, and how easy are these to configure and work with? It's totally new to me.

Any recommendations on what products are good and work well? I'm Mac based, so something that works nicely with Macs would be good.

Thanks for your thoughts!

You may want to have a look at Drobo 5D3.
I have several Drobos and I have been using them for about 4 or 5 years now. I really like them a lot, and I'm a Mac guy.

Here are some highlights:

* easy set up and use.
* 5 Storage bays
* mix and/or match HDD's (best however to have same speed)
* Hot swapable
* Notifies you if a HDD is about to fail
*Multiple configurations- I think it is RAID 8 (check web site)
*Runs off of Thunderbolt/ USB C* I do LR editing using the files that are stored on it.
*Use any HDD you want
*Run low on storage, add another HDD or a bigger drive.
*It is easy, it works.
*Relatively inexpensive vs. other brands
Hope this helps.

sek

yeahright · Nov 8, 2017

I am also currently considering buying a NAS, since my desktop PC's hard drives are near full with photos now. Since we are two people accessing the same files from different computers in our home network, I suppose a NAS (instead of a DAS) is the way to go. At the moment I think it'll be the Synology DS918+ 4-bay NAS with two 8 TB Seagate Ironwolf HDDs configured as RAID-1 in the first two bays. What puzzles me at the moment is what backup strategy and -media will be sensible.

Buy another two 8 TB HDDs in external cases, and regularly backup onto these two drives alternately from the NAS and store both off site (so that there is always one backup off site at all times even during backup onto the other backup drive)?
Are the NAS bay cases mechanically suited for transport so that I can simply put the backup drives in bays 3 or 4 and take them out after backup to store elsewhere (instead of external HDD cases), or are they merely delicate frames that still leave all the HDD contacts exposed and vulnerable so that it's unwise to cram them in a bag for transport?
What backup strategy: incremental or always simply mirror the NAS onto the backup drive? Will 8 TB for the backup drives for incremental backup be sufficient, or should they be larger to accomodate the larger size of incremental backups (since they also store previous file versions)?
Are backups in a proprietary format using proprietary software dangerous anyway and it's better to have all your backups in a standard file system so that they can be read by the operating system without the use of backup software?

Zeidora · Nov 8, 2017

Yeahright: Re multiple computers and access to DAS/NAS, you could give second computer remote access privileges to first computer with all its attached peripherals such as DAS; would require that first computer is running.
Remember, networks (even 1 GB) are still slower than thunderbolt (I think it is 20 GB). There are 10 GB RJ45 copper switches (as opposed to optical), but making everything 10 GB on your network may be quite costly. So first computer would get fast access, second slower. Glass half full at least.
You could also connect the computers via thunderbolt and set up a thunderbolt bridge rather than LAN, if hard-wire is an option.
On PC, no idea.

LDS · Nov 8, 2017

yeahright said:
Are the NAS bay cases mechanically suited for transport so that I can simply put the backup drives in bays 3 or 4 and take them out after backup to store elsewhere (instead of external HDD cases), or are they merely delicate frames that still leave all the HDD contacts exposed and vulnerable so that it's unwise to cram them in a bag for transport?

Usually, no. Some NAS even don't have disk frame at all, you simply insert disks inside the slots. There are disks carrying cases or protective boxes for storage, if you need it.

yeahright said:
What backup strategy: incremental or always simply mirror the NAS onto the backup drive? Will 8 TB for the backup drives for incremental backup be sufficient, or should they be larger to accomodate the larger size of incremental backups (since they also store previous file versions)?

It has to suit your needs - including backup and recovery times. Incremental backups are quicker, but may slow down a full recovery - but also require less space to keep more copies of a file modified several times, and perform a point-in-time recovery. There's not a size that fits all - depends on your storage space availability, how back in time you need to go, the frequency of file changes, etc.

yeahright said:
Are backups in a proprietary format using proprietary software dangerous anyway and it's better to have all your backups in a standard file system so that they can be read by the operating system without the use of backup software?

Dangerous, if you use a reputable application, no. You'll be able to access the backup through the vendor software only, though.

Good backup applications are designed to track what they backup and often to add enough data to check for errors, and remediate them if possible. They can also automate many tasks (scheduling, snapshots, compression, deduplication, recovery, verification, etc.).

Some simple tools just make file copies, they just automated it, more advanced ones may use their own backup formats and databases to speed up file searches for retrieval.

Making copies to a standard file system is OK, especially if you have the experience to set everything up so backups don't require many manual steps to be repeated every time, which can lead to mistakes. Again, that depends on your needs and how complex your backup needs are. Backing up a few directories can be done with simple copies using utilities like robocopy.

cayenne · Nov 8, 2017

PrivateByDesign: Just curious, what is your network speed internally?
I find I have little to no lag I can notice on my NAS units......

Just curious.....

Everyone have a great day!!

C

CSD · Nov 8, 2017

Jopa said:
Unfortunately all proprietary solutions more or less suck. If something happens to the controller, firmware, data integrity - you're completely on your own. It's usually very hard to recover a proprietary file system. The "factory-built" NAS devices are usually small and lack proper cooling, which is also critical for mechanical hard drives. Ideally - build it yourself. The drives are cheap (large size but slow RPM), Linux is free. You need a good RAID controller with a battery backup - performance and recoverability will depend on it, and a decent size enclosure with a bunch of fans. Any motherboard, CPU and 16GB of RAM will work. For extra reliability you can add a UPS that could shut down your storage machine in case of a power outage. You can buy all parts on NewEgg, tax free.

You can mount Synology RAID into a Linux distro to recover data quite easily, fans are fairly easy to swap out and to source replacements. Also Synology and QNAP offer ease of use and regular, tested updates to the core OS. If you're a geek then go for roll your own, but for everyone else then a COTS NAS is the best solution.

CSD · Nov 8, 2017

Some factors to consider when buying a NAS:

1) How are you going to back it up?
2) Does your network support jumbo packets/VLANs?
3) What other purposes do you envisage the NAS to do (VPN, OS Back up/version control, CCTV station etc).
4) How much data do you generate per annum and envisage for the future?

I currently have a 4 bay NAS to 4 bay NAS both with 8Tb drives that mirror each other on separate sites. The main data/backbone network is on it's own VLAN to facilitate throughput and jumbo packets. This is segregated from day-to-day traffic which wouldn't benefit from the jumbo packets. The NAS is also connected using a dual link 1Gb/e connection (link aggregation). You'll need a managed switch to be able to create a VLAN and facilitate LAG connections.

I also have a LTO tape drive for back ups, these are for archival and off-site storage. I know this is overkill for a lot of people, but I prefer to use something I can grow into.

Note: RAID isn't a back up solution, it's purely for up-time. RAID data can easily become corrupted or suffer from bit rot. That's why you need a secondary solution to back up the NAS whether it's a NAS-to-NAS or NAS-to-Offline solution such as a external drive or tape.

privatebydesign · Nov 8, 2017

cayenne said:
PrivateByDesign: Just curious, what is your network speed internally?
I find I have little to no lag I can notice on my NAS units......

Just curious.....

Everyone have a great day!!

C

Everything but my modem is 1Gb (though I haven't done actual read write tests I never seem to have the time!), my router is the DNS assigner for the network including my office switch, cables are 5e or 6. I know I do not have it running optimally and would love to find a QNAP orientated site that doesn't go off into tech speak in the first sentence or that doesn't just push you into their Apps.

Search

Search

Storage options - RAID?

MusoD

pwp

MusoD

LDS

LDS

Zeidora

MusoD

cayenne

MusoD

Mikehit

privatebydesign

Jopa

scottkinfw

Wildlife photography is my passion

yeahright

Zeidora

LDS

cayenne

CSD

Photographer, WP Developer, and IT Geek.

CSD

Photographer, WP Developer, and IT Geek.

privatebydesign

Similar threads