Storage options - RAID?

For years I've been struggling to find and not finding a good solution. I think I may have finally found it earlier this year:

https://www.datoptic.com/

I now use their Thunderbolt 2 8-bay RAID towers with my Mac Pro.

Notes:

* I use only enterprise HGST He10 hard drives (10 TB now, and formerly 8 TB).
* I use a tiered storage approach.
* Most photos, anything older than six months, are stored on enterprise RAID 1 arrays and backed up on individual enterprise hard drives.
* Due to the speed of workflow, all photos newer than six months are stored on a four-drive RAID 0 array, and backed up onto separate enterprise hard drives. Those backup drives are never accessed or spun up except when backing up photos. The RAID 0 array is ejected and powered off unless work is happening with it. Previously I never would have considered RAID 0, until an expert pointed out that the wear and tear on hard drives is actually the smallest when RAID 0 is used, and the probably of a four-disk RAID 0 failure in five years with these drives is less than 5% even when the drives are in use 24/7. If failure does happen (which I simulated to make sure), then with the backup drives data can instantly be copied to my working SSD drive or or a new RAID 1 array, if/when the RAID 0 ever experiences failure.
* Over the years, like many others, my only failure experience has been with RAID 5 and 6 (the RAID 6 eventually recovered all data, but it was very scary). I would never recommend using RAID 5. It burns out drives four times faster than normal, and probability of failure during a rebuild is scary to say the least.
* No photos are ever deleted from the memory card unless at least two copies exist on separate enterprise hard drives.
* At the end of the year a third copy of all photos is made and put into remote storage.
* Perhaps most importantly, all the best photos have already been published / sent out to clients, so even if I lost all the photos, much of my best work would still carry on strong.

I hope this helps!

Helpful,

I am sorry, your advise in regards to Raid 0 and RAID 5 is far from being helpful

raid 0 improves write/read speed only (performance array) but increase probability of failure and data loss by 2 times - dramatically. should one out of 2 drives fail, you won't be able to rebuild the data set from the second one as each drive contains only a half of the entire data set.
Raid 5 is a redundancy array of drives. each bit of information being written to 2 drives at a time and then next bit is written to next pair of drives, and so on and so forth. you can add additional or replace existing drives in the array as you go. there is no risk of data loss as long as at least 2 healthy drives left in the system.
you statement about raid 5 wearing out drives 4 times faster is false. RAID 5 system is an old tech and your statement is not supported by industrial evidence. I am not suggesting that your RAID system is non functional , but your explanation is far from being ... helpful


]

helpful said:

For years I've been struggling to find and not finding a good solution. I think I may have finally found it earlier this year:

https://www.datoptic.com/

I now use their Thunderbolt 2 8-bay RAID towers with my Mac Pro.

Notes:

* I use only enterprise HGST He10 hard drives (10 TB now, and formerly 8 TB).
* I use a tiered storage approach.
* Most photos, anything older than six months, are stored on enterprise RAID 1 arrays and backed up on individual enterprise hard drives.
* Due to the speed of workflow, all photos newer than six months are stored on a four-drive RAID 0 array, and backed up onto separate enterprise hard drives. Those backup drives are never accessed or spun up except when backing up photos. The RAID 0 array is ejected and powered off unless work is happening with it. Previously I never would have considered RAID 0, until an expert pointed out that the wear and tear on hard drives is actually the smallest when RAID 0 is used, and the probably of a four-disk RAID 0 failure in five years with these drives is less than 5% even when the drives are in use 24/7. If failure does happen (which I simulated to make sure), then with the backup drives data can instantly be copied to my working SSD drive or or a new RAID 1 array, if/when the RAID 0 ever experiences failure.
* Over the years, like many others, my only failure experience has been with RAID 5 and 6 (the RAID 6 eventually recovered all data, but it was very scary). I would never recommend using RAID 5. It burns out drives four times faster than normal, and probability of failure during a rebuild is scary to say the least.
* No photos are ever deleted from the memory card unless at least two copies exist on separate enterprise hard drives.
* At the end of the year a third copy of all photos is made and put into remote storage.
* Perhaps most importantly, all the best photos have already been published / sent out to clients, so even if I lost all the photos, much of my best work would still carry on strong.

I hope this helps!

Click to expand...

SecureGSM said:

Raid 5 is a redundancy array of drives [...] there is no risk of data loss as long as at least 2 healthy drives left in the system.

Click to expand...

This is untrue.

In fact, RAID 5 uses one parity bit, so if more than one drive fails, you're toast. And when that one drive does fail, experienced sysadmins know that RAID 5 rebuilds are extremely dangerous. An amazingly high number of times another drive fails during the intense and slow rebuild process.

Here is what happens when you do just one write onto a RAID 5 array:

1. Read the old data
2. Read the old parity
3. Write the new data
4. Write the new parity

You are also incorrect about RAID 0. You stated: "raid 0 improves write/read speed only (performance array) but increase probability of failure and data loss by 2 times"

In fact, RAID 0 doesn't have ANY redundancy all all. It increases the probability of complete failure FAR more than just two times compared to other redundant levels of RAID.

However, the amortized incidence of failure per drive goes down with RAID 0 since it does not burnout the hard drives as quickly due to less reads and less writes. Also, with some extremely reliable enterprise hard drives, the chance that a small RAID 0 array will go bad is less than the probability that your RAM will go bad or your power supply will go bad, etc.

Finally, remember that RAID is NOT backup. No amount of RAID is sufficient. Just one messed up RAID controller, and even the best RAID array of all can be destroyed. Secure instant delete can even be activated inadvertently by hackers determined to do harm.

Most importantly, data must be backed up to separate disks or another RAID off of the main RAID. Period.

There are several good NAS choices, it depends on what you are looking for. I'd get a 4 -6 disk unit, it will handle not only your files, but your backups.

I prefer one that is networked, so it can be accessed by any computers or equipment on your network. One with a thunderbolt port is limited to what can use it, and has to be close to your computer.

Synology and Qnap are top choices, so look closely at them.

I'd get this one

https://www.amazon.com/QNAP-Professional-Grade-Attached-Supports-TS-453A-4G-US/dp/B017YB7T6U/ref=sr_1_1?ie=UTF8&qid=1510203114&sr=8-1&keywords=ts-453a

It has a celeron processor, not a atom, 4 network jacks, two HDMI jacks, USB3, its capable of playing 4K video which some cannot do.

Here is a good site to read reviews and see actual test results.

Certainly, Synology is very good, The DS916 is fast, but has no HDMI out. That only matters to some users who want to connect it to a TV set or video monitor. Pay attention to the Drobo review, its not so capable when compared to the competition.

https://www.amazon.com/Synology-DS916-8GB-DiskStation-Diskless/dp/B01EMSGNCY/ref=sr_1_1?ie=UTF8&qid=1510203673&sr=8-1&keywords=synology%2Bds916&th=1

http://www.storagereview.com/reviews/consumer/soho_nas

helpful said:

In fact, RAID 5 uses one parity bit, so if more than one drive fails, you're toast. And when that one drive does fail, experienced sysadmins know that RAID 5 rebuilds are extremely dangerous. An amazingly high number of times another drive fails during the intense and slow rebuild process.

Click to expand...

There are a very large number of small RAID5 arrays in use, and if it was so dangerous it would have been abandoned years ago, although, with larger and cheaper disks RAID6 or RAID10 became more common (and nested RAID like 50 or 60 as well, but you need far more disks).

The stories you heard often come from far larger systems, with tens or hundred of disks and heavy workloads, where the sheer scale needs different approaches, because multiple disk failures became a far bigger risk. A home or studio NAS with a few disks is a different league.

Anyway, anytime you need a full rebuild or restore, you put the source of data at higher strain and you increase its risk of failure, whatever they are the remaining disk of an array, an external disk, another NAS, your backup tapes, etc. etc.

helpful said:

Here is what happens when you do just one write onto a RAID 5 array:
1. Read the old data
2. Read the old parity
3. Write the new data
4. Write the new parity

Click to expand...

Actually, not always - that happens for small I/O operations, when data inside a stripe needs to be changed, and not the whole stripe. That's why the stripe size (and the number of disks it is divided on) you select when creating a RAID array is not irrelevant. A "full stripe" write doesn't need to read old data - the old parity is irrelevant.

It is important if your applications write data in small chunks. RAID5 was often used for databases, that usually handle file data in small chunks of a few KB each, on several disks, and do a lot of "random" writes continuously.

When you write sequentially your 25MB RAW, or 100MB TIFF, on a few disks RAID (maybe just 3 or 4), a lot of full stripe writes will occur, and the penalty will be far smaller - only for the part of data that doesn't fit a full stripe.

In turn, the more disks needs to be read to obtain a full file content, the more I/O operations on the disks will be required. In this regard, RAID0 or RAID5/6 arrays work in the same way.

That's why the workload type matters - write intensive ones need a different setup than read intensive ones. Most photographers' NAS will be used writing files on imports or backups once, and then mostly reading them.

That's why multiple configurations are in use - one may want to use a RAID1 or RAID10 for data that are changed frequently, especially in small chunks, and RAID5 or RAID6 for data that are mostly read, and changed infrequently.

I would use RAID0 only for data already existing elsewhere that have to be accessed at high speed (it was often used for video editing), but today, any decent SSD will outperform a RAID0, and will be more reliable.

helpful said:

Also, with some extremely reliable enterprise hard drives, the chance that a small RAID 0 array will go bad is less than the probability that your RAM will go bad or your power supply will go bad, etc.

Click to expand...

No. Spinning disk reliability is still far lower than RAM (especially ECC), CPU, and so on. And even a bad RAM module won't kill instantly all your data like RAID0 can. And enterprises hard drives are still used with some kind of redundancy anyway, because the costs of downtime and data loss can be too high.

Frankly, thinking a small RAID5 is more dangerous than RAID0 looks to me like being more afraid of a zombie invasion, instead of a flood or earthquake...

PS: sorry for the highly IT contents, but IMHO was need to put things in the right perspective.

I've used Raid 5 for many years and have had disks go down with no issues. Early Raid 5 systems were less reliable, and large industrial Raid systems that incorporate 12 or so drives may be more at risk merely because of the thousands of drives they use. Raid 6 is the answer if you have a large array of disks. I have a 6 drive array, and after 8 or 10 years of continuous use, one drive failed this spring, it was rebuilt successfully, but the drives were getting old, so I replaced them all. I also have a newer 4 drive array which runs 24/7.

In any event, a Raid system is not a backup. It gives high availability, but do a proper backup always.

Hi guys. This thread is many years old. Any new products in the market? I currently have 2 Drobos. Both are working fine but for obvious reasons I need to replace them. I need a simple storage system with Double Reduncy (Raid 1?). That's all. No need for NAS. Thank you! I hope that someone replies, or I may have to start a new thread....