Network setup for workgroup editing

[Stephen Armour]

Originally Posted by Harm Millaard

I have 12 disks, 1 TB each in a raid30 on an Areca ARC-1680iX-12 controller with 2 GB cache and a BBM (battery backup module) with expansion capabilities for another 4 disks. Due to the nature of raid30 (2 raid3 arrays of 6 disks each and then striped to form a raid30) of course you lose 2 disks for parity, so the effective net space is only 10 TB.

Harm and Pete and any others, this is very off-post, but if you guys could give me your two bits on this question in a separate post or email, I'd sure appreciate it:

Three of us work as a team on all video productions, with 3 workstations now and probably a fourth later. All are quads, all have Production Premium CS3 or CS4, all use Cineform HD, all are in a Gb peer-to-peer network and all use 64bit Win OS (x64 still). All productions are 1920x1080p. Since these are long term productions with a number of target distribution channels, and sub-titled in 21 languages, regular backup and near-oline storage of all material is necessary for corrections and updating. One person does all AE comps and color grading, I am the main editor, and one is our second editor/flash/website/etc, with another soon to join the team.

Main prob is continuously having to duplicate projects and material and backup each ws. We have been seriously looking for a unified workgroup solution that has the throughput and ease of a single project point for all filesharing and backup. Gb ethernet is too slow, but we are a non-profit org. so funds can also be a problem.

Any suggestions? Throughput speed and stability between ws is absolutely essential, as is compatibility between us. Program duplication is necessary, as we often use any empty system for rendering when available, then use laptops for emails, research and smaller stuff.

Sorry again for the offpost, but I've got your attention here and need experienced suggestions with this to try to find a better solution.

[Harm Millaard]

Stephen,

I dropped this question with my son, who is a network consultant. We'll both discuss it and I will get back to you ASAP.

[Steve Kalle]

Stephen: if you had the money, a fiber channel SAN would be the fastest. However, something you can do easily and for little money is this: (assuming each pc has a PCI-Express x4 or greater slot or PCI-X slot available) get an Intel Dual-Port Gigabit network card(NIC) and then you can "Team" the 2 connections together to double the throughput. I also highly recommend the D-Link Gamer Gigabit Router and Gigabit Switch. A few months ago, I setup the IT infrastructure for a small office; I built some pc's and bought the rest from Dell; and I used the D-Link router and switch. I was blown away at the speed from the Dell's to the server - 70-80MB/s. This was just one gigabit port.

1) What are your current speeds if you were to copy a file from one pc to another?
2) Ethernet cable - Cat 5, 5e or 6? You need Cat 6 for maximum throughput, so you might need to rerun the cables.
3) What is the current layout of your computers?

I should add that you can also get 2 single-port Intel NICs in either PCI-E x1 and/or PCI if you don't have a PCI-E x4/PCI-X slot available.

To have a centralized workflow, you need a server. I don't recommend a NAS or something like the Drobo because their speed is always limited. For the prior mentioned IT project, I used Acronis Workstation on all computers and had them backing up to the server. With Acronis, I could also backup to a 2nd destination with that being on the host pc.

If this sounds good to you, let me know and I can answer more questions.

[Harm Millaard]

Stephen,

Just as Steve figured, the (not easily affordable) solution would be to have a look at fiber SAN switches and SAN solution like HP MSA Storageworks 1500. However, this is based on your remark about IP being too slow. In addition to Steve's remarks about teaming, you may also have a look at jumboframes to ease the overhead on your network, but your NIC's and switches need to support that. If that is achievable, you can also have a look at some iSCSI NAS solutions like Thecus.

Fiber SAN would be optimal but costly. A very basic switch starts at around € 2500...
or look here: http://www.superwarehouse.com/Fiber_...itches/c3/2178

[Stephen Armour]

Originally Posted by Steve Kalle

Stephen: if you had the money, a fiber channel SAN would be the fastest. However, something you can do easily and for little money is this: (assuming each pc has a PCI-Express x4 or greater slot or PCI-X slot available) get an Intel Dual-Port Gigabit network card(NIC) and then you can "Team" the 2 connections together to double the throughput. I also highly recommend the D-Link Gamer Gigabit Router and Gigabit Switch. A few months ago, I setup the IT infrastructure for a small office; I built some pc's and bought the rest from Dell; and I used the D-Link router and switch. I was blown away at the speed from the Dell's to the server - 70-80MB/s. This was just one gigabit port.

1) What are your current speeds if you were to copy a file from one pc to another?
2) Ethernet cable - Cat 5, 5e or 6? You need Cat 6 for maximum throughput, so you might need to rerun the cables.
3) What is the current layout of your computers?

I should add that you can also get 2 single-port Intel NICs in either PCI-E x1 and/or PCI if you don't have a PCI-E x4/PCI-X slot available.

To have a centralized workflow, you need a server. I don't recommend a NAS or something like the Drobo because their speed is always limited. For the prior mentioned IT project, I used Acronis Workstation on all computers and had them backing up to the server. With Acronis, I could also backup to a 2nd destination with that being on the host pc.

If this sounds good to you, let me know and I can answer more questions.

Originally Posted by Harm Millaard

Stephen,

Just as Steve figured, the (not easily affordable) solution would be to have a look at fiber SAN switches and SAN solution like HP MSA Storageworks 1500. However, this is based on your remark about IP being too slow. In addition to Steve's remarks about teaming, you may also have a look at jumboframes to ease the overhead on your network, but your NIC's and switches need to support that. If that is achievable, you can also have a look at some iSCSI NAS solutions like Thecus.

Fiber SAN would be optimal but costly. A very basic switch starts at around € 2500...
or look here: Fiber Channel Switches

Thanks much for the comebacks. (If I could move this post to a separate thread, I would...sorry!)

I have experimented a bit with teaming single port Intel NIC's, as they are fairly cheap, but really didn't see the gain I was expecting. What I did gain was better reliability.

As to true throughput, it varies quite a bit, as right now, we have a mix of two Gigabyte boards (x38 and x48) and one Intel server/ws board. I shouldn't complain about some of our throughput, as sometimes it's pretty fast (+500 MBps sustained, ie 70+MBps).

For sure, the bottleneck for a single-access point server would be the GbE, though. FC is definitely superior, but costs shoot straight up very fast with enterprise drives and all.

Sigh.

I guess the real bottom line is to sell a car and get something like WinchesterSystems stuff? That RAID 6 security of theirs sure looks good, but I know I could build something "almost" as good for less than half probably.

It's hard to be in the "tween zone" of needing "enterprise class speed and security"...but being little guys...and working non-profit in Brazil to boot.

I really appreciate any more info you might have for this scenario though. If the conditions come soon for a good upgrade, I want to be ready to jump, and only guys like you with experience can see all the factors at stake.

Just one more comment. It seems that an SSD in the server/RAID mix somewhere could surely boast things significantly if it was serving as a "cache" for current projects? We could take the hit with older projects off the HDDs with speed, but get enormous speed gains with current cached material off the SSD.

Any light on this or other possibilities? (thanks much)

Oh, forgot to answer Steve's questions: all cabling is CAT5 and we are within 10-15 metres of our Netgear GS 108T switch. Jumbo frames are ON (4K), though I saw no gain with larger frames than 4K. All systems have a least a single RAID 5 and one has dual RAIDs (RAID0 and RAID5), with single, mirrored bootdrives, and single disk mirrors for the RAID0 (cheaper) via MirrorFolder.

[Steve Kalle]

Thanks Harm for reminding me about iSCSI. I just read a whitepaper comparing 1Gb iSCSI vs 2Gb Fiber, and it appears that you should be able to combine ethernet ports for iSCSI to increase throughput.

Harm's son should know more but I find this very interesting as I always like to make things go faster.

[Steve Kalle]

Check this out:

EnhanceRAID RS8 IP (iSCSI) Benchmark

The 2nd benchmark with MPIO(Microsoft Mulitpath I/O) which doubles the throughput by using 2 ports and increases reliability.

[Stephen Armour]

I think the teamed ports could theoretically work well for small workgroups, but the latency under Windows seems to be not very good. I'm trying to imagine 3 workstations, all pulling multi-gigabyte HD files at once and somehow have never seen GbE run fast enough or well enough to do that consistently.

I added up our HDD's here and we have 16 HDDs installed (not including older 10K Cheetah's in an older dual processor ws), and often another couple attached via external USB or e-SATA to retrieve older data. It's faster that way, than our GbE SANs. They are slooooooooow bad.

If we added it all up, we'd easily have plenty of drives for a fast NAS RAID. All are 1 TB or 500 GB and 7200 rpm. But, I'd rather start from scratch with any new system and use these current disks for duplicate offsite backup. Our internet link is too slow to do it that way.

[Steve Kalle]

The iSCSI protocol would greatly help with the latency.

With a centralized server, why would you need to download the video to each workstation?

[Stephen Armour]

Originally Posted by Harm Millaard

...Fiber SAN would be optimal but costly. A very basic switch starts at around € 2500...
or look here: Fiber Channel Switches

Harm, for that price, it might be cheaper to go to 10 gigabit ethernet? This Netgear switch has 4 of it's ports running at 10Gb:

Newegg.com - NETGEAR GSM7328FS 10/100/1000Mbps + 10 Gigabit Switch 4 x RJ-45, 24 x SFP, 4 x 10 Gigabit Ethernet/24G Stacking Module Bays, 1 x RS-232 8K MAC Address Table 334 KB embedded memory per port Buffer Memory - Switches

Not sure how much the NIC's are though, or how much you'd actually gain. Maybe it'd be cheaper and faster to just to use 10GbE NIC's and run direct to the server, bypassing the switch? Not sure how that'd work, but would be fast if it did. The server would be the switch and NAS, maybe?

[Steve Kalle]

I was looking at 10Gbe on newegg too but I thought it was out of your budget. Do you have an idea what your total budget might be?

However, the 10Gbe latency and reliability would be similar to regular 1Gbe because they are the same protocol. Actually, 10Gbe iSCSI would be better and then all you need is the storage unit (ie no server).

[Stephen Armour]

Originally Posted by Steve Kalle

The iSCSI protocol would greatly help with the latency.

With a centralized server, why would you need to download the video to each workstation?

You wouldn't "download" for sure. But you'd have to have enough throughput to create previews locally or try to run them from the server. Can't imagine how that'd work with Premiere, though. If you can fool it into seeing the server as a "local drive", it'd work I guess.

I've never tried to run multiple workstations with Premiere from the same location on remote network drives (I've never had enough speed to do that), but can imagine Adobe somehow blocking that. Plus all those needing 3-5 streams of video for any "realtime" output under Cineform...

[Stephen Armour]

Originally Posted by Steve Kalle

I was looking at 10Gbe on newegg too but I thought it was out of your budget. Do you have an idea what your total budget might be?

However, the 10Gbe latency and reliability would be similar to regular 1Gbe because they are the same protocol. Actually, 10Gbe iSCSI would be better and then all you need is the storage unit (ie no server).

We're jumping each other's posts. I can surely see the advantages for iSCSI if it's direct-to-storage, as Premiere (and all apps) would then see it as a "local" drive.

Our "budget" is completely relative. If we really get against the wall, we'll let interested parties know of our major needs and sometimes the resources will come in. I'd need some true TCO figures to even know if it was something we could shoot for.

Any other good links on solutions like that are appreciated. Especially any heads up comparisons.

[Steve Kalle]

Free Data Storage Software V6 Lite (DSS V6 Lite)

This is free to download and try out with a maximum 2TB. Allows you to test iSCSI on your current workstations as well as using two gigabit ports in MPIO(don't know if they must be identical). I remember seeing one benchmark with the Open-E server getting 100-110MB/s on a single Intel NIC.

It seems this software/OS allows you to use regular hardware rather than buying a premade iSCSI storage system which should save some money.

[Stephen Armour]

Originally Posted by Steve Kalle

Free Data Storage Software V6 Lite (DSS V6 Lite)

This is free to download and try out with a maximum 2TB. Allows you to test iSCSI on your current workstations as well as using two gigabit ports in MPIO(don't know if they must be identical). I remember seeing one benchmark with the Open-E server getting 100-110MB/s on a single Intel NIC.

It seems this software/OS allows you to use regular hardware rather than buying a premade iSCSI storage system which should save some money.

For sure something that robust would give great flexibility to us. I think it comes bundled with a pretty great variety of hardware SAN and NAS vendors, though, so is cheaper that way too.

Might be overkill for a 4 seat outfit, though. Although sometimes the main feature sets are worth getting, even though you get all the bells and whistles too, so it's a candidate for keeping in mind for sure.

At this point, the most important thing for us, is the cost/benefit. If the gain is great enough, the pain of robbing Peter to pay Paul is sometimes worth it. It's getting that true picture of what "gain" is, that is hard. For us, it seems to be consolidated speed, accessibility, long term data security, TCO, and reliability. That way, no matter when a ws goes down or when we fry another HDD, we can keep on truckin...