Input on a PC Build for Video Editing

[Bart Walczak]

Originally Posted by Randall Leong

Actually, with some NLE software, RAIDs are required especially for HD material because such software actually decompress compressed video material to compressed RGB and then recompress it back to the original format on the fly. And just a single track of 1080i/p uncompressed RGB video requires well over 200 MB/s (1.6 Gbps) worth of disk throughput.

Ummm... doesn't the decompression happen in the memory? You're implying that the software reads compressed, writes uncompressed, and then reads uncompressed again to play back. I don't know any NLE that actually does that. Some compositing apps like Nuke or After Effects, perhaps Mocha do cache the uncompressed frames for playback, but they do it for another purpose.

I've been able to easily work with Premiere Pro without RAID, with an older drive that had about 60-70 MB/s throughput. Granted, no multicam edits. But you don't need top shelf hardware to get a decent experience.

[Randall Leong]

Originally Posted by Bart Walczak

Ummm... doesn't the decompression happen in the memory?

Actually, depending on how much system RAM is installed in the computer, the decompressed frames are written to, read from and then dumped from the disk's media cache folder that many NLEs use. As a result, if you have a really slow disk system, you often have to render the timeline (previews) first in order to get decent playback especially if your system has only a small amount of system RAM installed. No wonder why Premiere Pro CS6 runs at its best with more than 16GB of RAM installed: If you have less than 16GB installed, the decompressed frames will use the media cache disk far more frequently.

[Vincent Anthony]

Alan & Harm, I have the following components for the build listed below. I would like to ask if you know of any good links/tutorial offhand that describe how to get the ASUS P9X79 up and running correctly with RAID enabled?

I have read that the bios should be updated, followed by the Marvell and Intel Drivers.

Some basic info Info I need is for:
Which port should the OS drive be connected to?
Which ports should each pair of the RAID0 drives be connected to? (4 drives total)

I have the following links for the RAID setup, etc:
How to setup a RAID System: How to Setup a RAID System | Hardware Secrets
Problem setting up RAID0 on ASUS p9x79Pro: Adobe Community: Problem setting up RAID0 on ASUS p9x79Pro

Parts list:
Motherboard: ASUS P9X79 PRO LGA 2011 Intel X79 SATA 6Gb/s USB 3.0 ATX Intel Motherboard with USB BIOS
CPU: Intel Core i7-3820 Sandy Bridge-E 3.6GHz (3.8GHz Turbo Boost) LGA 2011 130W Quad-Core Desktop Processor BX80619i73820
Video Card: MSI N660 Ti PE 2GD5/OC GeForce GTX 660 Ti 2GB 192-bit GDDR5 PCI Express 3.0 x16 HDCP Ready SLI Support Video Card
OS: Windows 7
RAM: G.SKILL Ripjaws Z Series 32GB (4 x 8GB) 240-Pin DDR3 SDRAM DDR3 1600 (PC3 12800) Desktop Memory Model F3-12800CL10Q-32GBZL
Drives: Western Digital WD Black WD1002FAEX 1TB 7200 RPM 64MB Cache SATA 6.0Gb/s 3.5" Internal Hard Drive
Drive Config: 1=Operating system, 2x1TB=RAID0, 2x1TB=RAID0 (Backup will be to dual 3TB External Drives)
Power Supply: CORSAIR HX Series HX850 850W ATX12V 2.3 / EPS12V 2.91 SLI Ready CrossFire Ready 80 PLUS GOLD Certified Modular Active PFC Power Supply
Burner: ASUS Black 14X BD-R 2X BD-RE 16X DVD+R 5X DVD-RAM 12X BD-ROM 4MB Cache SATA Blu-ray Burner BW-14D1XT
Case Speaker: APEVIA Model CVTCSPK 2" Case speaker
CPU Cooler: COOLER MASTER Hyper 212 EVO RR-212E-20PK-R2 Continuous Direct Contact 120mm Sleeve CPU Cooler
Case: Fractal Design Define XL Black Pearl w/ USB 3.0 ATX Full Tower Silent PC Computer Case
Card Reader: Kingston's USB 3.0 Media Reader Computer Memory FCR-HS3
USB 3.0 BackUp/Export Drives: Seagate Expansion 3TB USB 3.0 Black Desktop Hard Drive STBV3000100
Extra Case Fan: Rosewill ROCF-11003 Hyperborea 140mm Case Fan

[Bart Walczak]

Randall, with due respect, I think you're confusing several issues here.

If what you were saying was true, then I would have never been able to reach any sensible playback in any NLE. Full 1080 frame decompressed is 1920x1080x24 bits = 6,2 MB. Even with only 1 GB for the video data you can fit about 160 frames, which is more than enough to get a decent playback. In fact, you most likely don't need more than 4-5 frames if you are streaming in real time. Which means you could feasibly display quite a lot of 1080p streams even with 1 GB of RAM, assuming no other bottlenecks in the system.

Also, assuming a video with data rate 26 Mbps, for a typical drive that has more than 100 MB/s throughput you could theoretically fit about 30 streams. Heck, you could put about 10 streams through the old USB 2.0 interface! The reason you can't in the real time is the drive's seeking time - playing several files at the same time is a serious performance hit on the transfer rate. Even RAID systems are not free from this problem, this is why 10,000+ RPM drives were invented, and this is where SSDs really shine.

The true bottleneck with compressed video is the CPU power. Some compression types are more CPU-intensive, than the others. AVCHD is very hard, MPEG-2 is easier. Long GOP formats are atrocious for random seeking, because the whole GOP has to be processed to get to the single frame, sometimes meaning 75 frames or more. This is perhaps the only place where possibly the amount of RAM could have any impact at all on video decompression.

The only places where RAM is important are computationally heavy effects that are not accelerated by the GPU on a multi-core machine perhaps with floating point (32-bit vs 8-bit) color space. On a multi-core machine, each core gets assigned its own amount of RAM to do things. If you have 8 GB on a 4-core machine with threading means that each core gets at the most 1,5 GB of RAM, perhaps even less. For some types of image manipulation this might not be enough, and this is why 16GB is better than 8GB. Plus you have to hold the project structure, caching, etc. But it has very little to do with compressing and decompressing actual video, and even less with writing decompressed frames to the swap drive, especially during playback.

The final proof is in monitoring the disk usage during playback. I hardly ever get writes during playback even with effects. 99% of time I get reads, and guess what? They are at about the bitrate of the compressed footage that my NLEs pull from the drives. This is why you can easily edit from an external USB 2.0 drive. The main culprit for dropped frames with compressed video is the CPU. With uncompressed it's mostly HDD throughput.

That said, I do agree that for optimal performance you need to balance your system's RAM against the amount of cores that you have, and ideally you'd have at least 4GBs per core. But even with 32 GB on my main machine, Premiere hardly ever comes over 10-12 GBs of usage during typical work.

[Noa Put]

What type of footage are you planning to edit? If you are uncertain how to set up a raid system correctly it might be better just to use the drives separately, WD black drives are plenty fast for editing mpeg2 or avchd footage, if you are working with uncompressed data and doing a lot of multi-cam, then faster raid drives are better. Also, your motherboards manual should give you the needed info how to connect the drives for a raid setup, is there no way you can download the manual to see what the possibilities are?

[Chris Medico]

Here is another thought on setting up a high speed data drive.. How large are the projects being worked on? It would be rare for them to be really huge unless it was an uncompressed workflow. For something like that I would strongly recommend an offline/online setup..

BUT

If you are looking for speed and the projects aren't huge why not consider something like this:

http://www.bhphotovideo.com/c/product/897042-REG/Samsung_mz_7pd512bw_512GB_2_5_inch_SSD_840.html
This thing is faster than any motherboard based RAID0 is going to be (it is basically is a RAID internally using flash chips). If you want to get really crazy you could always RAID a couple of those SSDs. I can't imagine you needing more data flow than that. :)

[Noa Put]

I have read some concerns about SSD having a limited write and read cyclus which was the reason why they are best used for the OS. I"m not so sure about it though but might be something to think about before putting any critical data on it that is going to be access almost constantly. I also think Vincent pointed out he had to watch his budget and these 500 gb SSD are very expensive compared to regular HD's.

[Chris Medico]

While SSDs do have limited write cycles, it isn't an issue with the application because if you have a dedicated source media drive in your system it only gets written to when you load on a new project. In reality it is a problem that is somewhat overstated. When you look at the cycle limit of a professional grade SSD it is high enough that it won't become an issue for more than 6-10 years of use. Are any of us expecting our conventional hard drives to last that long? I certainly don't.

This drive isn't cheap but it could be cheaper than building a RAID0 to equal its data performance. To get to the continuous READ/WRITE rate of the 840 PRO SSD it will require no less than 4 WD VelociRaptor 10,000rpm drives (and then only if they are under 50% full). That will get you a larger volume of storage but it will also consume much more power, tie up 4 SATA ports and not be any more reliable than a single SSD.

If you don't have a 4 port RAID controller built into your computer you will have to buy one of those as well. At that point the SSD is even cheaper in total capital outlay.

[Tim Polster]

Vincent, I just read the entire thread and I think a lot of your questions were answered. I can see by your post number that you are new here. You have to understand that your topic of discission is huge and has many ways to accomplish the end result. This same internet site that you are frustrated with has brought you all of the information for free. There is a lot to learn here. Sure you can not always control the theads but if you are nice to people the process goes a lot smoother!

To your question about RAID. If the RAID you are setting up is NOT for the operating system then you do not need to do anything special other than set the motherboard up for RAID in the BIOS and then create the RAID in Windows Disc Management. So first load Windows and get the rest of your system up and running then add the drives, and create the RAID.

The OS drive should go in the "port 0" SATA connector (grey color). Install the 4 drives for the RAIDS in the light blue connectors.

[Vincent Anthony]

Noa Put - Thanks! I edit 5D and 7D footage, with the occasional odd video camera thrown into the mix. No RED or anything really heavy. I also do work in After Effects, so that is something I really want this PC to do better in than my old machine. I am reading the manual and it seems straightforward. Just saw some warnings on other threads about the need to enable RAID in the bios prior to installing windows and then the need to install drivers during the windows7 install process. So I guess my question is when I enable RAID, do I do so for all the ports, or just specific ones? (I don't know how the bios screen or these options even look yet in the GUI) -- Also, should the RAID drives take up all 4 6GB/s Ports (2 Intel, 2 Marvell) and the OS drive just go on a 3 GB/s port? The manual doesn't get so specific as to point out where to plug in each drive and how to enable RAID. It assumes a minimum level of user knowledge that I don't fully possess at this point.

Chris - Thanks as well! SSD is something I'll do on the next build when the prices come down some. Also, I already put together the machine...just haven't turned the thing on yet because I want to go through the MB manual carefully first. The ASUS P9X79 Pro has RAID controllers built in.

Tim - Thank you. I guess I am just a little confused about the part about enabling RAID, then installing the appropriate drivers at the right time. If I take it slowly and step by step, I suppose I could get it running. Don't want to make any dumb mistakes though, as I have read of people making windows unbootable by attempting to enable RAID after installing windows. I'm a bit of a wise ass Tim, but you're absolutely correct in what you said about being nice. I get frustrated too quickly for sure.

[Tim Polster]

In your case, you only need to enable RAID support in the BIOS before you start installing Windows. Since you are not putting Windows on a RAID you do not need to worry about drivers. You are going to just create a RAID once Windows is up and running using the tools inside Windows.

I would start with only one RAID and leave two discs as single to start out. You can always reformat the two drives and create a second RAID later if you find that you need more RAID. With 5D & 7D footage I do not think you NEED a RAID to edit.

I would also get an SSD for your OS drive. Just a 120GB drive which can be had for about the same cost as a 1TB drive these days. They are fast.

You might want to put your OS drive on the 6GB SATA for more speed. So use the two grey connectors for non-OS drives and put the OS drive in one of the blue SATA ports.

After Effects likes a fast CPU and a lot of RAM which you have so your system should be nice once it is up and running.

[Noa Put]

Regular harddiscs do not benefit from 6GB/s Ports, they are just not fast enough but SSD's are. I"m not experienced in setting up raid arrays so I wisely leave those tips to others :) But like Vincent said, for dslr footage you don't need raid, it does however speed up copying data between drives but the actual editing relies much more on the cpu or gpu.
If you are using after effects a lot then fill your system with as much ram as it can carry and you can afford, AE east ram for breakfast.

[Harm Millaard]

Vincent,

With your intended motherboard you have:

2 x SATA 6G ports on the Intel controller,
4 x SATA 3G ports on the Intel controller,
2 x SATA 6G ports on the Marvell controller.

Before you start installing Windows from a DVD, choose the F6 option to install a raid driver for the Intel controller. This is the first question asked when booting from the DVD. Once that driver is installed the Windows installation proceeds as usual. Using the F6 option does not mean you HAVE to create a raid, it gives you the choice to raid or not, but if you do not use the F6 you never have that choice on the Intel ports. The Marvell controller does not require the F6 use during installation.

Conventional HDD's do not profit from a SATA 6G connection, they are too slow, but SSD's can profit from a SATA 6G connection. DVD or BR devices must be installed in the Intel controller and the most logical port to use is one of the SATA 3G ports.

With these general comments out of the way, the question is what is the best setup in your case with your intended disks?

C: OS & programs & static pagefile (if using Win8) on SATA 6G Intel, single disk.
D: Media & projects on SATA 6G Marvell, 2 disks in raid0
E: Media cache & previews on SATA 3G Intel, 2 disks in raid0
F: USB3 for exports and backup of mainly the D: drive.

This would be my suggestion. And the other suggestion I have is to exchange the Corsair HX PSU for the AX Gold label PSU.

Hope this helps.

[Sareesh Sudhakaran]

Originally Posted by Harm Millaard

C: OS & programs & static pagefile (if using Win8) on SATA 6G Intel, single disk.
D: Media & projects on SATA 6G Marvell, 2 disks in raid0

Is this a typo, Harm? I wouldn't advise RAID 0 for media or projects. For small projects RAID 1 would be perfect, and for large projects I'm leaning towards RAID 10.

On a side note, even with Prores HQ at 220 Mbps (27.5 MB/s) one should be able to read four streams on a Sata II 3 Gbps consumer drive at 7,200 rpm.

Sata III 6 Gbps is good to have as 'future proofing'. But if the future is 3-5 years away, some new tech might take its place. Unless one is transporting 'that much' data over SSDs on a regular basis, Sata III is probably overkill for DSLR and Prosumer footage.

I could be wrong. Are there any scenarios where this is not true?

[Tim Polster]

In reality, RAID 0 is fine if you have the data backed up.

I might have looked at the wrong mobo as the one I saw on Newegg had 6 SATA ports.

I agree, the 6Gb SATA is overkill for spinning discs, but can be used with a modern SSD for the Operating System.

Very difficult to get a concensus on computer building!