Building a computer for hd

[Ron Evans]

I too do not use a RAID in my system yet happily edit HDV and AVCHD. My system has Q9450, 8G RAM, 250G boot disc, 250G temp disc, 2 x750 video storage and three external eSATA drives, 2x 500G and a 1T. I use Edius to edit and convert the AVCHD to Canopus HQ but edit the HDV native on the timeline realtime in a three camera multicam edit with no problems. I keep a regular image of the Boot drive on one of the external drives and a rescue CD to boot from. In editing/rendering I always arrange for read and writes to come and go from different drives so that one drive doesn't have to read and write. Performance wise this is faster than a two disc RAID that is only about 1.7 times single drive speed for a single action ( read or write) because of head movement wereas the single drive case can stream the files if they are contiguous on the drive. IF I was in a production environment rather than a hobby then I would use a multidisc RAID with fast discs too but more than two discs in the array for sure for the reasons described earlier. About the only reason to use just one 2 disc RAID is to make life easier since it just looks like a big drive. But in this case it may be slower than two separate drives is some cases.

Ron Evans

[Tripp Woelfel]

Ron... Well said. There are performance gains to be had without a RAID but to this point, in this thread at least, no one has pointed them out.

A truly wise man looks at the whole situation from many angles and devises the best strategy to address it. Nicely done.

[Shaun Roemich]

Originally Posted by Graham Hickling

Keep in mind the datarate for HDV is no higher than for DV - I agree with Tripp's comments on Raid being potentially more hassle than it's worth.

The data rate is no higher but the interframe nature of MPEG-2 requires more processor horsepower.

Edit: My previous statement, while correct, is irrelevant as part of the discussion it is part of.

[Ken Steadman]

I use Raid 1+0 but honestly I am more concerned about losing a drive and losing footage. The Raid O is just to speed up the drives so I don't lose so much speed in the mirror.

[Justin Hewitt]

Seperation of read and write drives offers the most basic level of performance gain because you avoid "drive head contention".

Using one array (RAID 0 or other) to perform simultaneous heavy READ and WRITE (eg render ops) probably does offer less benefit than just having a two seperate disks, one reading, one writing. (benchmarking needed to confirm this)

If both your READ and WRITE drives are arrays, that will beat two seperate single drives for performance, because the READ array can channel more data to the processor so the processor has fewer WAITs and the WRITE set is able to keep up with the demand to offload processed data so the processor has fewer WAITS.

How you scale up drives into arrays is dependent on budget and your need for fast productivity. For the home hobbiest two drives are sufficient, unless your playing with RAW . Render times are not critical ...
For pro's ... You scale up your hardware until you are able to cover your workloads in a time frame that's satisfactory.

In the film and game industrys; I have heard of companies using render farms of hundreds of boxes to process single render jobs. Thats big scale, because time is money ....

OT:
RAID 10, Stripe + Mirror, from practicle experience has a slower write time than RAID 0, but still offers very fast READ times. Its biggest advantage and why its the array of choice for big database storage is its redundancy.

[Graham Hickling]

I suspect part of the debate here is being fueled by a lack of clarity/agreement on what we all mean by "performance".

For example, I color correct all my footage and so performance gains for me come from a faster CPU, or in the case of tools like Colorista a powerful GPU. So at present my non-RAID drives essentially hang around waiting for me to save up the $$$ for a CPU that can match their exisiting datarate.

[Justin Hewitt]

Good performance is when tasks complete within an acceptable timeframe. REALTIME is always the goal.

Jobs lengthen in time when the sum hardware or a component of, hits a limit. The processing bounces along the ceiling until it completes. Run the same job on much bigger equipment and the job will complete faster because it is not constrained by same level of limits.

Jobs that 10 years ago would been considered very heavy for PC class equipment, are now almost instant on your iphone or windows PDA. However, jobs now considered heavy for current PC grade equipment, 10 years ago would have only been possible on big mainframes .... Our hardware is getting cheaper, bigger and faster, but also the amount of data that we are processing and the level of complexity is also getting bigger, so processing times in relative terms are not getting shorter ....

eg. 2 years ago, let say it took 2 hours to render a DVD on your PC at the time, that same job now on your new PC takes 30 minutes, so you have made a great performance gain. But clients now want HD quality on BluRay. So that takes your current PC 8 hours to produce ....

With performance you have three choices , scale vertically , scale horizontally or a combo

If the jobs you do are very heavy for floating point OP, then verticle scaling of the ram and CPU/GPU will offer the largest performance gain.

If your job is very large data processing, RAID arrays are an example of horizontal scaling;
eg. moving from 2 disk RAID 0 to 4 disk RAID 0

however, if you had existing arrays of SATA 7200 spin seagates, you could vertically scale these to 10K SATA or even 15K SCSI to increase performance.

[Tripp Woelfel]

Originally Posted by Ken Steadman

The Raid O is just to speed up the drives so I don't lose so much speed in the mirror.

Generally speaking, a two disk RAID 1 mirror will have similar write speeds as a single disk but its read speed can be a little faster. With RAID 1 a read will come from the drive whose head is physically closest to the data you want on the disk.

RAID 1 writes can be slower than a single disk if your configuration doesn't mark the write complete until it physically writes the data to the platter, which I don't think is the norm. Most systems mark the write complete when it's moved to the drive's memory buffer, which is good for speed but puts data at risk if there is a power loss.

[Eddie Coates]

I too have an i7 920 CORE with 12 gigs of RAM and a ATI 4870 1gig GPU

This machine renders full HD mpeg4 video straight out of the cam FAST!!

I am currently using CS4 Prem Pro and it will edit files straight from the EX1 cam!

I just rendered a 15 minute 1920x1080 30p clip in about 21 minutes!!

i7 core is the way to go all the way!! It blows away everything else out today!

To me it does not matter anymore PC or Mac , its all in the hardware now.
Mac and PC are both great with CS4 Prem Pro !!

[Nikolaj Marquez von Hage]

Justin, I'd like to understand what you say about disks dedicated for reading, and disks dedicated for writing.

As I understand you, the source footage (for example, MPEG files for native HDV editing in Premiere) should be placed on the READ disks.
Exported movies should export to WRITE disks. Is that correct?

Render files should also, at rendering, be written to the WRITE disks?

If you could clarify just what kind of files should be where, I'd be deeply grateful. Planning to update to a DVCPRO HD editing system.

[Ron Evans]

Nikolaj , my system currently has a 250G boot, 250G temp, and two 750G for video files. I also have two external eSATA 500G and a 1T. All programs use the temp disk for their temp files and I arrange for source files to be on one disc and final render to another. I have a lot of choices. Motherboard is an X48 with Q9450, 8G RAM and running Vista 64. For most of what I do the CPU is still the limiting factor not discs. It is my understanding that a two disc RAID 0 is about 1.7 times the speed of a single disc in a single read or write operation so recommendation is to use at least three discs. If the one RAID 0 has to read and write at the same time, and deal with temp file operations it has a high probability of being a lot slower than separate discs. With the current discs having sustained transfer rates of over 50Gps( HDV or DV only needs 3.5G) and large capacity I see no reason to use a RAID. Uncompressed files may be different.

Ron Evans

[Nikolaj Marquez von Hage]

Excuse my ignorance, but what exactly is the temp disk for then?

READ DISKS: source material
WRITE DISKS: render files, export files

Where does "temp" fit in this picture?

[Ron Evans]

IF you look in the preferences of all the NLE's there is location for temp files( normally under tools>preferences>general ) Current versions of Vegas have just one location needed but Adobe CS3 has 7 locations needed) If you do nothing to redirect these temp/scratch files will be processed in the default folder normally where the program was set( boot drive "C" !!!). This can lead to the boot disk having a lot of disc access. The OS and all these program uses as well as these temp/scratch files. This can easily become the performance bottleneck hence the need for a temp drive. That way the OS can access the boot drive, the program can also have assess to the boot as well as use scratch files on this temp disc, but high volume process and transfer will be done between the read and write discs. These temp files are used mainly for preview etc but may be used for pre-rendering too.

Ron Evans

[Nikolaj Marquez von Hage]

Thanks Ron for that clarifying description!
Regards,
Nikolaj

[Ron Evans]

Having just re-read my post I should correct the data rates to be 50MBps for disks transfer NOT Gps and 3.5MBps for DV and HDV, I think I must have been half asleep writing that!!!!

Ron Evans