4:4:4 12-bit Uncompressed DVX100

[Rob Lohman]

I'd say the major problem is storing all of this. Even if you can
get a RAID array to store the information that fast you would
have to carry your PC with you on your shoots? And not to
mention how to edit it, but that's a more easily solved problem.

A working prototype would definitely be the way to go. Also
to see what the difference in qaulity would be. Put the camera
on a tripod and take a framegrab of the firewire stream from
a highly detailed scene and at the same time from the 4:4:4
12 bit out as well. That way you can easily see in any paint
program what the increases are.

What camera controls will be missing with such a system?
(gain & white balance at least I'd assume)

[Stephen van Vuuren]

Juan:

I encourage you to move full speed ahead. Based on what people pay for increased rez on the DVX100 and other cams via anamorphic adaptors and/or software (Magic Bullet's deartifacting etc.), I would think $1000 would be an easy starting price point for a system.

I also think that with hard disc beginning to replace tape (the new 2.5" drives from Seagate would be ideal for this), that the DV codec days are numbered.

It's the early part of this change, so your timing is just right.

Plus, some people are willing to drop $8000 on a mini35 setup or rental rates approaching 16mm film cameras which I think is too much to spend for DV footage.

[Juan P. Pertierra]

I understand why one person thinks that it is closer to concept than reality, but this is not true. it is very close to being reality and this is why:

Right now I have a digital capture system which captures at more than the speed necessary to take in all the data, and I am capturing sections of frames but with very reduced precision, because of the simple problem that in order to get the full signal, I have to probe around 36 tiny pins on a surface mount chip. I only have a few surface mount probes....thus the capture system IS capturing all 36+lines, but they are hooked up to a dummy signal because i don't have the probes.

This last part i am referring to is a chip-clip that snaps on the surface mount chip allowing me to probe all 200+ pins at the same time, and thus completeing the test setup. I know that the card captures this fast, once I have the clip it's just a matter of assembling frames from the R,G,B data, which is straightforward.

The reason why I ask, is that any clip of this kind is expensive, but the smaller kind that can be put on the chip and then close the camera is even more expensive. So, this is why I ask.

I think i'm going to go all out and even if I don't end up doing anything else with it, I can leave the setup on my own DVX.

Rob: i HIGHLY doubt you need a RAID array or anything that complex to capture this data. The data rates involved are very well within the data rate supported by firewire 800. All you need is a laptop with a firewire 800 card. I've designed firewire interfaces before, and i'm positive this is not a hurdle to overcome.

Juan

[Rob Lohman]

Well let us do the math (correct me if something goes wrong).
We are using 12 bits per YUV channel. We have one YUV sample
per pixel which is 3 * 12 = 36 bits or 4.5 bytes per pixel. Assuming
you are going to store the laste 4 bits in a half byte and then
on with the next pixel (thus 2 pixels form 9 bytes completely),
we multiply the 4.5 by 720 and then by 480. This results in
1,555,200 bytes per frame or 1.48 MB/frame. Now assuming we
do 24 frames of that each second the datarate is 37,324,800
bytes per second or 35.59 MB/s (megabytes!). Even if you where
going to drop to 3 bytes per YUV (8 bit) this would still be 23.73
MB/s.

Which harddisk will keep up with that *sustained* without
dropping frames? Especially through a laptop!! You can't
use any internal laptop harddisks because they definitely aren't
fast enough and I think you'd be hardpressed to find any ATA
harddisk that can keep up with that sustained rate. The firewire
protocol is fast enough. Even a 400 mbit connection must just
be able to do it (although probably not due to overhead).

Let's just assume you find a SCSI harddisk or S-ATA or ATA
10,000 RPM harddisk (just out) that can keep up with this, what
about power consumption for this all? Personally I'd much rather
have the harddisk mounted directly to a camera without the
laptop.

Last time I checked there were chips and boards available with
complete ATA and firewire interfaces available, so this wouldn't
probably be that difficult to do. I'd bet you would get a way better
transfer rate out of that harddisk without going thourh a "real"
computer and the extra overhead that gives.

Now please do correct me if you think I mixed up somewhere or
am missing some points. I'm not trying to bring the project down
or anything, but I just have my "thoughts" about the actual
realworld implementation at the end. That's all.

[John Gaspain]

cool thread Juan!

I wanna do this to my DV953, it would be great if you made a PDF tutorial with some pictures of the chip and probes. I would love to help out in any way I can because I was thinking about this exact type of project- because the miniDV format should be replaced by a Hard drive simply because the CCD's are out performing the tapes capabilities!

Also my home PC could handle the data rate, as it is the baddest ass mofo thing that ever existed!

[Juan P. Pertierra]

Rob:

Your math looks fine, the problem comes when you start talking about hard drives, because you don't just talk about RPM and not transfer rates.

I have two laptops. They both have 5400RPM UltraATA-100 drives which can do a max of 100MB/s. This is well in excess of the 36MB/s you estimated yourself.

Furthermore, my desktop also has a 7400RPM ATA-100 drive, and it is where the capture card is mounted...it has NO PROBLEM capturing the data sustained and directly to disk.

Where am i going wrong? The data's being recorded.

[Rob Lohman]

A harddisk that does a max of 100 MB/s. Can you please post
a graph from a testtool which confirms this? Because I don't think
I've ever seen or read about an ATA drive that actually has a
throughput of 100 MB/s. Keep in mind that writing is even slower
than reading! ATA100 or 133 might have a theoretical limit of
100 and 133 MB/s, but the only way I see you getting that if
something is the 8 MB buffer of the drive and then you will only
be getting that at a fraction of the speed. Drive RPM usually
has a lot to do with maximum transfer speeds, which is logical,
since it can read the data faster of the discs.

See one of the latest harddisks on test at Tom's Hardware on
this page. Look especially at the graph there. That's
one of the fastests harddisk around that is doing 65 MB/s at it's
peak and falls of to 35 MB/s at the end (READ!). That's not a lot
of room for error. It gets even worse (slightly) when looking at
the write speeds. This isn't a 2.5" 5400 rpm drive either. I just
don't believe you can pull it off on a laptop harddisk, you might
on a normal fast ATA drive, but not on those small buggers.

Also take a look at the second page on that report. It lists
interface speeds as well as a bunch of common drives with their
read throughput (which again is faster than writing). I'm betting
that all of those are 7200 RPM drives.

You say you are capturing it on your harddisk now. How much
MB/s are you really writing to disk? For how long?

[Juan P. Pertierra]

I am using an Adlink PCI7300A digital IO card. I can do 32 digital channels with a maximum 40MB sustained throughput. it can mantain that transfer rate until my hard disk runs out of space.

The card is interfaced directly to the PCI bus. On a firewire800 interface, the data is simply serialized instead of transferred over 32 different lines.

It works, because I wrote the program to capture it, and i've captured hundreds of megabytes of test data and i've verified that there are a negligible amount of errors with no error correction protocol as long as the probes are shielded correctly, and the computer is not doing any other bus intensive processes.

Will it work at full rate on ~any~ laptop or hardrive? Of course not. But who here expects to capture uncompressed video in real time on a slow laptop/hard disk?? I think we are all aware that this is not a concept directed at the general home-movie consumer. This is for those who want to get the best possible movie-like video for their next indie movie, or commercial, without having to pay $30,000+ to get an SDX900, or any other expensive monster.

Juan

[John Gaspain]

Rob, I think we should just do it and get the ball rolling, hell by the time we actually get the bugs worked out out of the adapter Serial ATA will be up to 300mb a sec.

Im not going to let any ney-sayers get in my way, so lets work as a team and just do it!

[Stephen van Vuuren]

Rob:

Benchmarks are tricky to translate to real-world use, but here's benchmarks for the Hitachi 7200 rpm drive that's in my notebook. It's widely available as a drive, including as a second dedicated and/or RAID 0 config in a number of notebook models.

It sustains plenty of write throughput to handle this as a single drive. In a notebook RAID 0, it would handle it just fine.

These are also available as upgrade drives, so you could be a kit sans notebook in single or RAID 0 to mount on a camera.

The one in my notebook is quieter and cooler than my old notebook's 4200 rmp drive. Not bad for 2.5" drive that is energy efficient (I get 4-5 hours of battery, notebook is a Thinkpad t41p).

[Rodger Marjama]

Hello Juan,

I'm curious what HD you may be using. I've used PCI buss mastering controller technology for years in various forms, but they still require the HD to be able to sustain data throughput to hold anything close to there rated transfer rates. I could do this using SCSI or RAID but aside from that, I really don't know of any IDE type drives that would hold the nearly 40 MB per sec sustained data transfer rate necessary for this project.

This is not meant to be a poke at you BTW. A few years back I owned a small computer service company, but have since moved (literally moved also) into other areas, and have not looked back so to speak, so I really would like to know. I'll will be building a new PC shortly for video work and could make use of this info.

Thanks.

-Rodger

[Rodger Marjama]

I see Stephen just posted while I was typing. I'll take a look at this Hitachi drive link. Still curious though. What drive would you suggest as the highest throughput currently available.

Thanks.

-Rodger

[Stephen van Vuuren]

Rob:

Most any new 7200 RPM drive will sustain around 50 MB/s write:

http://www.xbitlabs.com/articles/sto...roundup_6.html

Older PCI setups and OS issues causes problems before, but new drivers on ATA5/6 and SATA setups (generally 865 and 875 Intel chipsets) are hamstrung by PCI bottleneck issues.

[Stephen van Vuuren]

Rob:

The new 75 GB WD Raptor SATA is the king of the hill for desktop drives.

[Rob Lohman]

" I think we should just do it and get the ball rolling "

Definitely! Lets go and do this. I'm just trying to explain what
my biggest fear for such a project is, that's all.

I'm impressed by those laptop harddisks, seems like technology
is moving along. Thanks for the pointers on that.

Keep in mind though that those tests are not showing how
the curve is from beginning to end of the drive (which can greatly
vary with data throughput).

I'm still for a portable solution without a laptop, though. But lets
first see some uncompressed RAW frames at full 4:4:4 12 bits to
see if it is worth it all...