4:4:4 12-bit Uncompressed DVX100

[Rodger Marjama]

I'm sure Juan is looking very closely at what it will take in HD, protocol, support hardware, and will find answers. I know QuickStream is have more then it's share of difficulties with their streaming tapeless capture system and it only captures compressed DV.

I have always thought this will be the "killer" problem to overcome on this idea. As always, I wish Juan luck and hope his ideas are better then my instincts in this regard.

-Rodger

[Joel Corkin]

Juan, I wonder if you would be able to implement an aspect ratio (letterbox) function (like for 16:9 or 2.35:1 or custom aspect ratios) that would actually crop the images before writing them to disk to reduce data rate and storage requirements.

So, for example, let's say instead of capturing all 480 vertical pixels for each frame, you could only capture the center 270 or 300, etc if you wanted to, and have this reflected in the monitoring somehow. Of course, if you were to capture only the center 270 pixels, then your tiff frames would be a much smaller 720x270. At uncompressed data rates, that would add up to a lot of savings over time.

Would that be too hard to implement? I think it would be useful.

[Les Dit]

His posts didn't make any sense whatsoever with regards to the file system issue.
He said he was going to just write to the drive, not regarding the file system on said drive. Nonsense.
What do you make of it?
Juan, please clear this up !
-Les



<<<-- Originally posted by Obin Olson : les...how would Juan know how to build the box if he did not know what a file system is???? -->>>

[Eduardo Soto]

I don't mean to speak up for Juan, but the tone of your posts are leaning towards aggressive/argumentative. Juan has been an absolute professional in responding to posts promptly and informatively. You probably don't mean to sound this way intentionally, nevertheless, it comes off as somewhat negative.


es

[Les Dit]

I just want to nip any possible problems in the bud before they are ' show stoppers' as we say in the biz.

-Les

[Randall Larsen]

Juan,

If you want the disks to be readable on both MACS and PCs perhaps you could consider SGI's xfs file system. Special FX houses are using this on their sans so that both MACs, PCs, and SGIs can access the same files.

As I understand it Preformat on disk drives is usually just the hard sectoring and low level format. The high level format as you know is somewhat O/S dependent. The NTFS Les mentioned is of course specific to windows NT and I believe XP. MAC OSX probably uses some variant of BSD (Berkeley Softwared Distribution) formatting. "Wine" from MAC 0S and Unix systems should be able to read NTFS and FAT PC files.

XFS has two advantages.

One) each OS can be setup to read XFS files.

Two) Xfs is a journalizing file system so it is easier to recover from
system crashes and other problems with the file system.

If the data rate goes up in the next generation (HD or uprezzed
SD) you might have to go to a raid 0 configuration.

I agree with some other posts that we should be patient with Juan on the question of what File system he is going to use

The program he wrote to use with his test capture device had no trouble writing Tiff files to the MAC file system. So I am not worried he can figure out how to write files to a disk from firewire or gigabit or SDI in any case.

I only suggest XFS for Juan's consideration. There are any number of reasons why he might take another direction.

[Rob Lohman]

I agree with Les. Filesystems are complicated beasts and you
cannot just write to disks and read it on a PC / Mac without
additional software.

Personally I would go with FAT32 at first as almost all direct-to-disk
options do. This does mean implementing a FAT32 filesystem
manager. I'm not sure if there is some free source we can find
for this. Otherwise I might be able to write such a system.

I would not go with another filesystem like NTFS or the proposed
XFS (at least to start with). For the following reasons:

1) Mac/Pc might not mount it per default. Mac's cannot do NTFS
and both cannot read XFS per default and need drivers installed
which adds to the getting read to use part

2) most other file systems like NTFS and XFS are much more
difficult to implement (especially in such devices) [especially when
they are journelling file systems]

3) you will also need information on how this filesystem works
which is not available for NTFS for example. I'm not sure if such
information is available for XFS. Such information is readily
available for FAT32.

One thing I'm also worried about is number of files in a directory.
Let's say you are recording a shot that lasts 10 minutes at the
default NTSC framerate of 30 fps. When you are recording in
plain TIFF files you will have 18.000 files!! I'm not sure what the
directory limit of FAT32 is, but that number of files in a single
directory is very slow on your computer and possibly on "our"
device as well. With such a system you at least need to use a
different directory for each recording you do.

The only other option is to use a movie format. This probably
means also writing a "codec" (uncompressed) for each system.

Or is there some "codec"/file format we can use that the big boys
are using as well?

[Jason Rodriguez]

stay away from codecs!

The only codec I know right now that can support Juan's mod is the new 12-bit 4:4:4 Quicktime RGB codec from Blackmagic for the HD Pro card. Other than that you'd have to use some 16-bit codec like None-16 etc., but then Quicktime doesn't work on Linux, so . . . the best bet is to use Tiff files like Juan is already doing.

I agree that FAT32 is probably the best bet-Although watch out-there's a 127GB file-size limit on FAT32 on OSX. I believe it has something to do with partition size limits on FAT32 and how they've been getting around those limitations using virtual partions, etc.

[Randall Larsen]

Juan, with thanks to Rob Lohman for his informed comments,

FAT32 might be OK. Does anyone know the limit on the number of files? Of course it should be implemented so that one has access to 160Gb disks!

I have run up against a limit on both too many files and too many directories under UNIX however that was because I didn't allow enough space for the root partition.

I don't know exactly what the spec is for standalone "firewire" drives. I read that on a powerbook G4 you can press T at bootup to cause the G4 to emulate a standalone firewire drive. Details of what the MAC does should be available.

The benefit would be that people could use the powerbook G4 as a portable drive with the added benefit of being able to do a rough cut in the field.

When they got back from the shoot they could use the same trick to mount the G4 to their G5 edit system to read the edited files or the unedited files.

There is also a tool available for free from snell and wilcox {MXF} to encapsulate avi and other mulitimedia files with metadata so that the multimedia files can be handled by most filesystems. This would have to be applied after capture.

[Randall Larsen]

Juan, Rob, with special thanks to Jason,

Jason thanks for your helpful comments. Do you recommend the HDpro by blackmagic. Are their 12 bit in and 14 bit out more than hype?

I prefer the AJA HD capture device on paper because it has upconvert from SD to HD. I don't think the DECKLINK has this?

A 12bit quicktime would be nice though for high end film transfer work. Where can we get the none-16 codec?

So far there is no SDI format that supports more than 10-bit.

As far as a protocol for direct to disk transfer goes for HD I prefer dual link SMPTE 392 but its only 10-bit. Sony's high end 444 recorder uses this.

Quote from SMPTE site:
SMPTE 372M-2002: Television – Dual Link 292M Interface for 1920 x 1080 Picture Raster
This standard defines a means of interconnecting digital video equipment with a dual link HD SDI (link A and link B), based upon the SMPTE 292M data structure. The source formats for this dual link interconnection are the picture raster formats, and digital interface representations as defined in SMPTE 274M. The total data rate of the dual link connection is 2.970 Gb/s or 2.970/1.001 Gb/s. This dual link also supports carriage of the embedded audio, the ancillary data, and the ID of the stream.

Some have suggested fiber channel which is OK for storage but bad for networking (see):

http://www.broadcastpapers.com/asset/GigE%20vs%20FC.htm

Firewire 800 and Gigabit will handle uncompressed SD just fine. Uncompressed 10-bit HD might need multiple firewires or multiple Gigabit ethernets with possible syncronization problems.

There is a reason to go with Industry standards or to create industry standards. Maybe Juan can come up with something that becomes and industry standard.

[Randall Larsen]

Juan, Rob, Jason, Les, and listmembers,

Without buying the standard for HDSDI from SMPTE there may be details posted in the SMPTE journal see the following references.

1. SMPTE 292M, .Television.Bit-Serial Digital Interface for High- Definition Television Systems,. SMPTE J., 107:849, Sept. 1998.
2. SMPTE 274M, .Television.1920 x 1080 Scanning and Analog and Parallel Digital Interfaces for Multiple Picutre Rates,. SMPTE J., 107:837, Sept. 1998.

See also

http://www.broadcastpapers.com/dcinema/dcinema.htm

a 19 frame per second HSDL format has been suggested for
moving 14 bit depth images around a post production plant.

The advantages of applying a logarithmic scale to image data are highlighted. I suggest that maybe a 10-bit logartithmic scale is as good as raw 12 bit data?

Could a logarithmic scale be applied to the data at the FPGA in Juan's mod? Would there be a benefit to taking the log and sending the data 10 bit? This could save bandwidth and disk space.

Of course maybe 12 bit 444 would be nice too! No compromises.

[Juan P. Pertierra]

Ok i just got back, so i'm trying to catchup! Here goes:

Joel:
That is a great idea, i totally forgot about that option. It is very easy to implement and i've added it to the list of features..i for one use letterbox mode all the time in the DVX, and a hefty amount of bandwidth and space can be saved. I will make sure to do it such that the user can pick the actual frame size.

Les:
I still stand by what I said: The firewire interface doesn't care what format the drive is, and this information is contained in the website i gave ya.

I think the confusion here is that some people want non-technical 'end user' information, and others want down to the bone technical info. I can't tell the difference of who wants what type of info just by a question.

From a user's perspective, it doesn't matter what FW800 drive you buy. Every FW800 drive in the market right now can handle the bandwidth. Just hook it up and go.

From a technical standpoint, we have to go back to what stephen was drilling into our heads a couple of weeks ago: there is no such thing as a 'firewire' drive. In essence, it has to be handled like if the firewire interface wasn't there, it is just a transport method. Think about SDI. Does it matter if what you have on the other side of the camera is a tape deck or a monitor?

The 'box' i am referring to, is the device I am building which will be mounted under the DVX100. And yes, there are specific areas of the FPGA dealing with the low-level filesystem duties, but i know little of the details because I am using megafunctions already available in the environment I am using.

For starters, I am going to keep it simple and do the exact same process I am doing on my experimental setup...it will be a single file containing the raw R,G,B frames packed together. At this point, the number of files will not be a problem, and in the case that the file system will limit this, we can always keep the raw format in one contiguous file.

During these days I am little more worried with hooking up the hardware. I already have all the physical parts for the prototype so all that will be left is FPGA configuration changes which are quick.

Blue Screen Test:
I could only find a (dark?)blue board but I do have it here so i'm going to try and do it now. One of my felines devoured a surface mount probe so i'm fixin things...

Juan

[Les Dit]

OK, Juan,
Thats what I thought you were doing ( the right thing ) ... simply writing the rgb data stream to the drive in as raw of a format as possible.
You might want to reserve the first 200 K of the drive to be an index for the rest of the drives contents.
Call it a primitive file system, if you want.
Format of the data structure:
unsigned long START_block_number_of_stream,END_block_number_of_stream;
Then have a counter for how many streams you have in the table.
Each time someone shoots some shot, it gets a new table entry on the reserved part of the drive.
When a new shot starts , your BOX looks and sees where the next free disk space there is.
Don't bother with deleting files, and fragmentation, you don't need that anyway. The drive is like 'film' or 'tape' a sequence of contiguous storage. It's fast and simple.

Encode the data with a LUT and stuff three 10 bit colors into each 32 bits, ala Cineon. 10 bits is enough to get ALL the dynamic range off of film.

Now, you will have to write a driver to mount this drive on a computer to allow the users to get to the shots.
Not too hard.
On UNIX you can just use dd to read the sectors off the drive, and you could even just write a little shell script to decode the index and get to the shots ( takes ) !

I too am looking into preparing a camera system for feature work. Kind of like an independent film shooters version of the Viper. I'm looking at including the D.I. editing and color grading software package with it as well, all based on >8 bit source images. This will be more flexible than working with film and doing a Spirit HD transfer, since that locks you into color grading choices very early in the post process.

Beware of Sony paid assassins. They won't like this type of activity. It will upset their apple cart, so to speak. I don't think it's even on their radar yet, though!

-Les

[Jason Rodriguez]

Write a driver????

Please, no drivers, that takes everything out the "simple" category. I've used enough buggy drivers in my lifetime to know that if we can just keep it simple with firewire800, etc. then everything will be fine.

[Gianluca Jandelli]

Hi all,

This is my first post. I have been following this thread from the very beginning. Juan - my compliments.

There has not been much talk about PAL. Juan, if I remember properly, you said that you had an NTSC chip camera with a resolution larger than NTSC (I do not remember the exacty resolution out of the top of my mind). Another reader noticed that changing your raw chart frame to NTSC resolution and then applying NTSC pixel ratio, one would have an NTSC frame with perfect rounded circles.

I am not sure if you had already noticed it, but from the same original larger raw frame it is possible to obtain a PAL frame: changing resolution to PAL 720x576 and then applying 1.066 PAL pixel ratio, one has a PAL frame with perfect rounded circles.

I am sorry if this is rather "silly", but it seemed to me that this PAL issue was never brought out.

Gianluca

PS: the inclusion of SDI output is definetly a good move!