4:4:4 12-bit Uncompressed DVX100

[Les Dit]

Drivers are nothing compared to what the circuit design and fpga programming is needed for a project like this.
Firewire 800 or plain firewire drives need drivers or other custom software to make things happen.
Firewire drives are not video tape.
-Les

[Rob Lohman]

I agree that we should stay away from drivers at all costs. They
are difficult to write and even more difficult to maintain (in the
future).

So this leaves us with systems that are already supported.

I think Juan was talking about writing a SINGLE FILE for each
recording to a FILE SYSTEM. But correct me if I'm wrong. From
reading his post he seems to be talking about some functions
in the FGPA that already do low-level file system access?? If so
I'm wondering what the file system is (probably FAT32, keep
in mind that there is a 2 or 4 GB file size limit in that case as
well!!!! -> i'd say 2 GB if you want to support Mac's).

From searching a bit around it looks like the maximum file size
is 4 GB - 2 bytes. I could not find a maximum number of files
per directory or for the root directory:

" The root directory on a FAT32 drive is not stored in a fixed location as it is on FAT16 and FAT12 drives. On FAT32 drives, the root directory is an ordinary cluster chain. The A_BF_BPB_RootDirStrtClus member in the BPB structure contains the number of the first cluster in the root directory. This allows the root directory to grow as needed. In addition, the BPB_RootEntries member of BPB is ignored on a FAT32 drive "

So you can probably create new cluster chains unlimited. Ofcourse
I can easily see why this can create drastic performance problems
with lots and lots of files since you will need to traverse lots of
cluster chains as well.

But if it is indeed just 1 RAW file (possibly with a header to indicate
which recording format it is in, Juan?) this shouldn't be a real
issue.

The maximum disk size it can support is 2 terabytes. Some sites
also claim this is the maximum partition size, but I'm not sure
how this would work.

Juan: have you actually tested your setup with a drive mounted
to your device and then moved that over to your PC/Mac? It
feels like you are missing some crucial parts there.

[Mark Grgurev]

how come you haven't been posting pics, juan?


And by the way, if I'm right, this mod applied to a pal DVX would give you an image around the size of 800x600. Just so you know.

[Juan P. Pertierra]

Rob:
Yes, I was talking about a single file in the file system itself. However, I do have to point out that like I said before, this is not the part I am concentrating on during these days because it's just software and reconfiguration. My immediate goal is to have a prototype setup such that I can do instant tests and then start fiddling with the rest of the FPGA configuration.

Afaik the megafunctions I have deal with FAT16/32 and NTFS. I can get my hands on a lot of designs, but because of the nature of this application, I really would like to simplify the design as much as possible...if I use the megafunctions I have right now, i can probably complete it fast, but i will run the risk of having to use an even bigger and more expensive FPGA.

Rob, can you point me in the direction of some good PDF's outlining the details of FAT32 or NTFS? I'm betting that if I can implement just exactly the circuitry that I need to only write files, i can speed up things and keep the FPGA size to a minimum.
I can sit back and use the functions someone else wrote but it would take longer for me to reverse-engineer them and simplify rather than start from scratch and write something simple.

Juan

[Rob Lohman]

Juan: I can understand why you are not bothering too much with
this, yet. I would probably do the same. That's why we are
reminding you about it <g>

Are you saying that FPGA (which one was that again?) you are
working with comes with FAT16/32 & NTFS libraries? Are you SURE
NTFS is included? That would be very strange since NTFS is almost
licensed to no-one by Microsoft and there is no definitive guide on
the internet on all the technical details.

If they already have those libraries I would not start looking into
writing my own code. Writing file system code is pretty dangerous
and time consuming thing. If you can spare the room and cycles
for now just leave it at that to get it working first with drives.

We can always upgrade the code later if need to be. I personally
would much rather clean up working FAT32 code then create new
one from scratch.

Keep the following things in mind:

1) for simplicity I would not start off with NTFS or any other more complex filesystem. Stick with FAT32 for first release. We/you/someone can always add support for more file systems later

2) forget about FAT16, that does not support large enough files etc.

3) keep in mind that we should split/start new files automatically before we reach the 2 GB barrier (which seems to be the most compatible barrier)

Does your chip/code have any way to either give back date/time
and/or timecode (timecode could be generated, ofcourse). That
would be handy to use in filename generation (which people can
perhaps customize as well).

I'll see if I can dig up some technical detail regarding FAT32.

[Randall Larsen]

Juan,

One thing to consider. Most but not all users will want to edit using final cut pro. So whatever file system is chosen, the file system should be mountable under Mac OSX10.

Don't know for sure what Mac OSX10 uses for disk format some BSD standard format? The file system underlying the GUI is Free BSD Unix.

Some users of course will want to access the filesystem chosen by Juan from Windows XP and eventually Longhorn (yikes). Some will be using Linux or SGI.

FAT32 doesn't make very efficient use of disk space and there is the 2GB limit on partition size. Nonetheless if FAT32 can be mounted on most platforms it might be a good place to start.

I still suggest looking into SGIs multiplatform xfs file system as
a long term solution or option. See sgi.com.

[Randall Larsen]

XFS runs only under LINUX and IRIX right now but can serve files to Windows XP using Samba. I know there is also someway for MACOSX (bsd unix) to also mount the file system (I'll look into this).

Don't know if Juan wants to implement an embedded Linux on a soft processor on the FPGA (sounds like a formdible challenge); however the advanges of XFS are highlighted in this IBM article:

http://www-106.ibm.com/developerwork...ary/l-fs9.html

See also:

http://oss.sgi.com/projects/xfs/faq.html

If you had an embedded linux xfs drive, you could run samba and netatalk to interface with macs and pcs see foote cone and belding section at:

http://oss.sgi.com/projects/xfs/xfs_...ture%20Company

If you had a fast enough satellite, wireless, or landline internet connection you could use samba tunneling to send your files directly to the editing machine harddrive!

See who is using xfs?:

http://oss.sgi.com/projects/xfs/xfs_users.html

For filesystems with many small files, ReiserFS is preferred. For
filesystems with lots of huge data files, XFS is the ticket.

See DKP Effects.

See also echostar (satellite television).

At another special fx house Moving Picture Company:

http://www.linuxuser.co.uk/articles/...he_picture.pdf

So I think in the long run Xfs is the way to go. Gigabit ethernet or SMPTE 372 dual link HD-SDI s/b the preferred transport protocol.

Firewire800 is of course capable for SD at 270mbps and 540mbps.
The main drawback is that the disks have to be located close to he camera.

[Rob Lohman]

Randall: please take the time to read this little FAQ so you can
create working links yourself. Thanks!

I'm not sure where you are getting your info from, but FAT32
can definitely have partitions large than 2 GB! It can go to at least
127 GB and possibly to 2 TB.

The file limit is 4 GB but most systems limit that to 2 GB. This is
not a real problem because you can easily split the capture over
multiple files if it reaches such a boundary. You will need to post
process the footage anyway so it is easy to include reading from
a batch of files.

Personally I would definitely not go with the native Unix / Mac OS
filesystems and XFS (for now) since it cannot be mounted on
Windows systems. I (and I doubt others) don't want to install
some unix box to get access to a camera.

Windows, linux & Mac OS X can read FAT32 partitions and file
systems. It is what ALL the direct-to-disk records for DV are
using as well. There is a good reason for that.

I want to take the harddisk I recorded on to a partner of mine
who has PC's and Mac's and be able to mount the drive. I don't
want to bring a unix machine with me.

What's wrong with splitting the files if they get too large? More
easy for archiving anyway (2x 2 GB or 1x 4 GB more easily fits on
a DVD for example).

You can always enable more advanced file systems later if
someone really wants to use it. For now I vote for simplicity
and getting it working. Everybody can read and work with FAT32
RIGHT NOW, out of the box.

I will find out what the exact limit is on the partition size for FAT32.

[Rob Lohman]

I'm doing a bit of research and the maximum partition size seems
to be a bit of an unknown. Perhaps we should run some tests
if people have large disks lying around.

According to this interesting article (from a trustworthy source)
Microsoft has builtin a limit in XP to format FAT32 to only 32 GB
so people switch to NTFS. You can actually format larger partitions
under Windows 98 (and possibly Windows 2000). How stupid is
that. It's easy to include a small tools for Windows XP users
though. Most drives come formatted as FAT32 as well.

Update 1: it looks like Windows 2000 has the same limitation

" You cannot format a volume larger than 32 GB in size using the FAT32 file system in Windows 2000. The Windows 2000 FastFAT driver can mount and support volumes larger than 32 GB that use the FAT32 file system (subject to the other limits), but you cannot create one using the Format tool. This behavior is by design. If you need to create a volume larger than 32 GB, use the NTFS file system instead. "

Notice the 'by design" line. We can easily work around that.

Update 2: it appears one can download the FAT32 specs from Microsoft

FAT32 File System Specification

[Rob Lohman]

Juan:

Linux NTFS drivers?

Some NTFS technical info (menu is hard to see in the middle)

About NTFS

NTFS is something that wouldn't get my vote. Only Windows
supports it.

[Jason Rodriguez]

yes, please stay away from NTFS, that will basically alienate all your FCP users on OSX.

OSX can read an NTFS partition, as well as Linux (actually OSX is using the Linux open-source drivers), but it can't write to the volume. Also it can't mount an NTFS partition that's segmented into a couple partions.

Right now the maximum mountable FAT32 partition on OSX is 127GB, JFYI.

The other file system out there that OSX can read is UFS (unix file system), but I'm not sure how popular that is. So that's not really recommended either.

I'd say go with FAT32, but again, I'm not sure what to do with the 127GB partition size limitation under OSX.

[Randall Larsen]

Juan, Rob, Jason, and listmembers,

Yes FAT32 may be the easiest solution to implement. I don't know however whether FAT32 has an ideal block size for recording a digital video stream. Fat32 will also eat up the disk faster than some other file systems. Large EIDE Disks are cheap now so why should we care.

My suggestion (admittedly more difficult to implement) would be set up the drive attached to the mod box as its own standalone computer (running embedded Linux). The xfs formated drive would be accessed from Windows and MacOS as a network drive
using SAMBA or netalink (running on the modbox). This would require a gigabit ethernet rather than a firewire connection.

The downside is that the modbox would have to be present when the location recording drive was accessed by the editing machine.

However, the camera has to be present now when downloading a DV file unless you have a DV deck attached to your edit machine. So this shouldn't be seen as a big disadvantage.

If one wanted to be able to plug bare drives into both systems, one would need to build a player interface box with embedded linux on the edit machine.

File transfers (assuming one had only one remote recording disk) would be relatively fast with a gigabit ethernet. Files could be archived on DLT if necessary.

[Randall Larsen]

Rob,

Thanks for the link to the FAQ.

Is vb enabled on this forum? I belong to a lot of email lists that don't even allow HTML posting (everything is plain text) so I inadvertently assumed listmembers might have to copy and paste if they wanted to follow a link.

I will use the vb tags with links in the future if they are necessary on this forum.

[Rob Lohman]

Randell: just do {url}link{/url} where {} should be []

FAT32 blocksize will not be too much of a problem since we are
only working with large files. Block sizes are basically only a
concern if you work with (lots of) small files.

[Juan P. Pertierra]

Ok, it seems like there's some decisions to make here. I think FAT32 sounds like the best option because both mac and windows can access it. Sound good? I am going to avoid at ALL costs, needed any sort of driver to work the hard drive. My goal is to be completely plug and play, ready in NLE friendly format.

A driver IS needed if you want to plug in the raw capture device on the DVX DIRECTLY to a PC for capturing to the PC's drive.

About the blue screen shots:
I just now finished hooking up the 10-bit experimental setup back up, and took some test frames to verify everything is fine.

I need pointers on the lighting, and scene selection. Unfortunately, the 'scene' is right next to a sliding door and there is outside light coming in, so i might need to wait till dark to then fire up my work light and get a somewhat uniform lighting across the blue screen.

What I plan to do is to upload photoshop files with RAW separate R,G,B frames, and knowing that the green channel is a little different size, what are everyone's guesses on the perfect alignment and resizing. If several people do this separately, we are bound to get good guesses.

I will shoot a resolution chart which will also help with the alignment.

Post any suggestions now! :)

Juan