No bother at all! Thanks for reminding me, i've been so caught up working on the prototype that i totally forgot. The setup is a little torn apart right now but I can probably manage to do it tomorrow, it doesn't take that much to hook it all up.

Juan

What would everyone suggest is the cheapest way to test SDI video output?

Rental is not an option because I probably need to have it for a relatively long time. Are there any cheap SDI monitors out there? Or is my best choice an SDI computer card, and if so which?

Oh and another dumb question...why exactly wouldn't one use a computer CRT monitor instead of an expensive production video monitor? AFAIK the computer monitors have at least a thousand lines of resolution, and ~some~ control over the image. I have an old small TV as an excuse for a video monitor for editing, and i'm considering using a large Sony computer CRT which is lying around at work.

My little TV is maxed waaaay out as far as resolution goes and the color is terrible...so a better monitor would really help in working with the raw footage off the DVX.

I'm not exactly sure why SDI is a requirement for this. What is the point of it when you can capture onto a firewire 800 disk and monitor with a regular S-video monitor? This seems far more economical than introducing SDI into the mix. Who wants to go out and buy an SDI monitor or deck to capture/monitor this?

My only question about FW800 is how you would design the IO so that the drive captures the data. AFAIK, there is no FW800 version of something like the FW400 quickstream or the Laird capdiv.

This is an exciting project. Let me know if you need any help with marketing it online in terms of content or design. I work for Amazon.com, so I have some experience setting up good customer experiences!

Thanks.

I would imagine that the FW800 drive has no file system as we know them, and is treated as a raw drive. Maybe a basic indexing data structure to mark where shots start and end. Simple.
Personally, I too would dump the SDI stuff. I hate video stuff :)

-Les

A 'FireStore' type drive is NOT needed. ANY Firewire800 drive WILL work. The drive will be formatted in the same manner that you use with your PC/Mac, and the files will be readable by a computer likewise. There is no special formatting, because the Box acts as a host, so it's just like connecting a drive to a computer....except the computer is now the raw capture box mounted on the DVX.

I am doing the SDI because it really isn't that much harder, and i'm sure someone can use it.

Juan

Ya,

Final Cut Pro HD is allowing output to computer monitors so you don't have to use expensive HD video monitors. I'm surprised no one marketed that idea before.

Go Juan! Go Juan! Ya, baby, YAAAAAAAAAAAA!!!

Juan,
So what file system will be on that firewire drive? NTFS ?
You aren't worried about having to implement the file system directory structure and it's associated latency while pushing the data to the drive?
Perhaps I didn't understand what you said you are doing.
-Les

Dear Laurence
There is no way in Final Cut HD to monitor input while in full screen.The capture mode desactivate the digital cinema desktop preview.

Thanks for clearing that up Emmanuel. Maybe what they meant was just during editing? Not capture? Or heck, maybe I just got my info screwed. Never worked with FCP HD, just saw it on the Apple site.

Juan,

Both AJA and Blackmagic make SD and HD SDI pci cards for the MAC. Blackmagic's works also with PCs. I think the price of an SD card in about $300-400 depending on whether you need genlock capability.

Another option for about the same money Marshall makes a decoder box (monitor quality).

You could also build your own decoder box with a National semiconductor all in one chip decoder. You can implement a pattern generator (color bars) in your fpga if you like.

Dual capable SD -HD sdi cards go for about $2000. AJA's will do both upconvert and downconvert if I am not mistaken.

This project has grown quite enormous, as has the thread, but I'll try and offer something of value here, since I'm feeling left out. :P

Juan, to try to answer one of your questions, I believe the reason it's not ideal to use a computer monitor is that most of them (maybe all of them, I'm not sure) don't have SMPTE certified phosphors. What it takes for a phosphor to meet SMPTE specifications I don't know, but the issue at hand, as I understand it, is image reproduction accuracy. A computer monitor, or average television, WILL display the image, just not the way you'd want it to. Sorta like how a cheap pair of speakers might technically be able to recreate really high or low frequency sounds, but they'll end up being a few decibels louder or softer than they should be (i.e. when they were recorded).

Given the proper funds, it's preferable to get a production monitor, but I think the computer display will be physically able to draw the image. And with enough control, I imagine one could get such a display to be relatively accurate.

I think. I could be wrong. I'm wrong a lot.

WHOAAAA RANDALL!!!

Kona & Black Magic make $2000 HD-SD capture cards? I thought these things were 7K from Kona (I didn't know about Black Magic). So far the best bang I found for the buck was Igniter X. But if you know the models of these specific Kona, Black Magic, or others, please let me know, becasue I'm putting together a Mac system to use with FCP HD.

Thanks!!!

Les:

The box doesn't really care what format the drive is in, the protocol used to talk to a Firewire800 storage device in this manner doesn't do all the low-level operations associated with the file system. The drive will need to be usable i.e. formated in any manner, but all Firewire800 drives come pre-formatted so that shouldn't be a problem.

Juan

Ooh! Something just occured to me (pardon me if it's been covered already, but I didn't see mention of the subject): will this device offer anything in the way of stop motion capability? Might be useful for some people, being able to record one frame at a time to the disk...can't do that with a tape.

Any plans?

Laurence,

Because of moores law and competition today's $8,000 board is replaced by tommorows $2000 board (well $2500 list).

Here is the info on AJA's KONA 2:

Quote from: http://aja.com/K-2%20FINAL-PR.htm


Other exciting features found in Kona 2 include:

o RS-422 machine control – connect to any professional tape deck

o Kona desktop output for broadcast design and paint

o Automatic HD/SD genlock

o Input AES sample rate conversion – no audio source sync required

o Break out cables standard (including both BNC and XLR AES cables)

o Optional “K Box” 1RU breakout box with additional features

o Kona 2 QuickTime drivers written and supported by AJA

o AJA award winning telephone support



Kona 2 and K Box Availability and Prices
Kona 2 is priced at US $2,490.00. K Box is priced at US $299.00. Both Kona 2 and K Box will be available in June 2004 and carry a three-year unconditional warranty.

Unquote*****************

The SD board is of course a lot cheaper but can't upconvert SD to HD. This is a monster board at the price. A break out box is available.

Blackmagic's SD board is very competitive to AJAs SD offering. On paper I prefer AJAs KONA 2 HD board over Blackmagic's HD capture board. I haven't read any lab tests yet or tried either.

If you had one of these boards you would want SDI output on the DVX100.

Another outfit is selling gigabit interfaces for analog outs, LVDS outs, and CameraLink outs. Don't know the price yet. I am guessing its around $400:

http://pleora.com/resources/document_library/index.cfm

Another outfit Silicon Imaging will be selling single chip HD cameras using the Altasens 2/3" chip (announcement to come in the next 8 weeks). Based on that company's current Y media sensor based prices $3500 to $4500 these cameras could come in at less than $6,000 with a raw 12-bit output!

Juan's mod for the DVX100 will probably outperform these more expensive cameras because don't forget the DVX100 is a three chip camera (even though SD).

The cheapest HD camera with 3 chips will be the JVC 870 at $20,000 list!

Can't wait to find out what Juan will charge for the basic mod!

What is the 'box' you refer too? Does the 'box' just ask the FW800 drive for a file handle to write to?

Are you saying that this firewire drive you want to write to has a built in file system , and you send high level commands to it to accept a big block of data ? i.e. Fopen Fwrite etc, and not raw sectors ?
Then firewire800 is *way* more than a speed up of an old firewire drive !!!!!!

You say that all Firewire800 drives come pre-formatted?
Preformatted with what file system?
How does the PC or Mac OS see the drive?
Please explain or tell me where to learn about this new development of drives that don't rely on the operating system to manage the directory structure! I'm intrigued !

-Les


<<<-- Originally posted by Juan P. Pertierra : Les:

The box doesn't really care what format the drive is in, the protocol used to talk to a Firewire800 storage device in this manner doesn't do all the low-level operations associated with the file system. The drive will need to be usable i.e. formated in any manner, but all Firewire800 drives come pre-formatted so that shouldn't be a problem.

Juan -->>>

Les:

Try www.t10.org all the info is there.

Juan, I looked at the site, and unfortunately I've seen no reference to file management within 1394b drives. 1394b drives are just faster 1394 drives. They are 'dumb' just like SCSI drives.

Lets see, how can I put this:

I think you are making an interface (box) that writes to a disk drive with attached 1394b interface.
I think you are doing this without a host PC or Mac, it's just the 'box' getting image data from the camera, and it is saving the image stream to the 1394b disk drive.
You state "The 'box' does not care what format the drive is in"

Sounds like writing raw sectors ( no standard file system ) to the drive to me. With a simple dir structure to keep track of the shots, like I said in my first post on this topic.
But.... then you say " The drive will need to be usable i.e. formated in any manner, but all Firewire800 drives come pre-formatted so that shouldn't be a problem"

So which is it? Are you proposing to implement the Mac/PC file system allocation and management in your 'Box' .... or are you doing a custom file system?
That will need PC/Mac software to read it on a personal computer. You can't just mount it and see the data with any old computer.
Perhaps you need to think about this some more?
Do you understand what a file system *is* and how they work?
-Les



<<<-- Originally posted by Juan P. Pertierra : Les:

Try www.t10.org all the info is there. -->>>

the real thing to beat all would be if you could somehow overcrank that puppy with your mod...can you make it shoot 60fps? I know the chips can do it because of the 60i option in the camera...can you force the chips to run at 60fps? and capture that? this is one of the most wanted things for digital ....I know I would pay more for that option ..Juan?

les...how would Juan know how to build the box if he did not know what a file system is????

I'm sure Juan is looking very closely at what it will take in HD, protocol, support hardware, and will find answers. I know QuickStream is have more then it's share of difficulties with their streaming tapeless capture system and it only captures compressed DV.

I have always thought this will be the "killer" problem to overcome on this idea. As always, I wish Juan luck and hope his ideas are better then my instincts in this regard.

-Rodger

Juan, I wonder if you would be able to implement an aspect ratio (letterbox) function (like for 16:9 or 2.35:1 or custom aspect ratios) that would actually crop the images before writing them to disk to reduce data rate and storage requirements.

So, for example, let's say instead of capturing all 480 vertical pixels for each frame, you could only capture the center 270 or 300, etc if you wanted to, and have this reflected in the monitoring somehow. Of course, if you were to capture only the center 270 pixels, then your tiff frames would be a much smaller 720x270. At uncompressed data rates, that would add up to a lot of savings over time.

Would that be too hard to implement? I think it would be useful.

His posts didn't make any sense whatsoever with regards to the file system issue.
He said he was going to just write to the drive, not regarding the file system on said drive. Nonsense.
What do you make of it?
Juan, please clear this up !
-Les



<<<-- Originally posted by Obin Olson : les...how would Juan know how to build the box if he did not know what a file system is???? -->>>

I don't mean to speak up for Juan, but the tone of your posts are leaning towards aggressive/argumentative. Juan has been an absolute professional in responding to posts promptly and informatively. You probably don't mean to sound this way intentionally, nevertheless, it comes off as somewhat negative.


es

I just want to nip any possible problems in the bud before they are ' show stoppers' as we say in the biz.

-Les

Juan,

If you want the disks to be readable on both MACS and PCs perhaps you could consider SGI's xfs file system. Special FX houses are using this on their sans so that both MACs, PCs, and SGIs can access the same files.

As I understand it Preformat on disk drives is usually just the hard sectoring and low level format. The high level format as you know is somewhat O/S dependent. The NTFS Les mentioned is of course specific to windows NT and I believe XP. MAC OSX probably uses some variant of BSD (Berkeley Softwared Distribution) formatting. "Wine" from MAC 0S and Unix systems should be able to read NTFS and FAT PC files.

XFS has two advantages.

One) each OS can be setup to read XFS files.

Two) Xfs is a journalizing file system so it is easier to recover from
system crashes and other problems with the file system.

If the data rate goes up in the next generation (HD or uprezzed
SD) you might have to go to a raid 0 configuration.

I agree with some other posts that we should be patient with Juan on the question of what File system he is going to use

The program he wrote to use with his test capture device had no trouble writing Tiff files to the MAC file system. So I am not worried he can figure out how to write files to a disk from firewire or gigabit or SDI in any case.

I only suggest XFS for Juan's consideration. There are any number of reasons why he might take another direction.

I agree with Les. Filesystems are complicated beasts and you
cannot just write to disks and read it on a PC / Mac without
additional software.

Personally I would go with FAT32 at first as almost all direct-to-disk
options do. This does mean implementing a FAT32 filesystem
manager. I'm not sure if there is some free source we can find
for this. Otherwise I might be able to write such a system.

I would not go with another filesystem like NTFS or the proposed
XFS (at least to start with). For the following reasons:

1) Mac/Pc might not mount it per default. Mac's cannot do NTFS
and both cannot read XFS per default and need drivers installed
which adds to the getting read to use part

2) most other file systems like NTFS and XFS are much more
difficult to implement (especially in such devices) [especially when
they are journelling file systems]

3) you will also need information on how this filesystem works
which is not available for NTFS for example. I'm not sure if such
information is available for XFS. Such information is readily
available for FAT32.

One thing I'm also worried about is number of files in a directory.
Let's say you are recording a shot that lasts 10 minutes at the
default NTSC framerate of 30 fps. When you are recording in
plain TIFF files you will have 18.000 files!! I'm not sure what the
directory limit of FAT32 is, but that number of files in a single
directory is very slow on your computer and possibly on "our"
device as well. With such a system you at least need to use a
different directory for each recording you do.

The only other option is to use a movie format. This probably
means also writing a "codec" (uncompressed) for each system.

Or is there some "codec"/file format we can use that the big boys
are using as well?

stay away from codecs!

The only codec I know right now that can support Juan's mod is the new 12-bit 4:4:4 Quicktime RGB codec from Blackmagic for the HD Pro card. Other than that you'd have to use some 16-bit codec like None-16 etc., but then Quicktime doesn't work on Linux, so . . . the best bet is to use Tiff files like Juan is already doing.

I agree that FAT32 is probably the best bet-Although watch out-there's a 127GB file-size limit on FAT32 on OSX. I believe it has something to do with partition size limits on FAT32 and how they've been getting around those limitations using virtual partions, etc.

Juan, with thanks to Rob Lohman for his informed comments,

FAT32 might be OK. Does anyone know the limit on the number of files? Of course it should be implemented so that one has access to 160Gb disks!

I have run up against a limit on both too many files and too many directories under UNIX however that was because I didn't allow enough space for the root partition.

I don't know exactly what the spec is for standalone "firewire" drives. I read that on a powerbook G4 you can press T at bootup to cause the G4 to emulate a standalone firewire drive. Details of what the MAC does should be available.

The benefit would be that people could use the powerbook G4 as a portable drive with the added benefit of being able to do a rough cut in the field.

When they got back from the shoot they could use the same trick to mount the G4 to their G5 edit system to read the edited files or the unedited files.

There is also a tool available for free from snell and wilcox {MXF} to encapsulate avi and other mulitimedia files with metadata so that the multimedia files can be handled by most filesystems. This would have to be applied after capture.

Juan, Rob, with special thanks to Jason,

Jason thanks for your helpful comments. Do you recommend the HDpro by blackmagic. Are their 12 bit in and 14 bit out more than hype?

I prefer the AJA HD capture device on paper because it has upconvert from SD to HD. I don't think the DECKLINK has this?

A 12bit quicktime would be nice though for high end film transfer work. Where can we get the none-16 codec?

So far there is no SDI format that supports more than 10-bit.

As far as a protocol for direct to disk transfer goes for HD I prefer dual link SMPTE 392 but its only 10-bit. Sony's high end 444 recorder uses this.

Quote from SMPTE site:
SMPTE 372M-2002: Television – Dual Link 292M Interface for 1920 x 1080 Picture Raster
This standard defines a means of interconnecting digital video equipment with a dual link HD SDI (link A and link B), based upon the SMPTE 292M data structure. The source formats for this dual link interconnection are the picture raster formats, and digital interface representations as defined in SMPTE 274M. The total data rate of the dual link connection is 2.970 Gb/s or 2.970/1.001 Gb/s. This dual link also supports carriage of the embedded audio, the ancillary data, and the ID of the stream.

Some have suggested fiber channel which is OK for storage but bad for networking (see):

http://www.broadcastpapers.com/asset/GigE%20vs%20FC.htm

Firewire 800 and Gigabit will handle uncompressed SD just fine. Uncompressed 10-bit HD might need multiple firewires or multiple Gigabit ethernets with possible syncronization problems.

There is a reason to go with Industry standards or to create industry standards. Maybe Juan can come up with something that becomes and industry standard.

Juan, Rob, Jason, Les, and listmembers,

Without buying the standard for HDSDI from SMPTE there may be details posted in the SMPTE journal see the following references.

1. SMPTE 292M, .Television.Bit-Serial Digital Interface for High- Definition Television Systems,. SMPTE J., 107:849, Sept. 1998.
2. SMPTE 274M, .Television.1920 x 1080 Scanning and Analog and Parallel Digital Interfaces for Multiple Picutre Rates,. SMPTE J., 107:837, Sept. 1998.

See also

http://www.broadcastpapers.com/dcinema/dcinema.htm

a 19 frame per second HSDL format has been suggested for
moving 14 bit depth images around a post production plant.

The advantages of applying a logarithmic scale to image data are highlighted. I suggest that maybe a 10-bit logartithmic scale is as good as raw 12 bit data?

Could a logarithmic scale be applied to the data at the FPGA in Juan's mod? Would there be a benefit to taking the log and sending the data 10 bit? This could save bandwidth and disk space.

Of course maybe 12 bit 444 would be nice too! No compromises.

Ok i just got back, so i'm trying to catchup! Here goes:

Joel:
That is a great idea, i totally forgot about that option. It is very easy to implement and i've added it to the list of features..i for one use letterbox mode all the time in the DVX, and a hefty amount of bandwidth and space can be saved. I will make sure to do it such that the user can pick the actual frame size.

Les:
I still stand by what I said: The firewire interface doesn't care what format the drive is, and this information is contained in the website i gave ya.

I think the confusion here is that some people want non-technical 'end user' information, and others want down to the bone technical info. I can't tell the difference of who wants what type of info just by a question.

From a user's perspective, it doesn't matter what FW800 drive you buy. Every FW800 drive in the market right now can handle the bandwidth. Just hook it up and go.

From a technical standpoint, we have to go back to what stephen was drilling into our heads a couple of weeks ago: there is no such thing as a 'firewire' drive. In essence, it has to be handled like if the firewire interface wasn't there, it is just a transport method. Think about SDI. Does it matter if what you have on the other side of the camera is a tape deck or a monitor?

The 'box' i am referring to, is the device I am building which will be mounted under the DVX100. And yes, there are specific areas of the FPGA dealing with the low-level filesystem duties, but i know little of the details because I am using megafunctions already available in the environment I am using.

For starters, I am going to keep it simple and do the exact same process I am doing on my experimental setup...it will be a single file containing the raw R,G,B frames packed together. At this point, the number of files will not be a problem, and in the case that the file system will limit this, we can always keep the raw format in one contiguous file.

During these days I am little more worried with hooking up the hardware. I already have all the physical parts for the prototype so all that will be left is FPGA configuration changes which are quick.

Blue Screen Test:
I could only find a (dark?)blue board but I do have it here so i'm going to try and do it now. One of my felines devoured a surface mount probe so i'm fixin things...

Juan

OK, Juan,
Thats what I thought you were doing ( the right thing ) ... simply writing the rgb data stream to the drive in as raw of a format as possible.
You might want to reserve the first 200 K of the drive to be an index for the rest of the drives contents.
Call it a primitive file system, if you want.
Format of the data structure:
unsigned long START_block_number_of_stream,END_block_number_of_stream;
Then have a counter for how many streams you have in the table.
Each time someone shoots some shot, it gets a new table entry on the reserved part of the drive.
When a new shot starts , your BOX looks and sees where the next free disk space there is.
Don't bother with deleting files, and fragmentation, you don't need that anyway. The drive is like 'film' or 'tape' a sequence of contiguous storage. It's fast and simple.

Encode the data with a LUT and stuff three 10 bit colors into each 32 bits, ala Cineon. 10 bits is enough to get ALL the dynamic range off of film.

Now, you will have to write a driver to mount this drive on a computer to allow the users to get to the shots.
Not too hard.
On UNIX you can just use dd to read the sectors off the drive, and you could even just write a little shell script to decode the index and get to the shots ( takes ) !

I too am looking into preparing a camera system for feature work. Kind of like an independent film shooters version of the Viper. I'm looking at including the D.I. editing and color grading software package with it as well, all based on >8 bit source images. This will be more flexible than working with film and doing a Spirit HD transfer, since that locks you into color grading choices very early in the post process.

Beware of Sony paid assassins. They won't like this type of activity. It will upset their apple cart, so to speak. I don't think it's even on their radar yet, though!

-Les

Write a driver????

Please, no drivers, that takes everything out the "simple" category. I've used enough buggy drivers in my lifetime to know that if we can just keep it simple with firewire800, etc. then everything will be fine.

Hi all,

This is my first post. I have been following this thread from the very beginning. Juan - my compliments.

There has not been much talk about PAL. Juan, if I remember properly, you said that you had an NTSC chip camera with a resolution larger than NTSC (I do not remember the exacty resolution out of the top of my mind). Another reader noticed that changing your raw chart frame to NTSC resolution and then applying NTSC pixel ratio, one would have an NTSC frame with perfect rounded circles.

I am not sure if you had already noticed it, but from the same original larger raw frame it is possible to obtain a PAL frame: changing resolution to PAL 720x576 and then applying 1.066 PAL pixel ratio, one has a PAL frame with perfect rounded circles.

I am sorry if this is rather "silly", but it seemed to me that this PAL issue was never brought out.

Gianluca

PS: the inclusion of SDI output is definetly a good move!

Drivers are nothing compared to what the circuit design and fpga programming is needed for a project like this.
Firewire 800 or plain firewire drives need drivers or other custom software to make things happen.
Firewire drives are not video tape.
-Les

I agree that we should stay away from drivers at all costs. They
are difficult to write and even more difficult to maintain (in the
future).

So this leaves us with systems that are already supported.

I think Juan was talking about writing a SINGLE FILE for each
recording to a FILE SYSTEM. But correct me if I'm wrong. From
reading his post he seems to be talking about some functions
in the FGPA that already do low-level file system access?? If so
I'm wondering what the file system is (probably FAT32, keep
in mind that there is a 2 or 4 GB file size limit in that case as
well!!!! -> i'd say 2 GB if you want to support Mac's).

From searching a bit around it looks like the maximum file size
is 4 GB - 2 bytes. I could not find a maximum number of files
per directory or for the root directory:

" The root directory on a FAT32 drive is not stored in a fixed location as it is on FAT16 and FAT12 drives. On FAT32 drives, the root directory is an ordinary cluster chain. The A_BF_BPB_RootDirStrtClus member in the BPB structure contains the number of the first cluster in the root directory. This allows the root directory to grow as needed. In addition, the BPB_RootEntries member of BPB is ignored on a FAT32 drive "

So you can probably create new cluster chains unlimited. Ofcourse
I can easily see why this can create drastic performance problems
with lots and lots of files since you will need to traverse lots of
cluster chains as well.

But if it is indeed just 1 RAW file (possibly with a header to indicate
which recording format it is in, Juan?) this shouldn't be a real
issue.

The maximum disk size it can support is 2 terabytes. Some sites
also claim this is the maximum partition size, but I'm not sure
how this would work.

Juan: have you actually tested your setup with a drive mounted
to your device and then moved that over to your PC/Mac? It
feels like you are missing some crucial parts there.

how come you haven't been posting pics, juan?


And by the way, if I'm right, this mod applied to a pal DVX would give you an image around the size of 800x600. Just so you know.

Rob:
Yes, I was talking about a single file in the file system itself. However, I do have to point out that like I said before, this is not the part I am concentrating on during these days because it's just software and reconfiguration. My immediate goal is to have a prototype setup such that I can do instant tests and then start fiddling with the rest of the FPGA configuration.

Afaik the megafunctions I have deal with FAT16/32 and NTFS. I can get my hands on a lot of designs, but because of the nature of this application, I really would like to simplify the design as much as possible...if I use the megafunctions I have right now, i can probably complete it fast, but i will run the risk of having to use an even bigger and more expensive FPGA.

Rob, can you point me in the direction of some good PDF's outlining the details of FAT32 or NTFS? I'm betting that if I can implement just exactly the circuitry that I need to only write files, i can speed up things and keep the FPGA size to a minimum.
I can sit back and use the functions someone else wrote but it would take longer for me to reverse-engineer them and simplify rather than start from scratch and write something simple.

Juan

Juan: I can understand why you are not bothering too much with
this, yet. I would probably do the same. That's why we are
reminding you about it <g>

Are you saying that FPGA (which one was that again?) you are
working with comes with FAT16/32 & NTFS libraries? Are you SURE
NTFS is included? That would be very strange since NTFS is almost
licensed to no-one by Microsoft and there is no definitive guide on
the internet on all the technical details.

If they already have those libraries I would not start looking into
writing my own code. Writing file system code is pretty dangerous
and time consuming thing. If you can spare the room and cycles
for now just leave it at that to get it working first with drives.

We can always upgrade the code later if need to be. I personally
would much rather clean up working FAT32 code then create new
one from scratch.

Keep the following things in mind:

1) for simplicity I would not start off with NTFS or any other more complex filesystem. Stick with FAT32 for first release. We/you/someone can always add support for more file systems later

2) forget about FAT16, that does not support large enough files etc.

3) keep in mind that we should split/start new files automatically before we reach the 2 GB barrier (which seems to be the most compatible barrier)

Does your chip/code have any way to either give back date/time
and/or timecode (timecode could be generated, ofcourse). That
would be handy to use in filename generation (which people can
perhaps customize as well).

I'll see if I can dig up some technical detail regarding FAT32.

Juan,

One thing to consider. Most but not all users will want to edit using final cut pro. So whatever file system is chosen, the file system should be mountable under Mac OSX10.

Don't know for sure what Mac OSX10 uses for disk format some BSD standard format? The file system underlying the GUI is Free BSD Unix.

Some users of course will want to access the filesystem chosen by Juan from Windows XP and eventually Longhorn (yikes). Some will be using Linux or SGI.

FAT32 doesn't make very efficient use of disk space and there is the 2GB limit on partition size. Nonetheless if FAT32 can be mounted on most platforms it might be a good place to start.

I still suggest looking into SGIs multiplatform xfs file system as
a long term solution or option. See sgi.com.

XFS runs only under LINUX and IRIX right now but can serve files to Windows XP using Samba. I know there is also someway for MACOSX (bsd unix) to also mount the file system (I'll look into this).

Don't know if Juan wants to implement an embedded Linux on a soft processor on the FPGA (sounds like a formdible challenge); however the advanges of XFS are highlighted in this IBM article:

http://www-106.ibm.com/developerworks/library/l-fs9.html

See also:

http://oss.sgi.com/projects/xfs/faq.html

If you had an embedded linux xfs drive, you could run samba and netatalk to interface with macs and pcs see foote cone and belding section at:

http://oss.sgi.com/projects/xfs/xfs_users.html#Moving%20Picture%20Company

If you had a fast enough satellite, wireless, or landline internet connection you could use samba tunneling to send your files directly to the editing machine harddrive!

See who is using xfs?:

http://oss.sgi.com/projects/xfs/xfs_users.html

For filesystems with many small files, ReiserFS is preferred. For
filesystems with lots of huge data files, XFS is the ticket.

See DKP Effects.

See also echostar (satellite television).

At another special fx house Moving Picture Company:

http://www.linuxuser.co.uk/articles/issue20/lu20-Linux_at_work-In_the_picture.pdf

So I think in the long run Xfs is the way to go. Gigabit ethernet or SMPTE 372 dual link HD-SDI s/b the preferred transport protocol.

Firewire800 is of course capable for SD at 270mbps and 540mbps.
The main drawback is that the disks have to be located close to he camera.

Randall: please take the time to read this little FAQ (http://www.dvinfo.net/conf/misc.php?s=&action=bbcode) so you can
create working links yourself. Thanks!

I'm not sure where you are getting your info from, but FAT32
can definitely have partitions large than 2 GB! It can go to at least
127 GB and possibly to 2 TB.

The file limit is 4 GB but most systems limit that to 2 GB. This is
not a real problem because you can easily split the capture over
multiple files if it reaches such a boundary. You will need to post
process the footage anyway so it is easy to include reading from
a batch of files.

Personally I would definitely not go with the native Unix / Mac OS
filesystems and XFS (for now) since it cannot be mounted on
Windows systems. I (and I doubt others) don't want to install
some unix box to get access to a camera.

Windows, linux & Mac OS X can read FAT32 partitions and file
systems. It is what ALL the direct-to-disk records for DV are
using as well. There is a good reason for that.

I want to take the harddisk I recorded on to a partner of mine
who has PC's and Mac's and be able to mount the drive. I don't
want to bring a unix machine with me.

What's wrong with splitting the files if they get too large? More
easy for archiving anyway (2x 2 GB or 1x 4 GB more easily fits on
a DVD for example).

You can always enable more advanced file systems later if
someone really wants to use it. For now I vote for simplicity
and getting it working. Everybody can read and work with FAT32
RIGHT NOW, out of the box.

I will find out what the exact limit is on the partition size for FAT32.

I'm doing a bit of research and the maximum partition size seems
to be a bit of an unknown. Perhaps we should run some tests
if people have large disks lying around.

According to this interesting article (http://www.winnetmag.com/Article/ArticleID/38803/38803.html) (from a trustworthy source)
Microsoft has builtin a limit in XP to format FAT32 to only 32 GB
so people switch to NTFS. You can actually format larger partitions
under Windows 98 (and possibly Windows 2000). How stupid is
that. It's easy to include a small tools for Windows XP users
though. Most drives come formatted as FAT32 as well.

Update 1: it looks like Windows 2000 has the same limitation

" You cannot format a volume larger than 32 GB in size using the FAT32 file system in Windows 2000. The Windows 2000 FastFAT driver can mount and support volumes larger than 32 GB that use the FAT32 file system (subject to the other limits), but you cannot create one using the Format tool. This behavior is by design. If you need to create a volume larger than 32 GB, use the NTFS file system instead. "

Notice the 'by design" line. We can easily work around that.

Update 2: it appears one can download the FAT32 specs from Microsoft

FAT32 File System Specification (http://www.microsoft.com/whdc/system/platform/firmware/fatgen.mspx)

Juan:

Linux NTFS drivers (http://linux-ntfs.sourceforge.net/index.html)?

Some NTFS technical info (http://linux-ntfs.sourceforge.net/ntfs/) (menu is hard to see in the middle)

About NTFS (http://www.ntfs.com/)

NTFS is something that wouldn't get my vote. Only Windows
supports it.

yes, please stay away from NTFS, that will basically alienate all your FCP users on OSX.

OSX can read an NTFS partition, as well as Linux (actually OSX is using the Linux open-source drivers), but it can't write to the volume. Also it can't mount an NTFS partition that's segmented into a couple partions.

Right now the maximum mountable FAT32 partition on OSX is 127GB, JFYI.

The other file system out there that OSX can read is UFS (unix file system), but I'm not sure how popular that is. So that's not really recommended either.

I'd say go with FAT32, but again, I'm not sure what to do with the 127GB partition size limitation under OSX.

Juan, Rob, Jason, and listmembers,

Yes FAT32 may be the easiest solution to implement. I don't know however whether FAT32 has an ideal block size for recording a digital video stream. Fat32 will also eat up the disk faster than some other file systems. Large EIDE Disks are cheap now so why should we care.

My suggestion (admittedly more difficult to implement) would be set up the drive attached to the mod box as its own standalone computer (running embedded Linux). The xfs formated drive would be accessed from Windows and MacOS as a network drive
using SAMBA or netalink (running on the modbox). This would require a gigabit ethernet rather than a firewire connection.

The downside is that the modbox would have to be present when the location recording drive was accessed by the editing machine.

However, the camera has to be present now when downloading a DV file unless you have a DV deck attached to your edit machine. So this shouldn't be seen as a big disadvantage.

If one wanted to be able to plug bare drives into both systems, one would need to build a player interface box with embedded linux on the edit machine.

File transfers (assuming one had only one remote recording disk) would be relatively fast with a gigabit ethernet. Files could be archived on DLT if necessary.

Rob,

Thanks for the link to the FAQ.

Is vb enabled on this forum? I belong to a lot of email lists that don't even allow HTML posting (everything is plain text) so I inadvertently assumed listmembers might have to copy and paste if they wanted to follow a link.

I will use the vb tags with links in the future if they are necessary on this forum.

Randell: just do {url}link{/url} where {} should be []

FAT32 blocksize will not be too much of a problem since we are
only working with large files. Block sizes are basically only a
concern if you work with (lots of) small files.

Ok, it seems like there's some decisions to make here. I think FAT32 sounds like the best option because both mac and windows can access it. Sound good? I am going to avoid at ALL costs, needed any sort of driver to work the hard drive. My goal is to be completely plug and play, ready in NLE friendly format.

A driver IS needed if you want to plug in the raw capture device on the DVX DIRECTLY to a PC for capturing to the PC's drive.

About the blue screen shots:
I just now finished hooking up the 10-bit experimental setup back up, and took some test frames to verify everything is fine.

I need pointers on the lighting, and scene selection. Unfortunately, the 'scene' is right next to a sliding door and there is outside light coming in, so i might need to wait till dark to then fire up my work light and get a somewhat uniform lighting across the blue screen.

What I plan to do is to upload photoshop files with RAW separate R,G,B frames, and knowing that the green channel is a little different size, what are everyone's guesses on the perfect alignment and resizing. If several people do this separately, we are bound to get good guesses.

I will shoot a resolution chart which will also help with the alignment.

Post any suggestions now! :)

Juan