Rob:

Your math looks fine, the problem comes when you start talking about hard drives, because you don't just talk about RPM and not transfer rates.

I have two laptops. They both have 5400RPM UltraATA-100 drives which can do a max of 100MB/s. This is well in excess of the 36MB/s you estimated yourself.

Furthermore, my desktop also has a 7400RPM ATA-100 drive, and it is where the capture card is mounted...it has NO PROBLEM capturing the data sustained and directly to disk.

Where am i going wrong? The data's being recorded.

A harddisk that does a max of 100 MB/s. Can you please post
a graph from a testtool which confirms this? Because I don't think
I've ever seen or read about an ATA drive that actually has a
throughput of 100 MB/s. Keep in mind that writing is even slower
than reading! ATA100 or 133 might have a theoretical limit of
100 and 133 MB/s, but the only way I see you getting that if
something is the 8 MB buffer of the drive and then you will only
be getting that at a fraction of the speed. Drive RPM usually
has a lot to do with maximum transfer speeds, which is logical,
since it can read the data faster of the discs.

See one of the latest harddisks on test at Tom's Hardware on
this page (http://www.tomshardware.com/storage/20040209/seagate-03.html). Look especially at the graph there. That's
one of the fastests harddisk around that is doing 65 MB/s at it's
peak and falls of to 35 MB/s at the end (READ!). That's not a lot
of room for error. It gets even worse (slightly) when looking at
the write speeds. This isn't a 2.5" 5400 rpm drive either. I just
don't believe you can pull it off on a laptop harddisk, you might
on a normal fast ATA drive, but not on those small buggers.

Also take a look at the second page on that report. It lists
interface speeds as well as a bunch of common drives with their
read throughput (which again is faster than writing). I'm betting
that all of those are 7200 RPM drives.

You say you are capturing it on your harddisk now. How much
MB/s are you really writing to disk? For how long?

I am using an Adlink PCI7300A digital IO card. I can do 32 digital channels with a maximum 40MB sustained throughput. it can mantain that transfer rate until my hard disk runs out of space.

The card is interfaced directly to the PCI bus. On a firewire800 interface, the data is simply serialized instead of transferred over 32 different lines.

It works, because I wrote the program to capture it, and i've captured hundreds of megabytes of test data and i've verified that there are a negligible amount of errors with no error correction protocol as long as the probes are shielded correctly, and the computer is not doing any other bus intensive processes.

Will it work at full rate on ~any~ laptop or hardrive? Of course not. But who here expects to capture uncompressed video in real time on a slow laptop/hard disk?? I think we are all aware that this is not a concept directed at the general home-movie consumer. This is for those who want to get the best possible movie-like video for their next indie movie, or commercial, without having to pay $30,000+ to get an SDX900, or any other expensive monster.

Juan

Rob, I think we should just do it and get the ball rolling, hell by the time we actually get the bugs worked out out of the adapter Serial ATA will be up to 300mb a sec.

Im not going to let any ney-sayers get in my way, so lets work as a team and just do it!

Rob:

Benchmarks are tricky to translate to real-world use, but here's (http://www.xbitlabs.com/articles/storage/display/hitachi-25-7200_13.html) benchmarks for the Hitachi 7200 rpm drive that's in my notebook. It's widely available as a drive, including as a second dedicated and/or RAID 0 config in a number of notebook models.

It sustains plenty of write throughput to handle this as a single drive. In a notebook RAID 0, it would handle it just fine.

These are also available as upgrade drives, so you could be a kit sans notebook in single or RAID 0 to mount on a camera.

The one in my notebook is quieter and cooler than my old notebook's 4200 rmp drive. Not bad for 2.5" drive that is energy efficient (I get 4-5 hours of battery, notebook is a Thinkpad t41p).

Hello Juan,

I'm curious what HD you may be using. I've used PCI buss mastering controller technology for years in various forms, but they still require the HD to be able to sustain data throughput to hold anything close to there rated transfer rates. I could do this using SCSI or RAID but aside from that, I really don't know of any IDE type drives that would hold the nearly 40 MB per sec sustained data transfer rate necessary for this project.

This is not meant to be a poke at you BTW. A few years back I owned a small computer service company, but have since moved (literally moved also) into other areas, and have not looked back so to speak, so I really would like to know. I'll will be building a new PC shortly for video work and could make use of this info.

Thanks.

-Rodger

I see Stephen just posted while I was typing. I'll take a look at this Hitachi drive link. Still curious though. What drive would you suggest as the highest throughput currently available.

Thanks.

-Rodger

Rob:

Most any new 7200 RPM drive will sustain around 50 MB/s write:

http://www.xbitlabs.com/articles/storage/display/7200-250-roundup_6.html

Older PCI setups and OS issues causes problems before, but new drivers on ATA5/6 and SATA setups (generally 865 and 875 Intel chipsets) are hamstrung by PCI bottleneck issues.

Rob:

The new 75 GB WD Raptor SATA is the king of the hill for desktop drives.

" I think we should just do it and get the ball rolling "

Definitely! Lets go and do this. I'm just trying to explain what
my biggest fear for such a project is, that's all.

I'm impressed by those laptop harddisks, seems like technology
is moving along. Thanks for the pointers on that.

Keep in mind though that those tests are not showing how
the curve is from beginning to end of the drive (which can greatly
vary with data throughput).

I'm still for a portable solution without a laptop, though. But lets
first see some uncompressed RAW frames at full 4:4:4 12 bits to
see if it is worth it all...

I hope you go all the way with this and are successful.

Would I pay $1000.00 for it if I knew it worked? Almost certainly. I bet there are a lot of dvx owners out there who would.

guys I think that maybe some of you have not been on-top of what the latest standard disk drives can do..I put a cheap uncompressed capture card in my 2.6ghz Intell P4 system with ONE systemdrive that is a SATA and I can capture alldaylong iwth NO dropped frames...has this been the case for years? NO ...is it the case now? yes...things change and get faster all the time!

Hey anything I can do to help with the system call me in! this is the best thing since sliced bread to happen for a DV camcorder...KEEP AT IT TILL YOU GET IT DOWN! I want my dvx modified!

BTW I offer my services for any CNC work you may or may not need for this system!

I will be making radio control focus systems for the dvx shortly with CNC so let me know if you need anything! maybe a nice highquality design for a "box" that mounts on the camera?

juan...OMG I have a SICK idea! get your hands on a Panasonic Black Maba mx7000 it has 3 megapixels and is a 3ccd camera

http://www.supervideo.com/panason.htm

check the above link

I suppose everyone see a little clearer then then did a couple of days ago on the probability of this actually being possible - Myself included. As I have been involved with computers and this technology since 1970, I'm a bit of an 'ol dog and admittedly not a very watchful one for the past 4 or 5 years now.

As I said in my first post, I would love to see this happen and this is still true. I do still feel its a very long shot, but keep plugging away at it and once we can see actual clips I'm sure many will be willing to support and effort to complete the development necessary to manufacture this product.

If clips can be made, I could offer free hosting and enough bandwidth to move about 5 gig per day and possibly make more available as it becomes necessary.

-Rodger

<<<-- Originally posted by Obin Olson : juan...OMG I have a SICK idea! get your hands on a Panasonic Black Maba mx7000 it has 3 megapixels and is a 3ccd camera

http://www.supervideo.com/panason.htm

check the above link -->>>

I intend to do this with my MX5000(DV953), its a 3Megapixel, 3CCD, nearly the same as a Mx7000(GS100) internally speaking

added: I will also get higher Resolution, 934x768 is the actual CCD resolution, because the cam down converts to 720x480 to fit on DV tape.

Im getting excited!

Just a note:

There is no such thing as an MX7000 - that's just the Japanese GS100 cam. Visit our Panasonic DV forum here for more details.

All posts and info on www.supervideo.com, IMHO, should be taken with the largest grain of salt available...

Plus, his pages easily rank in the HTML hall of shame and seem to get worse with time rather than better....

Agreed Stephen

john are you really going to do this?

I am not completely certain, but I beleive the raw output from the DVX's CCD's is 966x492.

At this point, it all comes down to the chip clip. if it is small enough(and i think it is) to fit on the chip with the board installed, then I can order it and have it working soon.

A word about '1.3MP' video cameras and the like...
There are two possible reasons why this idea would not work and yield HD video in such cameras...one, it is common for the manufacturers to install a large CCD to be able to read a slow frame and take a picture, but only scan a lower res at a faster rate for video.

Second, the cameras in which the entire MP CCD IS read for video(and then resized to NTSC) such as the sony TRV70, are usually 1 CCD cameras, which means that they cram 10 or 12 bit color mosaiqued into a single CCD.

Now you can do the math...say 1.3MP, 30fps, and 12-bit's...that's a lot of data. In all the cameras i researched for this experiment, the ones setup like this worked at such a high bitrate that capture to disk would be VERY hard.

This is why I picked the DVX or any other standard frame size 3CCD camera...the relatively low rez and the fact that the data is spread in parallel between 3CCD's lowers the overall clock rate to something that is workable with a hard disk.

Will keep you guys informed..i'm waiting for a response from the test clip people.

Juan

can't wait to hear what happens! keep it up, don't stop now :)

what makes you think that the image is 966x492???

what your saying is that this thing is somthing like this:
http://www.dv3productions.com/test_images/dvx_raw_image_size.jpg

do you know if it is a clear sharp image all the way edge to edge? this will be GREAT for using steady software to smooth out shaky video and also for downconverting to SD size!

Question: what editing/capturing apparatus would one need to edit 12-bit uncompressed RAW video such as this?

Also, will a successful application of this technology on a DVX100 require that sound be recorded separately and synchronized in post?

-Craig Weinstein

Additional question: will the 24P Advanced and 30P shooting modes be preserved or will 60i be the only mode available?

<<<-- Originally posted by Obin Olson : john are you really going to do this? -->>>

yea, Obin I plan on it

I just dont really know what to do yet, aside from what Jaun has presented.

If I had more information as to which chip to jumper I would be well on my way, otherwise its a needle in a haystack.

Also what would be the prefered interface for video capture?


John

john your going to have to write code to do this? i think firewire 800 for a HD sized image would work ok?

I beleive it is 966x492, because i found the catalog for Panasonic CCD's and it's the only progressive scan CCD that could possibly be the ones in the DVX. Even though I can see the CCD/prism assembly, i can't read the model number off the CCD's because of the way CCD's are installed. Matsushita doesn't have any PS CCD's that small.

It's clearly wider, but there are usually several dummy pixels in each line, and on top of that there might be several other things. For example, the prism might be designed in such a way that the CCD needs to be wider than normal, or perhaps to compensate for the NTSC pixel aspect ratio if the CCD's have square pixels.

As far as editing it, i know for a fact that final cut pro does 10-bit 4:2:2 uncompressed...however to handle the 4:4:4 uncompressed 12-bit, i'm not sure...i think FCP has an option to import individual frames and compile them into a movie, i know Apple Shake does this. You can import in say, tiff or BMP format.

The sound can be taken directly from the DV tape. Since both recordings will be at the same frame rate, all you have to do is capture the audio from the DV tape(which records at the same time) and align it to the video. It's relatively easy to add sound once i have all of this working, but that's the way it can be done initially.

Juan

I plan to use a TI Firewire 800 chip along with all the required FIFO's, etc to interface the RGB signals to the transceiver. 800MBps is more than enough, even for the raw signal from the A/D's.

Thanks for all of the time put into this Juan. I look forward to your results.

Best,
Clayton

Oh and it should work in all modes, the interface should handle the higher bit rate of 30p/60i, although the driver will have to be programmed to handle these two additional cases....

I am looking forward to this "mod" so so much!

Hello boys,


I really dislike repeating myself, but


as there is already a working DVC50 (6,5 MB/S, 4:2:2) firewire protocol directly supported by FCP4,

and several harddisk recorder units, like for instance the Quickstream DV, are available on the market for some time now,

plus, thanks to your ingenuity, Juan, we know the Y/C stage signal is still uncompressed,


it should be worth a thought whether it might be possible to alter the way the information is processed before it is send over firewire, hence obtaining DVC50 quality with the DVX for a start.


Therefore, Juan, again it would be very helpful, if you could post the names of the various chips within the DVX´s circuit and perhaps a schematic, so that these can be compared with 900´s system.


Best thanks again for all your devotion to this bold project. My suggestions are not meant to question your primary approach, but to open up some sideways for those prefering a lighter and less demanding modification, perhaps for the price of a comparatively lower efficiency.

Lucia,

The data coming out of my mod can be toned down and piped through watever standard you wish, however there are only a couple of places where you can get the raw data out, and this mod covers both. Anything dealing with using DVC50 over firewire, or any other standard is simply formatting of the data that can be done in the external circuit once I have the data coming out.

The main chip on the DVX is an Altera APEX 20K which is an LPGA, and it's function depends competely on what panasonic programmed into it which we do not know, and for this application I am not interested in as long as I know the data pins.

In other words, this is not some standard chip that you can look up the specs of and know exactly what it's doing...there ARE specs for this chip but everything it does, is based on the software, which of course is heavily protected. But we don't need this to do the procedure.

Juan

http://www.altera.com/products/devices/apex/apx-index.html

I contacted the company that manufactures the chip-clip, and it seems that they have one that is perfect for this application.

Their design will allow for my idea of having the camera completely assembled and with only a small ribbon cable coming out the back or bottom. I think it even can be routed out without any modification to the camera.

it will take them around 10 days to assemble one, so it's a waiting game for now.

Juan

Juan, thanks to John´s link I am a bit wiser now. Apparently I underestimated the complexity of the issue, so the more your efforts are to be welcome. Did I understood you right that your construction does imply some kind of ASIC to format the data? For any compression technique will need the proper codecs to be recognized from the editing suite and the raw or Y/C output probably does not match any standards according to your results. Or is your favorized method of single frame sequencing only a matter of proper timing alone, for example sending one output sample every 40 or 41,7 ms?

Hello all. Just catching up on the posts as I haven't been around in a while.

Juan - Fantastic! I experimented on such a concept in 1996 with another 3 chip from Panasonic (the AG-3). The result on the computer was much better than the camera's taped recording ability. At that time I was recording 4:2:2 (but compressed at 5:1) - which for the day was respectable.

Please keep up the excellent work!

BTW, what are the images straight off the A/D converters going to look like? Will there be problems with noise, banding, etc.? You're probably already going to do this, but could you post some pictures/movies when you finally get this thing working? Also will this mod work with the DVX100A?

John...do you still have the 4:2:2 version working?

Jason,

At this point I am not sure, but from what I've seen in outputs straight from CCD's(such as the Thomson Viper) the raw output needs color adjustments and frame resizing...however if I do use the raw output I plan to have all of these corrections done in software either at capture time or right after.

An interesting note...if we use the raw A/D output, it doesn't matter which camera you have, the DVX100 or DVX100A, because afaik they both have the same CCD's and 12-bit A/D converters. If we endup taking the signal after the Altera chip which does all the color corrections, then the DVX100 will have 10bit color, but the DVX100A will yield 12-bit color.

Juan

In reply to Lucia's last post....

My implementation will only include the necessary hardware to interface the digital video stream to firewire 800(and a driver to capture it). There is no need to design additional hardware to do any compression, because that just complicates the project, and besides, any re-formatting or compression can be easily done in software once the raw data is captured on the computer.

Juan

Hi Obin. No, unfortunately that's long been dismantled. I still have the parts (somewhere - lol). I've gotten more into rigging cost effective alternatives to various studio equipment since then. ie - batteries, dollies, grip carts etc.

I trust Juan is on the right trail on this one. High quality film transfers is what I'm looking for and this sounds like the way to go.

Juan, export to firewire 800 is a great boon to this being a viable device modification. You state that the reconversion of exported data to a usable media will be done in some sort of software recompiler, what if you are shooting and need to see playback on set? Any idea on when your testing will be completed? As I would like to try this mod on a project in the early summer.

Thanx for your amazing research. A proud DVX owner.

I believe that Juan had said that there's a posibility the camera would function normal with the mod in place. Could playback occur from the normal DV tape? Or perhaps a secondary recorder fed from the video out of the camera?

That's right John...the camera would work just as normal, and you can record to tape normally while this is all going on. You can also have a monitor coming out of the camera or use the on-board LCD/viewfinder as normal.

If you want to preview the high-quality output, that's going to be difficult with my initial device, because it's going to be in a very fat raw format which would need a custom monitor for it. However, once i get this working it is possible to implement standard outputs in hardware like some have suggested.

Will the RAW files from the DVX100 have any more dynamic range than the normal DVX100 in use? Just wondering since I've played around with RAW Thompson Viper files, and those have a tremendous amount of dynamic range, roughly equal to color negative film stocks like Fuji F400 or Vision stocks-and those are both coming off 12-bit A/D converters.

BTW, to view files, I would just assume that you would simply hook up a laptop to the camera and view the frame grabs that get recorded on the hard-drive. Once you have a couple "shots" that give you an idea of what the scene will look like, then you can easily use the DV tape to preview the movement, etc., and then have the uncompressed files later for onlining purposes.

That's a good question, but i was under the impression that the extra dynamic range of the thomson was because of the CCD's characteristics, i might be wrong.

In the meantime while the chip clip gets here, i've been doing some more experiments. Since i don't have enough surface mount probes, my experiments today consisted of probing the BLUE A/D converter, and at that only had enough probes of 7 bits(of 12).

The images seem to be 771pixels wide...i am definitely getting more pixels than that, but they seemed to be masked off. It's kinda hard to tell from the blue CCD only, unless i have a special test scene which I didn't. All i did was shoot a spray can that was in the lab, which happened to be blue/white. :)

I am going to try this week and setup a blue grid and do some more tests....

Juan

good tips Jason,

Juan is it just a simple piggyback ont the chip? What is it tapping into?

Would you please post a pic of you probing the chip? I just want to where to put the test leds, I can host it for you if you need

Right now i am just using 'grabber' probes. The final application will be a piggy-back clip, so there is technically no modification made to the DVX itself, since there is no soldering done.

I'll take some pictures of my test setup whenever i work on it again...

Juan

I want a DIRECT output that NEEDS color correcting in POST..I don't want the camera to do ANYTHING to the picture...this will assure me of the BEST chroma key you can get and that is about all I would use this "mod" for anyway..unless you make a portable storage unit for it ;)