Also, thanks to everyone who posted info on the mosaic approach, it seems that this will indeed work, and I am modifying my code right now to do processing based on this. I think I can get a GREAT ~720P image with this system, while still mantaining the same bandwidth.....very cool. I say ~720P because with the anamorphic adapter the image will be more like 1.83 aspect ratio.

Juan

Juan,

Sorry be so eager to see something. I'm sure the more work you put into it, the better it will be.

Waiting patiently. Sorta.

Hey, that's ok...i just want to make clear that i'm not just sitting here idly. There's stuff going on which is critical to doing this correctly and legally. :)

Hi all. I haven't posted in a while. I forget - what's the file format that Juan's mod records in? And what software can edit it? For us film out guys, Photoshop isn't a viable nonlinear option. Thanks in advance =)

Right now it outputs TIFF's, but other formats can be included if needed. For example I have code working for cineon/dpx files as well.

TIFF's can be loaded into FCP as frames if your system is fast enough, or you can create an EDL using DV footage which was recorded while recording in RAW.

Juan

Combustion 3 from Discreet could be a good editor to use if you are into visual effects work or just high end color control. Combustion 3 has a basic editor built in while supporting uncompressed files in a 16 bit color space. This will allow you to do some really high quality color correction and compositing effects while also editing your project. There are two codecs out there that I know of right now that will allow 16 bit quicktime files to be created. You could convert your captured tiff sequences along with audio tracks to one of these codecs. You could then have full FX, editing, and audio inside of Combustion 3. You can also just load the Tiff sequences into Combustion 3 if you would like. I am not sure if Combustion 3 will load EDL files. Does anybody know if this works? If it does load EDL files or another type of editing project file then you could edit your dv version of the video in any program and export a edl file. The edl file could then load your actual raw tiff video into Combustion 3 once the editing is done for color timing/correction and effects work.

Can we get a video file sample so we can test which editors are best suited for editing, correction, speed, ability to accuruately monitor the image, etc?

Thanks.

Yes that would be a good idea. I use After Effects with the Color Finess plugin (well its now built in - vs 6.5) and would hate to pay for combustion too.

based off what i know about juan's mod, it seems like there is a very simple and relatively user-friendly solution to the editing/finishing paradigm...

edit offline (in 23.98 fps) using the 24pa, 4:1:1, 25Mbps footage that is laid to tape. this of course will be hassle-free, as long as you use a slate or something for synching later on. then once you have picture lock, you take note of the used takes/shots. locate those tiff sequences and load them into 16-bit software, such as aftereffects or combustion. batch render them out to 23.98 fps movies, using a 16-bit lossless codec.

import those new 16-bit movies into your same FCP/avid project. don't worry that the image size is bigger than 720x480. then manually synch them up to their DV shot equivalent. this labor of synching is going to be necessary if you intend to use the camera audio, anyway (or you could greatly simplify and quicken the whole synching process by making sure to capture your DV tape clips with the frame of the slate making visual contact as your first frame, then when you import the tiff sequences into AE/combustion, just trim the clips so that this is the first frame of the rendered 16-bit clips, then open your FCP/avid project and simply reconnect/replace the DV clips in the timeline to the new 16-bit movies).

then from FCP/avid, you would create an EDL (using automatic duck if you're using FCP) to port the complete edit into aftereffects or combustion. open it up and your final edit is neatly laid out for you, with each individual trimmed shot as its own layer, ready for color grading.

trying to edit the actual 16-bit footage is going to either require one helluva expensive workstation for realtime playback, or significantly hinder your ability to edit creatively and quickly. editing offline-to-online is a lot less headachey than it used to be.

hope this helps,
jaan

Juan,

Since most of the comments on how to edit the footage (and they make sense) is to edit the dv footage and then sync up the uncompressed files, would it not make sense to have a timecode-in hookup on the mod for SMPTE timecode? For film out use, it's pretty normal to record tracks of audio on another device that's receiving the same time code as the camera (sourced from a timecode transmitter) and then match the timecodes in post when you're onlining the production. Would this add too much to the cost of the mod?

Well, i still have a but of confusion as to what the difficulty in synching the video to the audio is...i've done it several times and it works pretty much as it does with film, but easier, because the audio is linked to the DV video.

I've shot both DV and RAW at the same time, the only thing needed is some reference to synch up two frames. Remember that you get 24fps in both RAW and DV, so once they are both aligned, you are done. The DV video has the audio also, so as long as you remember to move the entire DV footage(with audio) in your software, you'll be fine.

I've been using something pretty dumb, a stopwatch. I just start it up, and move it across the screen. This allows for a large margin of error and it makes it extremely easy to find two corresponding frames.

Another thing I should note, is that I have actually edited RAW uncompressed video using just the TIFF frames in FCP on my 1.33Ghz powerbook, and it works fine. The sequence was only over a minute long, but it worked just fine. I wouldn't be surprised if on a fast system such as a dual G5, you could get away with not even using DV footage at all.

For color correction you can use anything...in fact, i'm no photoshop genius but I bet that once you figure out the corrections for one section of frames, you could use some batch/script in photoshop to process the rest of the files.

On the pixel shift:
~~~~~~~~~~~
I'm still trying to figure out how to take advantage of this. I have confirmed that the green CCD is indeed shifted slightly along the Z axis such that it captures detail which lies in between pixels on the R and B channels.

The problem is that since only the green channel is shifted, the result is not a complete mosaic, but rather a pattern in which R and B coincide, there is solo G information for other pixels, and then there are locations where there is no actual information.

Now, i'm sure there's a way of de-mosaicing this somehow, i just haven't figured it out...shifting the green CCD seems to be common practice, anyone have any ideas?

Juan

regarding green pixel shift... juan i have some ideas that might be a possible solution to the green shift, though it'll be a lot easier to spend five minutes trying it, instead of fifteen minutes trying to type out specific instructions. it also uses some software and plugins that you may not already own. can you post a recent frame?

Well, in fact that Green CCD pixel shift has a clear explanation:
That is because every color column of a 4:2:2 image is placed between two columns of green.
So instead or doing (R1+R2)/2 and a (B1+B2)/2 you just shift the green CCD half pixel and you are ready.
So take note that it won't be exactly the same situation as a Bayer mosaic.
Anyway that doesn't avoid we can use it to increase resolution a little....


If you want, send me a couple of images (TIF or RAW format if more than 8 bit 16 bit per pixel) and a clear description of relative spatial position of pixels and/or the mosaic pattern it gives and I'll see what I can do....I guess I have an idea and a working demosaic that could just needs to be tweaked for that special case...

morsa a t morsa d o t net d o t ar

The whole thing(including my DVX ;) ) is torn apart right now so I am going to re-run some of the raw files I have and upload a PSD file without any resizing.

BTW, i'm also looking for a second DVX100 so we can install the device on it and lend it for demo. If anyone has an old standard DVX100 lying around and want to sell it, let me know. It doesn't matter how banged up it is. :)

Juan

<<<-- Originally posted by Juan P. Pertierra : Well, i still have a but of confusion as to what the difficulty in synching the video to the audio is...i've done it several times and it works pretty much as it does with film, but easier, because the audio is linked to the DV video.>>>

Would it be possible to grab the DV timecode from the Firewire connector and embed it into the current filename? Don't know if the DVX outputs timecode when in cameramode but some cameras do. Should be a walk in the park to de-embed it from the DV stream right :)

It's not hard to do, IF you have DV connected over firewire. however, what i've been doing is recording RAW to the computer and DV at the same time to regular tape. In this case, the timecode is on the tape itself, so the computer cannot really synch the RAW to the DV timecode. Maybe the computer could handle the DV stream on firewire and the RAW stream on USB2.0, but i think that's pushing it.

Since the DV tape is transfered later, the computer has no reference to synch the RAW with the DV footage, unless the use steps in and does it.

Another option would be to have a piece of software somehow compare the first RAW frame to the DV footage, and figure out which two frames are equal. Statistically this can be done but it might take a while, doing it in an efficient manner might be trickier. Synching the RAW footage to the DV footage is not hard at all and takes about 10 seconds at the most, i don't think it's a big deal.

Juan

Ok, that's right you would have to process the entire DV stream in order to obtain the timecode information.

Too bad the DVX does not have a TC Lcd as the PD100 had, could be easy to intercept the driver bus.

BTW. How do you start/stop the recording? Have you thought about using the tally light as a start stop for the computer RAW capture?

For those of you asking for an un-resized raw image for the shift de-mosaiqing: I ran an old file through the software again...i think this was shot with the anamorphic adapter at 10-bit RGB.

It is a PSD file with three layers, R,G,B and there is no color correction whatsoever, the only thing I did was flip the blue channel horizontally and nudge the layers so they are more or less aligned.

Please note like I said before, that the Red and Blue layers seem to line up perfectly while the green channel is one pixel wider and taller, and seems to capture detail in between R/B pixel elements.

http://expert.cc.purdue.edu/~pertierr/shift_work.psd

Juan,

Quick question, off topic a bit. Is it possible to adjust the minimal gain on the DVX's chips? Some say that the gain is to high on the DVX's even at 0db, would it be possible to change that 0db to like a -3db or -6db? Would the camera then be able to acheive less noisy images even though an increase in light would be necessary during shooting? Thanks.

Juan will your system ALSO have some of the great features found in traditional HDR like:

10 sec prerecorded RAM video for antishock and so you never miss a shot?

Firewire option for situation when you want to use it like a traditional HDR and just hook it up to any DV camera thru firewire and record the standard compressed DV stream?

If so your product will serve double duty and open yourself up to a much wider market.

I did a little test in Photoshop for the fun of it and I realized that the XL2 chip can produce a High Definition signal. Pixel shift supposedly increases resolution by 150%. well I took a 853x480(NTSC widescreen) frame grab from the XL2 and increased its size by 150 % and i got a perfect 1280x720!!! With a Pal Xl2, it would be 1536x864, thats only 25% less resolution than 1080p!!!! Of course if you really want to get crazy and put an anamorphic adapter on it for Cinemascope, you get 2016x864. Besides the fact that there is a Pal and NTSC version of the chip, this leads me to believe the XL2 was designed to be HD.

Mark, I don't know what process feature your using to get 150%, but to take advantage of the pixel shift you will need to get the raw frames (seperate with each pixel) for red green and blue and play with those mathematically to see what you can get. If Juan has posted colour seperated frames people you could test it for the 100 first.

I had a thread a while ago about getting HD on a XL2 by pixel shift. The XL2 has a software developers kit. If it allows you to program raw frames out (if it was really good it would) then you could do that.

I think it is all very good idea, as these cameras are going to be cheap come the time the true HD one come doiwn in price, and if you can do it the quality (4:2:0) shouldn't be too far off the HD ones (pluss the firewire hard drive to record it).

I wasn't taking advantage of pixel shift, I just heard that pixel shift boosts resolution by 150%. Since the raw images from the Xl2 will at least be the size of a regular DV image , I just bicubicly uprezzed the footage to see what size NTSC Widescreen footage should be with 150% more resolution.

The problem with dv tape I think is that even with the XL2 having chips of that size isn't the rez lowered to 720 x 480 to fit on tape? Then once in the computer the anamorphic 720 frame is pulled back up to 853? Or does the XL2 record a different type of dv frame?

That means even though the XL2 shoots a really nice sharp 16x9 the pixel count is still a 720 x 480 anamorphic frame once captured.

The best way to get HD from the XL2 would be to use an anamorphic adapter at a 1.5 to 1 scale and second to figure out a way to activate the full 720 V pixels of the chips.

This would then give you 1440 x 720 non-square raw pixels. You could then turn those into square pixels to get 1280 x 720.

The interesting thing about this is that even though the chips are only 960 pixels wide and you have to blow up to get 1280 the new SONY HDV camera does the same thing. The SONY HDV camera uses chips that are 960 x 1080. That means the chips are blown up by 2X to get 1920 pixels. Well they are first blown up to 1440 but that is anamorphic. The SONY HDV starts with high H pixel count but blown up V pixels. The XL2 could be made to do basically the same thing but only to get 1280 x 720.

Of course the odds of getting a 1.5 anamorphic lens and figuring out if those "non-fired" pixels can be used are very slim so I wouldn't hold my breath for it to happen.

To Brett:

My device does not have an internal hard disk, it's an interface to a PC or a external hard disk, so you couldn't interface it to DV like you say. However, the device does have user-set options which allow the recording of a stream which is approximately the same size as a DV stream...it is NOT the DV stream created by the camera though.

Juan

To Thomas:

If your method did work than the pixel shift with give you a full 1920x1080. The pal version would get you 2048x1152. It wouldn't be as good as the F950 but I won't complain.

Remember that I have actually posted a 3 layer photoshop file in which the green layer is pixel shifted. You can try your methods with that file and see if it really works.

http://expert.cc.purdue.edu/~pertierr/shift_work.psd

Good to hear from you, Juan. I already thought that the Men in Black from Canon or Sony had kidnapped you.

I thought this might be helpful to us for deciding which codec to use for our raw video.

This guy compared a lot of the top video codecs including 4:1:1, 4:2:2, and 4:4:4.

He compared and tested both the Bitjazz Sheervideo codec and the microcosm codec. While Microcosm gives an insane amount of lossless compression it is slow. Sheervideo doesn't save as much space but is faster than realtime so may even work to encode directly from camera.

This site is also a good place to check out other types of codecs and see how bad DV really is compared to other formats.

The only bad thing is that only Mac codecs are tested and none for the PC yet.

http://www.onerivermedia.com/codecs/index.htm

Sheervideo works fine for the PC, I am a beta tester. Microcosm also works on the PC as well

Unfortunately, Sheervideo and Microcosm are not free, so I do not think this option is for everybody.

Juan, it may be a good idea to contact the Cineform guys (www.cineform.com) who make Aspect and Connect HD. They've developed an extremely efficient and fast codec that encodes 10-bit avi files with HD resolution. It encodes and playback faster than realtime without any fancy hardware. You may be able to work out some sort of bundle deal, including the codec with the hardware modification.

This would be great if we plan on getting HD out of our dvx100 but what about SD?

Even though Sheervideo and Microcosm have a cost I did not think it was to high for people who actually want this type of footage. The only real purpose to this adapter is for high end film or visual effects work. Clearly it wouldn't be used to shoot cousin Billy's Bar Mitzvah.

At the same time people do not have to use these codecs. They could use Tiffs or any other image format they want to for free. Buying one of these codecs is to help deal with the footage better and have better storage. They are just to help and are an option.

People could always use other codecs as well such as animation with slight image loss but that is the tradeoff.

i think the whole point of juan's mod is to give us the highest quality output possible from the camera -- which means RAW frames. compression to a codec would just make juan's job more complicated, not mention more expensive.

once we get the raw frames on our hard drives, what codec we use to work with the footage is our problem, and we have scores of options to choose from based on our workstations and workflows. and in terms of user design, juan's mod being only raw output is actually the best approach, because if he were to settle on a particular codec, then i'm sure he'd be alienating some users who prefer a different codec. besides, there is just no getting around the fact that, currently, any good non-lossy codec requires either lots of computation power and/or significant render time.

seriously, we couldn't ask for anything better than a camera that outputs raw pristine frames while SIMULTAMEOUSLY outputting to tape a standardized, low bandwidth format (dv25) for hassle-free offline editing, all in one relatively small package with just one simple firewire tether to a hard drive. there are plenty of other options out there if you want a low or moderately compressed format (dv50, dv100, sonyhd, etc.) that your computer can handle in realtime, and the rental costs for those packages aren't that bad anymore.

Over at the digital cinema camera threads (see list at start of technical thread) there has been people eager about doing FPGA based compression solutions, in times past (there was even a short lived FPGA thread). I believe some people, are working on them or do intend too. One company has offered to incorporate such a FPGA design in their cameras for us, if we design it. So as there is some mutal overlap here (single chip bayer is only our present option), you might get some mutal help by combining efforts with them. The Russian company Elphel (??, see home made camera thread) has supplied ccompression FPGA to sourceforge for their own network cameras too (but I think it was a high loss codec, though I don't know exactly what their future plans are). So such a thing could be mounted to a PCI card.

Apart from that there is ways to program onboard DSP's and AGP GPU circuits (only 266MB/s back channel to main/CPU memory. This hasn't been researched yet, but maybe a cheaper way.

We have been looking at low end CPU for aquistion and the data rates for aquistion of bayer 720p and DVX100 4:4:4 is similiar, except for the extra debayering step in the preview. So you should be able to get a good idea of just how low in CPU power you can go from these projects (programming is not yet finished so cpu power requirments will still be dropping). But we hope to eventually have 12cm aquare nano-itxs PC's aquiring the image through PCI capture cards, firewire or Gigabit ethernet.

Cineform is probably the best (costs a nice penny too). There is (among others) Avid's DNxHD codec that goes upto 220Mb/s, but it is limited to 10 bit. Anyway such datarates are very good for video work and some film work (we go upto raw over at the Cinenema threads for highest cinema quality, and certain editing/special effects processing) but on many productions that won't be needed.

Note on interfaces, if you go the route of an external SDI capture board (sorry I haven't read all the thread yet to know your plans) then I hear Heroinewarrior has free Linux 12bit Capture and NLE app, Cinelerra, for SDI input.

Thanks

Wayne.

Juan, it may be a good idea to contact the Cineform guys (www.cineform.com) who make Aspect and Connect HD. They've developed an extremely efficient and fast codec that encodes 10-bit avi files with HD resolution.

That sounds like a great idea.

Now I think David Newman is a great guy and very helpful, and his codec is very nice, but why do we have to alienate all the Mac users on this forum (and possible Linux users too)? Especially when I'm sure there's a lot of us who use FCP for editing, especially with the DVX100. I've talked to David many times about Mac support for Cineform, and it's not coming for quite some time due to dependancies for certain libraries, etc. on Windows and it's programming environment.

So again, I have to agree with Jaan, keep it RAW!

u done with the website yet, juan?

I'm sure he would tell us if he was.
Juan, could you give us a time frame though, I'm sure we are all very eager.

stephen birdsong

Anybody figure out how to do the pixel shift, yet?

GREAT THREAD...JUST FOUND IT. and have a lot to catch up. Anyone knows what stage is the project in?? Is Juan already getting video out etc.??

Sorry for getting here so late. I just saw the uncompressed psd he posted and it looks great. Awesome job Juan.

Michal

Hmmm
It's been over 3 weeks since Juan's last post.


Juan, if you have any sort of update, let us know when you get a chance. Thanks, and keep up the good work.

-Luis

Is this project really over 9 months old? It seems like a really awesome idea. I just wish we had some more details.

looks like "they" finally got him.

R.I.P.
Juan

stephen

where are you, Juan?!!!!!!!

I'm here i'm here :)

So, the website is still pretty plain but we have added a Forum for beta testers and questions in general. I'm finishing up the FAQ, and it the website will be posted this week, finished or not.

I figured this thread has gotten large enough as is, so a message board might be a good idea.

We're pretty much done with all the formalities that we had to take care of, the only thing we are missing is that we still need a DVX! So if any of you are looking to sell your DVX(not A is fine), let me know. I can't seem to find anything on ebay besides the retailers. Banged up is fine, as long as the electronics work.

We figured the best way to start was to have a couple of Beta testers to try the device out in the real world and let us know your suggestions or if anything doesn't work quite right. There is no software on the device per-se, so it doesn't crash, etc, but proffessionals might want some features implemented differently.

Juan

Can't wait to see the website Juan!! I'm really excited :D

Juan, did you figure out the pixel shift thing yet?

Juan what is the site address?

also I would be a very happy beta tester..I am a professional

I'm a new kid on the block around here but, Juan (and fellow posters), what an awesome thing you've got brewing here!

I'm a visual effects artist and independent filmmaker in Burbank, California. Juan (et al), I'd love to try this modification of the DVX100 with respect to shooting greenscreen actors and elements.

Also, I'm shooting a short film using this camera right now (we'll actually be shooting some scenes in Indianapolis) and would like to see if this technology might be able to be implemented in a feature I'm currently writing and hoping to go into production summer 2005.

I've seen a spectacular 35mm blow-up from a few tests done with this camera by a Los Angeles-based post-house using their own "special sauce" proprietary scaling algorithm. I think they might be interested in seeing some tests with this modification. I would be happy to look into this possibility.

(www.imdb.com)

An initial version of the website has been posted with basic information. We will be adding the rest of the content and correcting any errors over the next week or so. Please feel free to email with any questions or mistakes you find.

We figured it would be best to get input from everyone as we add the rest of the content, such as the FAQ's. I know there are tons of questions i've heard over and over again, so be sure to point out anything you want covered.

There is also a forum, which is intended to (hopefully) take some of the weight off this thread which has been going on for quite some time now, and is hard to navigate.

http://www.reel-stream.com

Cheers,
Juan

Hi Juan,

The site looks good. The pricing and whatever wait time is involved is going to make or break this project, so hopefully you will figure that out soon and let everyone know.

Good luck!

Joel

Looking great Juan. As soon as I get my DVX I'll be looking into your service.