4:4:4 10bit single CMOS HD project

[Wayne Morellini]

Obin

I've had a look at your pictures, great, I love the colour, beuatifal. I'm impressed and would to see the results from a 1080 or SHD version oneday. I noticed all the artifacts that people were talking about, and that is why I think that Bayer will take away from the resolution and detail quality for resolution upscaling for the big screen (not to mention green screening as the bayer does not give a correct per pixel colour edge).

My latest Opera Web browser or Radeon 9000 driver, appears to be performing some sort of upscaling/deblocking, I can't see a square pixel even at 1000% magnifcation, and it looks good (except colour averaging of the bayer).

How about using a test chart to work out the real picture specs (latitude, low light and sensitivity), or side by side comparisons with the DVX 100 on the same images? Alltogther it leaves SD single chips for dead, and even the colour of 3 chip SD (I hate oversatuation), this is probably the sort of colour we will eventually see on single chip HD cameras in the future. Definetly good enough for cheap indie film, it looks almost natural (the latitude and falling away to black and blowout being the biggest variation). I think your way ahead of 28 days latter.

Obin, about NLE editing computer, we are close to the technology of being able to fit the NLE in the box with medium to high comrpression, maybe not this year (unless a low powered multiple processor server board is availble) but maybe next, so you can NLE from the camerra connected to a usb hub (for mouse, keyboard, external drives) and monitor. Your system is also the lightest camera in processing requirements, so you probably can do that right now. Your camera's USB2 connection, a nano-itx board (fastest) and a single laptop drive might be able to handle a pure packed bayer image. If you can get a faster cpu you might even be able to compress. People get caught up in the camera link thing (and I wonder why SI don't pack their data in the head, it is a simple circuit/proceadure) but a PCI cameralink board should have more than enough bandwidth for 1080. I wonder if it has something to do with the unreliability of USB2.0, or the fact that the data is read so quick (unbuffered) that it maxes out USB2.0? Steve?

[Wayne Morellini]

<<<-- Originally posted by Steve Nordhauser : Although it is possible to adjust the vertical blanking time somewhat, what you proposed hasn't been attempted yet. It might take some firmware changes in the camera. What Obin was proposing is based on the rolling shutter artifact. Each line is sequentially read from top to bottom. That means that there is a temporal difference on any image of one frame time from the top line to the bottom. The faster you go, the less the difference. Obin wants to readout at twice the frame rate he needs and toss every other frame. This will get him the desired rate with no timing adustments but reduce the single frame time in half.

I will have to think about what Rob suggested though - extending vertical blanking for a full frame time.....hmmmm. -->>>

The problem with these scheemes is that you will need as much shutter time (every 24th of a second) as possible to accumulate light, and also to blend movement from one frame to another (to stop the jerky motion). I think I'am prepared to live with upto one frame difference between top and bottom (as motion looks disorienting anyway), and at higher frame rates the jerkness dissapears and the motion difference will be much smaller anyway. I think it's possible that the horizontal oreintation of film shutters might be to get rid of the rolling shutter artifact, as movement tends to be horizontal in one direction. Whereas on the normal Video we are stuck with up and down movemnt, which is great for an overhanging camera on a assembly line.

Steve, for rolling shutter, am I understanding it right? Can you read out much faster than the 24th second exposure time, this should reduce the difference between top and bottom to much less than a frame?

One question I would like to ask about these cameras, is it possible to adjust the shutter, and gain to maximise the latitude of these cameras? Most of the time we have plenty of light and want to maximise the latitude (and minimise the blow outs), and most components have an optimum setting that expands their responsiveness. We then could use manual aperature to adjust the amount of light, therefore adjusting the range to teh desired level. So will this work Steve?

Steve to justify yourself to your boss, SI and Sumix are at the forefront of supplying this new market. But if you want the sales it has to be simple enough (aswell as good and cheap enough) to offer true pro performance at prosumer HDV prices. Ideally that would have meant either complete software, capture (with simple effective external camera/capture controlls/interface), and lense adaptor, pc computer, and display options worked out (which then can be sold to buyers), plus bundeling with one or more NLE (like Prospect HD) would be good. On the more advanced level (using the freeware prgrammable gate array work of that Russian fella (or a modified server like multiple processor mini-ITX with cameralink interface) a complete backend box system with raid storage, that can be sold to owners of any manufacturers industrial box camera. But neither SI or Sumix has expressed any desire to do offer complete simple systems, and that scares a lot of potential buyers, because it hasn't been worked out. So my guess you might be able to sell 10-100 a month as it is now until we work it out (then maybe 100+), but many more people would by a complete worked out package, and that is what people are trying to do on these threads, to effectively use SI and Sumix cameras. But what you are doing is helping to establish a new market for your boss. I myself, look forward to seeing your new HD cameras, I hope there will be 10+ stop box cameras, operating at below 2 lux, rendering true 1080 effective resolution and even three chip 1080 for $1-$3K. As you can see from the discussions that the requirements here are different from machine vision, for beautifal pictures, range, sensitivity, resolution and accuracy.

<<<-- Originally posted by Steve Nordhauser : David,

The word I hear in this group is that they don't want to compromise on what I've been calling pre-post processing - the Bayer filter, RGB->YUV, compression steps. If this can really be done in real-time, all the way up the camera scale, you have an impressive product (or will when the cheap camera link cameras are supported) and I would suggest that it is not worth anyone here writing their own code. $1K to record fully processed and compressed data with a preview window, maybe with basic camera controls would be great for the people here. -->>>

But what about the $500 cameralink bundle packages with the Prospect NLE lite software packages (actually I would prefer to pay $200, but I don't think so ;)?

<<<-- Originally posted by Steve Nordhauser :
I think the biggest processing and recording issues are a complete understanding of any step that is a potential loss of quality - like the Bayer filter. -->>>

Yes, hate bayer filter, love bayer filter (it is much of a love/hate relationship for me so far, but offer me a 8mp bayer camera and I'll forgtet all about it ;)!

[Wayne Morellini]

<<<-- Originally posted by David Newman :
through the trouble to design and build your our camera you don't want to compromise on the quality of it output (i.e. you want 10bit or better.) CineForm Prospect HD is priced the way it is as is designed for multi-stream real-time 1920x1080 compressed and uncompressed workflows over HD-SDI. i.e. expensive hardare and the software to manage it. Do you guys need that?
-->>>

David, we don't need that but some do. So you could offer two light versions of Prospect HD, one with HDSDI support high end support, and one for DV, HDV, HDV2, box cameras, and low end professional camera formats. I'm aiming on using pc interfaces, disks and component i/o to do my transfers.

<<<-- Originally posted by David Newman : >-- Originally posted by Obin Olson :

It is designed for HD online. It is equilevent (or better) than D5 (which is YUV 4:2:2 10bit.) D5 is the workhorse standard for HD masters.

HD> David why YUV 4:2:2??? whynot RGB 4:4:4 with 10/12/16bit?
..

YUV is a more natural compression format -- optimized for the human visional system. RGB compression is less efficient. 150Mb/s compressed RGB would look worse than 150Mb/s YUV 4:2:2 (this is even true for 4:4:4 YUV although to a lesser extent.) Basically is you want the benefits of compression, YUV is the way to go.

> the whole point that this camera can shoot images that you have more control in post is 4:4:4...maybe 4:2:2 would be ok if it was high bit depth...?? -->>>


Yes that's the name I've been trying to remember, D5, I think there is also a 4:4:4 storage format as well?

My take on how good 4:2:2 is:
Industry have claimed that reduced colour bandwidth signals are a good compromise to 4:4:4 for years but I've been able to see the obviouse difference, and my eyesight is not that great. The problem is that eyesight maxes out at 2400dpi in B&W and and 1200dpi in colour, and HD is nowhere near that (approx 75dpi and 150dpi, equivalent to 150dpi and 250+dpi because it is a emissive display rather than a reflective paper display). Even taking into account the reduced resolution of the eye as you go out from the center, the maxium colour resolution of 720p is so low you would probably have to be legally blind to be completely unable to pick the difference between 4:2:2 and 4:4:4. But from my own observations and estiimates the situation is helped because in a semi trance thought (or watching movies) people tend to slightly defocuss their vision and the mind starts integrating nearby pixels at above 150dpi (720p). So I take 720p 4:4:4 to be the mininium suitable resolution for theater (except for Imax, or if your sitting near the front)) that also offers suitable detail, colour and res quality for resolution upscaling to 1080+. That is probably why they upscale SD for cinema release (but you will notice that the image details are not 150dpi small or are blended too much).

Now with 4:4:4 the colour accuracy should help a lot in the accurate upscaling (because at 300dpi, upscaled 150dpi 4:2:2 colour detail canbe inaccurate and/or equivalent to 4:1:1 double hieght 300DPI, now go to 450dpi (around SHD 8mp) 6:1:1 triple hieght colour 450dpi. At 4:4:4 150dpi you should end up with 450dpi 3:1:1 triple height colour instead, but still 150dpi accurate colour areas and detail so the colours should be well blended, and match the corresponding detail edges because of the correspondance between detail and colour (in a non bayer system). This is also very important in effective special effects,as the boundaeries should be easily and acuratly bleneded.

It all seems trivial, I know, but that is the unfortunate nature of the movie business. I also have my own unique upscaling concept that I want to try one day on true 4:4:4 footage.

Now it maybe true that 4:2:2 offers better compression, but I imagine that is because we are throwing 1/3rd the information away that the eye does not really notice at these resolutions.

I think implementing 4:4:4, bayer, and higher bit depth side support in your software would be good. It would come in handy for regular customers too, as the industry is moving to single chip consumer cameras, three chip prosumer and professional cameras, and bayer movie cameras.

Thanks

David. I am still going through this thread before I get to your over post you told me about (spent 17 hours + on catching up on the three threads, so far)

[Wayne Morellini]

<<<-- Originally posted by Rob Scott : Product -
http://www.vinc.com/product.asp?ID=56&PID=21&SEQ=2

News articles -
http://www.cdfreaks.com/news/9852
http://news.designtechnica.com/article2925.html -->>>

If you look up Sigma Design on the web (even www.via.com.tw for embedded eden itx formfactor DVD players) I think they are the ones that make the chip sets for these. They ussually list their DVD player manufacturers somewhere on their site, or mention them in their news sections.


<<<-- Originally posted by Mike Metken : Guys,
..
There is a Sony HDC-X300 1080p camera that costs $15K. Canon HD auto focus/manual lens for it is $7K. It outputs HD SDI. If you can make the storage box work with this camera, you've made a Sony F900 killer.
..
Mike -->>>

Is that the 1440*1080 chip version there using in the Sony HDV1, and Canon XLHD?


<<<-- Originally posted by Les Dit :
There are a couple of other methods out there, one from Kodak ( patented, of course ) that seems to deal more with defect management than the actual demosaik itself.
-Les -->>>

I think somebody said that the Kodak routine was the best one out there.


<<<-- Originally posted by Steve Nordhauser :Maybe Obin could do some test sequences of fast panning on objects with large vertical lines (buildings?) to test the rolling shutter artifacts at 24 and 48fps. I really don't know how objectionable the skew really is - everyone talks about it but I've never seen images other than packing labels on conveyor belts.
Steve -->>>

:)

Well guys your doing a good job, and answering many of the questions I had, thanks, hope it all goes well.

[Wayne Morellini]

<<<-- Originally posted by Obin Olson : uggh...I posted what i have been doing with HD at dv.com and got a bunch of child-like responses. Guess I will stick to this site! -->>>

Don't let it faze you Obin, I have seen the same on other forums, they probably will sing a different tune when a worked out system is availble. You probably will be one of the most interesting people on the forums when you have it worked out and show off real work.

Have a look at these, they might help cheer you up.

www.hoojum.com/index.php

http://hoojum.com/gallery/slideshow....e&slide_full=1
www.mini-itx.com/news/nanode/

Seen that plugin article, at least we know who has the best lossless codec at the moment "BitJazz's", sheer is close behind.

[Rob Lohman]

Everyone: I've gone through the whole thread and merged almost
all posts that someone made back-2-back (except for your last
large ones above here Wayne). Some posts counts has come
down as a result. No information has been removed!

Please try to avoid posting several times in a row. Stick to one
post unless you have very large posts like Wayne above. Thanks!

Steve: how can my numbers be an average transfer rate? At
fixed framerates this should be a sustained (uncompressed)
transfer rate (one way, not two way).

Obin: Can you please answer the following that was still left
unanswered?

1) do you have a RAW 10 bit pre-bayer file for us? This CAN be a lower resolution STILL file. No movie, no high res.

2) which lens(es) are you using with your camera now?

3) could it be the stuttering in that movie is due to your harddisk not keeping up?

4) which camera settings did you use to record that movie? (gain / exposure)

5) did you do any color correction or other conversion to the material?

6) Which Bayer algorithm did you use?

This are important questions to the people looking at also getting
a chip or not...

Rob S: how do you determine how many FPS you can write to disk?

[Wayne Morellini]

Sorry Rob, so late, and I was trying to address significant posts to people individually. I don't think I will have much need to post here in the future as people are allready asking the right questions, and most significant issues have allready been covered (and I think that will be the trend in the other threads to).

Thanks Rob, your effort is most appreciated.

[Rob Scott]

Rob L wrote: how do you determine how many FPS you can write to disk?

I generate a set of bogus frames (of the correct size), shove 'em out to disk and keep track of how long it takes to write out n frames.

Er ... well, that's how I'm doing it with the code I'm writing. Is that what you meant, or was it more general?

[Rob Lohman]

Wayne: no problem! You wrote a lot as well. Your input is much
appreciated.

Rob S: well, both general and in this case. I've always had a hard
time wrapping my head around framerates and things like CPU
usage etc.

I was figuring you have one thread filling a circular buffer and
one thread reading is. When the the filling thread has caught up
with the reading thread you cannot write the data fast enough
to disk. How to express this in numbers....

Ofcourse you can write 1 GB to disk and time how long it takes
and divide that as well indeed....

It's just that I've never really dived into things like that.

[Rob Scott]

Rob Lohman wrote:
I was figuring you have one thread filling a circular buffer and one thread reading is. When the the filling thread has caught up
with the reading thread you cannot write the data fast enough to disk. How to express this in numbers....

Yeah, that's exactly right. I don't really have a scientific way of doing it. I have a "control block" class that acts as a gateway to the circular buffer. The frame generator grabs a buffer and stuffs in a frame at a predefined rate (24 fps, 30 fps, etc.). If the "frame generator" stuffs in frames too fast for the "writer" to handle then the "control block" marks up a "missed frame" but continues on. The "generator" runs for a preset period of time and stops. At this point, I stop disk activity; then count up the total time elapsed and the number of frames actually written to disk.

(Edit - at the end I also can determine how many frames were skipped.)

[Steve Nordhauser]

OK.
Rob L: average vs peak clock rates. The average is the visible image size x frame rate (1280x720x24fps = 22.1Mpix/sec). Due to blanking time, for the SI-1300 to do that rate, the pixel clock is 27MHz. This is the rate that, during image readout, the bus must meet. With an HD with a big buffer, the average # works fine.

Wayne: Rolling shutter. I think you have it. The rolling shutter 'artifact' is tied to the single frame readout time. The time from the first line read out to the last is always one frame time. You can run the SI-1300H up to 60fps. This will drop the time for one frame from 1/30 to 1/60 sec, reducing the artifact. As you say, this will reduce the integration time (light sensitivity). UNLESS (this was the hmmmm part), you read out the frame at 1/60 sec and then have a long vertical blanking time - all the pixels will continue to integrate during the trash frame time. Now you are integrating at 1/30th sec, reading out at 1/30th sec average, but the frame you read is done top to bottom at 1/60th.

Wayne: On making money. Yes, we will be interested in building systems and creating bundles. I have to wait for my R&D group (you guys) to come up with a product. Seriously though, I will always be clear what is public domain to future customers. I don't have the resources to solve all these problems, but I may help out. If that BitJazz codec does what it says - 2:1 lossless faster than the speed of light, I will consider negotiating a volume license and give a license away with the cameras. I do have the corporate blessing to be helping here. I think great things will happen.

Wayne: USB 2.0. To get high data rates, you eat a fair amount of CPU time. Also, transfer rates are highly dependent on the host controller. Intel ICH4 and 5 south bridges are 2-3x faster than external controllers. The typical USB interface chip has small FIFOs making real-time tough at high rates when you don't want to drop frames. We don't compress in the camera because either you want lossless, which only gets 2:1 or lossy and then you need to be able to select a lot of variables. CPU compression and raw recording are more cost effective.

[Jason Rodriguez]

Obin,

Right now Bitjazz is nice, but it's only 8-bit. I also believe that's real-time ecoding for 8-bit SD video, not HD video.

On the rolling shutter topic again:
How slow or fast does the shutter have to be in order to get away from the artifacts. From the footage that I saw on the wmv file, there's apparent stretching artifacts at all. How bad would it be at 24fps, with a 1/24 shutter speed? Also couldn't we just blank for 1/48 of a second and then take the next frame on the next 1/24th of a second, so it's imitating a mechanical shutter at 180 degrees (like a film camera). I think this was mentioned before, but I'm not sure what happened in the response. I heard something about a firmware upgrade needed.

The problem is that if we clock the chip at a high Mhz to satisfy the problem, and drop frames, then we're loosing the ability to shoot slow-motion. Not good.

[Rob Scott]

Also couldn't we just blank for 1/48 of a second and then take the next frame on the next 1/24th of a second ... I heard something about a firmware upgrade needed.

I believe the firmware upgrade was just to allow a long blanking time. By setting it to 48 fps and then grabbing every other frame, we get the same effect. (Unless I'm missing something.)

The problem is that if we clock the chip at a high Mhz to satisfy the problem, and drop frames, then we're loosing the ability to shoot slow-motion. Not good.

The way I envision it, the software would adapt to what you're trying to do -- if you want 48-60 fps, it would not try to skip frames in between. The skipping would only be valid in the 24-30 fps range.

[Jason Rodriguez]

I see what you're saying, but then you have the problem of too much motion blur when shooing from 48-60fps. Typical film shutters work at half the frame rate. So for 48-60fps, we need a shutter that's working effectively at 1/96th to 1/120th of a second.

[Steve Nordhauser]

Obin:
I think there are two sets of test images the gang would like to see. First is a horizontal pan with lots of vertical edges. This should show up the rolling shutter problems. Or, a stationary camera view and someone driving by.

Second is Les has asked to see two consecutive frames - no movement between them. He has a lot of experience in noise analysis and will help understand the requirements for image enhancement.

Group:
To sweeten the pot, anyone who *substantially* moves this project along - programming, research, testing can have a 20% discount on a camera or bundle purchase from SI. Again, Rob coordinates the software, not me. Obin, you are clearly carrying your load on test image generation and ponying up for a camera purchase. You don't have to be a programmer, just take the lead in some valuable aspect of getting to usable tools.
Steve