Development Platform for DIY cameras

Keith Wakeham · May 7th, 2005, 07:23 AM

Fpga's are fairly confusing when you start getting into command based protocols like usb 2.0, firewire, ata, and sata, Other than that they are fairly easy for some things, but you continually have to use a lot stupid commands.

From what it sounds like you don't understand how HD-SDI works, and if you don't then a lot of others don't either.

4:2:2 refers to y (luma) and uv (2 chorma) channels. The y channel has 1920 x 1080 active pixels. It doesn't matter what is in those pixels, whats coming off my ccd is is 1920 x 1080, it doesn't get debayered, just transported over the luma channel, the signal IS still perfectly bayer, untouched. So what is in the chroma channels. Well first, in SDI it becomes one channel of 1920 x 1080 rather than 2 of 960 x 1080. Initially nothing is sent over it, but i could send a second bayer if i wanted because it is one 1920 x 1080 channel, not the 2 that everyone thinks it is.

I emphasize, the bayer is not touched, just because it goes over the luma channel doesn't mean anything. I could send audio over the luma channel if i wanted so long as it is going at 74.25Mhz it doesn't care what it is. Preview won't look good but will be watchable with my design, and latter a plugin for avid and fcp can be made to debayer and the like.

As to hooking a hard drive up to a computer. That is why I said it would be a deck. Incorporating a file system would mean that you will need 3 things. FPGA, Processor, and Ram. I'd need a team of engineers to do this, not just me an engineering student. So that is why you control it through a serial port just like a betacam deck and use your nle to capture with a hd-sdi card. I guess a lot of people here are probably hung up on firewire and computer things rather than proper industry standards.

I'm stressing that i'm going to avoid protocol implementation at all cost (more like at huge time and cost savings). Whenever you have to start doing heavy responding to requests and having stuff do things like "handshake" its gonna slow down and become way more complicated. I'm not dealing with it. What i can do is a deck. A deck that would beat a hdcam deck and be way cheaper.

Rob Lohman · May 7th, 2005, 07:28 AM

It is true I know next to nothing about (HD-)SDI. However, I think the confusion
came from 4:2:2 which indicates some things.

It's good to know the bayer is untouched. That's all that matters in the end.

It never occured to me that we could use HD-SDI to get the footage into the
computer after it has been shot. That's a good and interesting point!

What price do I need to think about for such an HD-SDI card for my Windows
computer?

p.s. does this mean that HD-SDI has less bandwidth available for the U & V
channels? So either they are using 50% of the pixels or they are combined
into one channel (effectively getting 1920 x 1080 pixels)? That would mean
HD-SDI is two channels of 1920 x 1080?

Keith Wakeham · May 7th, 2005, 07:45 AM

Decklink by black magic designs is what i was thinking for hd-sdi capture. They have a barebones pci-x capture board that would be perfect for about 600 dollars usd. It has hd-sdi in, out, and the serial controller on it. It is only 4:2:2, but that is all i am working with, well, only half of that technically.

http://www.blackmagic-design.net/

You pretty much hit the nail on the head. HD-SDI is really only 2 channels of 1920 x 1080, but one of the channels gets chopped into 2 half channels at some latter point after being deserialized. What goes into the serializer is 2 10bit channels and an ancillary data channel (which i don't use).

I'm not sure exactly how the 2 colour half channels are combined, i think they are rotated (CbCrCbCr) or one might be apended to the other, but the rotating sounds so much easier because could be done with a handful of transistors.

I so need to shell out about $150 for all the stupid smpte documents i need, rather than going on the small amount of data from data sheets.

I know that nobody shoots steroscopic anymore (except for the spykids movie) but its so appealing because it is so easy to implement. All it would require is another sensor and to run my data packing code a second time. I don't know if anyone would actually be interested, just a thought.

Steven Mingam · May 10th, 2005, 03:15 AM

Hi, i've been a silent reader of all those HD camera thread for some time now, and have been really pleased by the developpement you're doing (no pc, fpga).

I wanted to point you out to this article :
http://www.linuxdevices.com/articles/AT3888835064.html
where the guy add some compression to the camera, using fpga... and the source code is available. Ok, the codec isn't a really good choice _at all_ (imho, it suck hard) but that prove that a sort of lossless codec _on the bayer_ (like DCT + huffman or whatever entroy coding) is possible using fpga ... cutting down a 1280x720@24fps stream to 25-30Mb/s would fit on a single harddrive bandwidth, even a 2.5" one.

Also, the guy aside me is working on a CMOS camera head using National, now Kodak KAC-9638 (color version is 9648) and i found the price really interesting (18$ heh) ... Is Rolling Shutter _that bad_ ? it would really cut down the price for a 720p camera.

Keith Wakeham · May 10th, 2005, 07:46 AM

I have been going through the stuff on cameras designed by elphel and its been somewhat useful. It looks like a route that could be useful to a camera design as they already have work done for an 11mpixel kodak sensor. The main think is that it actually runs an embedded linux for the gigabit link.

I do not want to have to put in a processor and run software as that is currently beyond me. HD-SDI isn't because that data flows regardless of if something is recieving it or not.

I don't know to much about rolling shutter, as I have never had to deal with it, since my design uses an interline transfer ccd. Their have been numerous discussions and the problems associated with it deal with objects moving and becoming skewed. I'd suggest searching the forum.

Cmos sensors are fairly cheap. Digikey lists a bunch, but most have tiny optical formats (usually 1/4" at most) and usually won't handle the framerate at the higher resolution. And not all cmos have performance like ibis or altasens. Older cmos technologies had much more noise in them, and this technology is still used today to make sensors for things like cameraphones.

I'm sticking with my ccd design because right now i've go it down to a handful of fpga programs that are easily handled. I had to scrap some of the stuff i'm working on but i did away with needing external ram in the camera head, which is good.

Keith Wakeham · May 10th, 2005, 01:13 PM

Update:

The design was getting pretty big and bulky, but because of some corrospondance with kodak I've found that I can get 30P that is compatible with SMPTE 274 without having to use asyncronous FIFO's. I'm crunching the numbers now to see if i can do that same for 24p because that is all that really matters to me right now, and all a single hard drive could handle in 10bit. (keep in mind only one channel, CbCr channel is empty)

Fpga will still be used for timing and appending the dual outputs together and controlling the commands to the serializer, but I feel that i'm really close to having a HD-SDI bayer camera design finished. Now i just need to get the couple of thousand dollars for all the parts to start testing.

Since i'm only a student I'm going to get some of the electronics guys in my university to look over my final designs in a week or two. Make sure i didn't miss anything.

Michael Maier · May 11th, 2005, 09:53 AM

Quote:

Originally Posted by Obin Olson

I am sure i'ts the IBIS5 sensor as that is the only sensor that I know of that's 2/3rd inch GLOBAL and color...it's shit..not a good image...looks like a spy cam used in undercover TV news..if it's firewire then it's also 8bit, not good enough for shooting you will need 10-12bit images..8bit will also look like a cheap 1ccd spy camera

I was re-reading this post of yours Obin and I was thinking, isn't the Drake using this very same sensor? It doesn't look like a 1 CCD spy camera to me.

Obin Olson · May 11th, 2005, 10:24 PM

I would guess it's early to make any decisions on the Drake as they have not shot any outdoor sunlit stuff yet...but I would bet the color is not that good as you would have to pump the color in post so hard for a ibis chip..I have seen the images from Ibis...but then again maybe they have found a way....don't think anyone knows yet...

one more thing..maybe FillFactory has a NEW IBIS chip that is better...

Obin Olson · May 11th, 2005, 10:26 PM

Keith you are telling us your FPGA design is done?!?! do you have RAW record-to-disk a viewfinder and camera control?

christ that would be GREAT news..I am not sure how you could have done it that fast..are you working with 12bit 1080x1920 images at 24fps?

Obin Olson · May 11th, 2005, 10:37 PM

I think the HD-SDI idea is REALLY great! good work Kieth, keeping a signal on a HD-SDI path works well because you can hook that to many professional decks, cards, etc...and I see how a bayer image could go over sdi as it's really just LUMA!!!!! amazing! that is great! and like you say we can just de-bayer it in post and or compress it!

so what is left to get camera control and images out for a viewfinder or monitor?

Steven Mingam · May 12th, 2005, 01:15 AM

Quote:

I am not sure how you could have done it that fast...

well where i work, the hardware guys would do it in 15 days with compression on a fpga ... they just have to add a ATA interface or something to some design they already have. Too bad their products don't have much in common with cinema.

Keith Wakeham · May 12th, 2005, 07:03 AM

I don't have the major parts yet, and by that i mean the sensor. I need about 1500 usd to get a quality sensor and an eng grade sensor, i'm still gathering money for that,i hate being a student.

I had this elaborate chain of programs started on. The flow chart was huge, and involved all sorts of FIFO's and buffers and stuff. But that all got scraped because I found out i could read fake lines, so even though the sensor only has 1092 real lines, i can read 1125 and then config line timings. I reduced the design to just 2 programs. A timing program and a line append program.

But now I have the timing program pretty much done. Doing simulation on it is all that is left and setting a method to enable switching its modes, which are 24p, 25p, 30p that still all complies with smpte 274 right of the chip. I have a poor program right now that interleaves both dual output channels together instead of the complete line append, but that is only for testing. Would result in a really stupid picture and lots of post processing to get it normal.

So one program is practically done, just making sure that everything works perfectly and i can switch modes. It also incorporates timing pulses for connecting to an rgb monitor, which means that you can monitor the bayer in b/w. Not a great solution but it should work. Just have to finish a line buffer program that i started because the sensor is dual output.

I stress, this is only the head design and isn't built - yet, you need a computer with hd-sdi to capture so far or a dvcpro or hd-cam deck. The hard drive deck design is still in its infancy because I need to get NDA's sent out to the companies I want to get IP cores off of for ata and sata. But from some of the companies it sounds that this can be pretty easy to do what i want and record directly to LBA's.

HD-sdi specs out 10bit per channel, but it is possible to expand this, just need to use the 2nd channel which i have left empty for now. Could go as high as 20bit then and you would have a lot of data.

Sure a company could do this in 2 weeks, but i'm a student with only a little fpga background and very little money. Also i took down the website recently, I was having problems with it and i really hated the free server. Shouldn't complain, it was free.

I had the design looked over by some people in know in the faculty and they are fairly confident that it will work as soon as i get the sensor as long as my support electronics are tested. which right now are kind of a jumble and mess of wires and stuff.

So to summary:
No sensor means no pictures, just fpga programs and simulations and electronic designs and testing.
The deck is waiting for an ip core and documentation.

Steven Mingam · May 12th, 2005, 08:06 AM

Sorry if i sounded offensive, it was just a quote for Obin to say that with the right guys, you could do it really quickly. I'm a student too, and perfectly understand the problems you're facing, and i'm still impressed (and pleased) by the quick progress you're making.

For the sensors price, well that's why i proposed this kodak cmos sensor, 18$ is much more affordable than 1500 for a student ;)

Obin Olson · May 12th, 2005, 08:24 AM

Keith can you call me or I call you?

910.262.3434

Keith Wakeham · May 12th, 2005, 09:33 AM

Steve I wasn't really retaliating i was more joking around. I knew what you meant and i knew you didn't mean any offense to me at all.

The problem for me is that it i can control the timmings for a ccd (as long as i got some documentation on the clock signals) but with a cmos it is harder to do that and doing things like increasing vertical clock time and waiting times for a cmos is much harder because you need to program registers and stuff like that and will make it much to hard to get the smpte 274 compliance without having to use asyncronous fifos, which I'm so happy that i can avoid.

I'd purchase an eng sample right now, but in order to do that i need to purchase an evaluation kit or at least one production sensor. I haven't asked if they would sell just one eng because i'm a student, but maybe i should asked, or see if i can get a few eng samples instead of a production one.

Obin:
I can't call you today because i'm on the road later today and I won't be home until 10 pm my time which is gmt -3.5. Is that to late to call you? I'll try then, if not i'll do it tomorrow if you don't see this in time.

May 7th, 2005, 07:28 AM	#77
Rob Lohman RED Code Chef Join Date: Oct 2001 Location: Holland Posts: 12,514	It is true I know next to nothing about (HD-)SDI. However, I think the confusion came from 4:2:2 which indicates some things. It's good to know the bayer is untouched. That's all that matters in the end. It never occured to me that we could use HD-SDI to get the footage into the computer after it has been shot. That's a good and interesting point! What price do I need to think about for such an HD-SDI card for my Windows computer? p.s. does this mean that HD-SDI has less bandwidth available for the U & V channels? So either they are using 50% of the pixels or they are combined into one channel (effectively getting 1920 x 1080 pixels)? That would mean HD-SDI is two channels of 1920 x 1080? __________________ Rob Lohman, visuar@iname.com DV Info Wrangler & RED Code Chef Join the DV Challenge \| Lady X Search DVinfo.net for quick answers \| Buy from the best: DVinfo.net sponsors

May 7th, 2005, 07:23 AM	#76
Keith Wakeham Major Player Join Date: Jan 2005 Location: St. John's, NL, Canada Posts: 416	Fpga's are fairly confusing when you start getting into command based protocols like usb 2.0, firewire, ata, and sata, Other than that they are fairly easy for some things, but you continually have to use a lot stupid commands. From what it sounds like you don't understand how HD-SDI works, and if you don't then a lot of others don't either. 4:2:2 refers to y (luma) and uv (2 chorma) channels. The y channel has 1920 x 1080 active pixels. It doesn't matter what is in those pixels, whats coming off my ccd is is 1920 x 1080, it doesn't get debayered, just transported over the luma channel, the signal IS still perfectly bayer, untouched. So what is in the chroma channels. Well first, in SDI it becomes one channel of 1920 x 1080 rather than 2 of 960 x 1080. Initially nothing is sent over it, but i could send a second bayer if i wanted because it is one 1920 x 1080 channel, not the 2 that everyone thinks it is. I emphasize, the bayer is not touched, just because it goes over the luma channel doesn't mean anything. I could send audio over the luma channel if i wanted so long as it is going at 74.25Mhz it doesn't care what it is. Preview won't look good but will be watchable with my design, and latter a plugin for avid and fcp can be made to debayer and the like. As to hooking a hard drive up to a computer. That is why I said it would be a deck. Incorporating a file system would mean that you will need 3 things. FPGA, Processor, and Ram. I'd need a team of engineers to do this, not just me an engineering student. So that is why you control it through a serial port just like a betacam deck and use your nle to capture with a hd-sdi card. I guess a lot of people here are probably hung up on firewire and computer things rather than proper industry standards. I'm stressing that i'm going to avoid protocol implementation at all cost (more like at huge time and cost savings). Whenever you have to start doing heavy responding to requests and having stuff do things like "handshake" its gonna slow down and become way more complicated. I'm not dealing with it. What i can do is a deck. A deck that would beat a hdcam deck and be way cheaper.

May 7th, 2005, 07:45 AM	#78
Keith Wakeham Major Player Join Date: Jan 2005 Location: St. John's, NL, Canada Posts: 416	Decklink by black magic designs is what i was thinking for hd-sdi capture. They have a barebones pci-x capture board that would be perfect for about 600 dollars usd. It has hd-sdi in, out, and the serial controller on it. It is only 4:2:2, but that is all i am working with, well, only half of that technically. http://www.blackmagic-design.net/ You pretty much hit the nail on the head. HD-SDI is really only 2 channels of 1920 x 1080, but one of the channels gets chopped into 2 half channels at some latter point after being deserialized. What goes into the serializer is 2 10bit channels and an ancillary data channel (which i don't use). I'm not sure exactly how the 2 colour half channels are combined, i think they are rotated (CbCrCbCr) or one might be apended to the other, but the rotating sounds so much easier because could be done with a handful of transistors. I so need to shell out about $150 for all the stupid smpte documents i need, rather than going on the small amount of data from data sheets. I know that nobody shoots steroscopic anymore (except for the spykids movie) but its so appealing because it is so easy to implement. All it would require is another sensor and to run my data packing code a second time. I don't know if anyone would actually be interested, just a thought.

May 10th, 2005, 03:15 AM	#79
Steven Mingam Regular Crew Join Date: Aug 2004 Posts: 91	Hi, i've been a silent reader of all those HD camera thread for some time now, and have been really pleased by the developpement you're doing (no pc, fpga). I wanted to point you out to this article : http://www.linuxdevices.com/articles/AT3888835064.html where the guy add some compression to the camera, using fpga... and the source code is available. Ok, the codec isn't a really good choice _at all_ (imho, it suck hard) but that prove that a sort of lossless codec _on the bayer_ (like DCT + huffman or whatever entroy coding) is possible using fpga ... cutting down a 1280x720@24fps stream to 25-30Mb/s would fit on a single harddrive bandwidth, even a 2.5" one. Also, the guy aside me is working on a CMOS camera head using National, now Kodak KAC-9638 (color version is 9648) and i found the price really interesting (18$ heh) ... Is Rolling Shutter _that bad_ ? it would really cut down the price for a 720p camera.

May 10th, 2005, 07:46 AM	#80
Keith Wakeham Major Player Join Date: Jan 2005 Location: St. John's, NL, Canada Posts: 416	I have been going through the stuff on cameras designed by elphel and its been somewhat useful. It looks like a route that could be useful to a camera design as they already have work done for an 11mpixel kodak sensor. The main think is that it actually runs an embedded linux for the gigabit link. I do not want to have to put in a processor and run software as that is currently beyond me. HD-SDI isn't because that data flows regardless of if something is recieving it or not. I don't know to much about rolling shutter, as I have never had to deal with it, since my design uses an interline transfer ccd. Their have been numerous discussions and the problems associated with it deal with objects moving and becoming skewed. I'd suggest searching the forum. Cmos sensors are fairly cheap. Digikey lists a bunch, but most have tiny optical formats (usually 1/4" at most) and usually won't handle the framerate at the higher resolution. And not all cmos have performance like ibis or altasens. Older cmos technologies had much more noise in them, and this technology is still used today to make sensors for things like cameraphones. I'm sticking with my ccd design because right now i've go it down to a handful of fpga programs that are easily handled. I had to scrap some of the stuff i'm working on but i did away with needing external ram in the camera head, which is good.

May 10th, 2005, 01:13 PM	#81
Keith Wakeham Major Player Join Date: Jan 2005 Location: St. John's, NL, Canada Posts: 416	Update: The design was getting pretty big and bulky, but because of some corrospondance with kodak I've found that I can get 30P that is compatible with SMPTE 274 without having to use asyncronous FIFO's. I'm crunching the numbers now to see if i can do that same for 24p because that is all that really matters to me right now, and all a single hard drive could handle in 10bit. (keep in mind only one channel, CbCr channel is empty) Fpga will still be used for timing and appending the dual outputs together and controlling the commands to the serializer, but I feel that i'm really close to having a HD-SDI bayer camera design finished. Now i just need to get the couple of thousand dollars for all the parts to start testing. Since i'm only a student I'm going to get some of the electronics guys in my university to look over my final designs in a week or two. Make sure i didn't miss anything.

May 11th, 2005, 10:24 PM	#83
Obin Olson Trustee Join Date: Jan 2003 Location: Wilmington NC Posts: 1,414	I would guess it's early to make any decisions on the Drake as they have not shot any outdoor sunlit stuff yet...but I would bet the color is not that good as you would have to pump the color in post so hard for a ibis chip..I have seen the images from Ibis...but then again maybe they have found a way....don't think anyone knows yet... one more thing..maybe FillFactory has a NEW IBIS chip that is better...

May 11th, 2005, 10:26 PM	#84
Obin Olson Trustee Join Date: Jan 2003 Location: Wilmington NC Posts: 1,414	Keith you are telling us your FPGA design is done?!?! do you have RAW record-to-disk a viewfinder and camera control? christ that would be GREAT news..I am not sure how you could have done it that fast..are you working with 12bit 1080x1920 images at 24fps?

May 11th, 2005, 10:37 PM	#85
Obin Olson Trustee Join Date: Jan 2003 Location: Wilmington NC Posts: 1,414	I think the HD-SDI idea is REALLY great! good work Kieth, keeping a signal on a HD-SDI path works well because you can hook that to many professional decks, cards, etc...and I see how a bayer image could go over sdi as it's really just LUMA!!!!! amazing! that is great! and like you say we can just de-bayer it in post and or compress it! so what is left to get camera control and images out for a viewfinder or monitor?

May 12th, 2005, 07:03 AM	#87
Keith Wakeham Major Player Join Date: Jan 2005 Location: St. John's, NL, Canada Posts: 416	I don't have the major parts yet, and by that i mean the sensor. I need about 1500 usd to get a quality sensor and an eng grade sensor, i'm still gathering money for that,i hate being a student. I had this elaborate chain of programs started on. The flow chart was huge, and involved all sorts of FIFO's and buffers and stuff. But that all got scraped because I found out i could read fake lines, so even though the sensor only has 1092 real lines, i can read 1125 and then config line timings. I reduced the design to just 2 programs. A timing program and a line append program. But now I have the timing program pretty much done. Doing simulation on it is all that is left and setting a method to enable switching its modes, which are 24p, 25p, 30p that still all complies with smpte 274 right of the chip. I have a poor program right now that interleaves both dual output channels together instead of the complete line append, but that is only for testing. Would result in a really stupid picture and lots of post processing to get it normal. So one program is practically done, just making sure that everything works perfectly and i can switch modes. It also incorporates timing pulses for connecting to an rgb monitor, which means that you can monitor the bayer in b/w. Not a great solution but it should work. Just have to finish a line buffer program that i started because the sensor is dual output. I stress, this is only the head design and isn't built - yet, you need a computer with hd-sdi to capture so far or a dvcpro or hd-cam deck. The hard drive deck design is still in its infancy because I need to get NDA's sent out to the companies I want to get IP cores off of for ata and sata. But from some of the companies it sounds that this can be pretty easy to do what i want and record directly to LBA's. HD-sdi specs out 10bit per channel, but it is possible to expand this, just need to use the 2nd channel which i have left empty for now. Could go as high as 20bit then and you would have a lot of data. Sure a company could do this in 2 weeks, but i'm a student with only a little fpga background and very little money. Also i took down the website recently, I was having problems with it and i really hated the free server. Shouldn't complain, it was free. I had the design looked over by some people in know in the faculty and they are fairly confident that it will work as soon as i get the sensor as long as my support electronics are tested. which right now are kind of a jumble and mess of wires and stuff. So to summary: No sensor means no pictures, just fpga programs and simulations and electronic designs and testing. The deck is waiting for an ip core and documentation.

May 12th, 2005, 08:06 AM	#88
Steven Mingam Regular Crew Join Date: Aug 2004 Posts: 91	Sorry if i sounded offensive, it was just a quote for Obin to say that with the right guys, you could do it really quickly. I'm a student too, and perfectly understand the problems you're facing, and i'm still impressed (and pleased) by the quick progress you're making. For the sensors price, well that's why i proposed this kodak cmos sensor, 18$ is much more affordable than 1500 for a student ;)

May 12th, 2005, 08:24 AM	#89
Obin Olson Trustee Join Date: Jan 2003 Location: Wilmington NC Posts: 1,414	Keith can you call me or I call you? 910.262.3434

May 12th, 2005, 09:33 AM	#90
Keith Wakeham Major Player Join Date: Jan 2005 Location: St. John's, NL, Canada Posts: 416	Steve I wasn't really retaliating i was more joking around. I knew what you meant and i knew you didn't mean any offense to me at all. The problem for me is that it i can control the timmings for a ccd (as long as i got some documentation on the clock signals) but with a cmos it is harder to do that and doing things like increasing vertical clock time and waiting times for a cmos is much harder because you need to program registers and stuff like that and will make it much to hard to get the smpte 274 compliance without having to use asyncronous fifos, which I'm so happy that i can avoid. I'd purchase an eng sample right now, but in order to do that i need to purchase an evaluation kit or at least one production sensor. I haven't asked if they would sell just one eng because i'm a student, but maybe i should asked, or see if i can get a few eng samples instead of a production one. Obin: I can't call you today because i'm on the road later today and I won't be home until 10 pm my time which is gmt -3.5. Is that to late to call you? I'll try then, if not i'll do it tomorrow if you don't see this in time.