Home Made HD Cinema Cameras - Technical Discussion

Rob Scott · July 22nd, 2004, 11:06 AM

Quote:

Wayne Morellini wrote:
No, I'm pretty certain there is DVI in (on some cards meant to take signals from HDTV receivers and sets).

OK, well I admit my ignorance. I don't have any experience with these sorts of standards :-)

Quote:

This means Rob doesn't have to build in support for HD-SDI, you just need to set the output format to something their is a transcoder for. Trying to do this live at the same time as recording might be a problem, unless Rob's software outputs a data stream to the other process

It might be possible to set up some sort of "frame server" system (see VirtualDub and Avisynth, or used shared memory across processes.

(edit - added "Avisynth" above.)

Ben Syverson · July 22nd, 2004, 11:09 AM

Quote:

No, I'm pretty certain there is DVI in (on some cards meant to take signals from HDTV receivers and sets)

Show me -- I want to see spec sheets.

Wayne Morellini · July 22nd, 2004, 12:08 PM

<<<-- Originally posted by Juan M. M. Fiebelkorn : Wayne,
Virtual memory is on disk, if it weren't the case it would disk writing. -->>>

Sorry my fault, very late, what I was meaning is the pages in memory before it is written to disk. The HD-SDI could access them (as it is synchronised live) before they go to disk, saving a disk read.

Ben, I am going to have to track that down, but I am sure that the most likey one that I have seen it in is the older ATI all in wonder cards (8500 and onwards though I don't know wether the 8500 had HD version) but I though equivalent top of the line Nvidia in/out cards have DVI in. It is interesting that ATI has comrpession hardware and uses the shaders/GPU to help, and clean up internet streams. It upscales 320H res WMV files very nicely.

Well after 4am here, I am going to have to go to bed (unless I just have to take up your challenge before I go to sleep).

Wayne Morellini · July 22nd, 2004, 12:12 PM

This must be a sign, my web browser is reporting that it can't find www.ati.com (there server must be down, good probably means some news release, better go to Toms and see if he has a review yet ;)

Juan M. M. Fiebelkorn · July 23rd, 2004, 02:25 AM

I've just made some more testing with Huffyuv (for the novice Huffyuv compresses but it is LOSSLESS, meaning that what you got compressed is the same as the original but takes half the storage space and half the bandwith)

It was as follows:

Source video 640x720 (the resolution of the Green channel of a 1280x720 sensor)
YUY2 (because my disk isn't a WD Raptor and I'm reading and writing at the same time, and because YUY2 gives me an aproximate situation of the real task, because of datarate)

Well, reading and writing from same disk, DEcompressing and REcompressing at the same time Huffyuv gives me no less than 28 fps on an Athlon XP 2000.

Really nice...:)

PS: if anybody here has the knowledge, please make a modification to take images directly from memory (eliminating Disk Reading) and test it,please.

A note to any programmer:
With this video things always remember to pay special attention to the CPU cache.Avoid cache misses as much as posible.

Wayne Morellini · July 24th, 2004, 07:49 AM

<<<-- Originally posted by Ben Syverson : Show me -- I want to see spec sheets. -->>>

OK

Just finished my search, and I must applogise, I was wrong, there are no 3D cards with DVI input (yet) that I can find. I put this down to a mistake in reading an all in wonder 8500 reveiw a few years ago. Even though I found ads advertising DVI input on some 3D cards, it looks like these are all mistakes. I did find some HD cards that had DVI Inputs, one expressly to recieve a signal from a HD receiver, but these are not the low priced commodity cards we would need to look for. But still it was only a suggestion for camera manufactures to research and consider, and still is a cheap option, but when commodity cards/mainboards come out with the inputs. (I did also find 10bit DVI reference, and you can also pack pixels for greater than 10bits). I expect that this will happen sometime.

Apparent not right DVI in:
http://www.pricewatch.com/1/264/3234-1.htm
http://www5.sharkyextreme.com/hardwa...211_1369251__3

Expert advice on DVI-in (if you sign up??)
http://www.experts-exchange.com/Hard..._21039171.html

Two actual solutions:
http://www.cellarcinemas.com/cgi-bin/store/HD3.html
http://www.primagraphics.co.uk/ds/pumapci.html

Sorry

Wayne.

Wayne Morellini · July 24th, 2004, 08:21 AM

Re-edited, actual performance figures incorporated.

Jaun, that is close to my estimates I did a longtime ago on either the viper, or 10 bit thread. I think I actually came closer to 1600Mhz, which is closer to the actual clock speed the 2000xp runs at. With use of mmx, SSE etc (they recon Huffy doesn't use) maybe we can bring that closer to 1ghz. This sounds strange though, because I remember a link that had huffy doing 38MB/s on a celeron 416, so 1.6Ghz should be more than enough for a single chip 720p. I think that there was a link to a even faster one. It would be good If we could find 3-5:1 true lossless, for not much more processing power.

People seemed to get interested in wavelets, as a solution for both lossless, near lossless, and high compression ratios (I think they are right), but we can eventaully use any plugin comrpession codec we want (hopefully).

People on these forums have a tendacy to not to reply to issues previously discussed (except for me). It canbe dissapionting to ask a question, or make a suggestion, and not get an answer. Unfortunately this means reading long threads first.

Good programming suggestions Jaun, I've been in contact with the Rob's on these issues for a while, and did a post a while back.

My estimates on I ghz VIA being enough for single chip 720p raw, is also because Rob is reporting 60FPS for his machine in his development blog at the wiki since he swapped to machine code routines. The Windows C slow down seems too much, but what do I know.

Have fun.

Wayne.

Juan M. M. Fiebelkorn · July 24th, 2004, 10:49 AM

Wayne,
Huffyuv uses mmx, what it lacks is SSE, which would be better for processing 16 bits..
If you read my post again carefully, may be the 38 MB issue is true but when capturing.
When in capture mode you are not reading and writing from disk at the same time (which is my situation)..

About wavelets, I'm working on that.But remember that for a lossless compression you can use DCT which is faster and gives more or less the same gains (always around 4:1 with my tests)

Wayne Morellini · July 24th, 2004, 12:01 PM

4:1 is perfect compared to 2:1 that huffy simular compressions get.

Sorry, about the mmx mixup, I thought I read it didn't. If this is the case then potential speedups shouldn't be that dramatic from using SSE.

I am surprised that there is that much of a difference between writing to memory and writing to disk, there must be something wrong, modern disk writing should not be very intesive compared to Huffy compression. The routine I saw is only 14.7k long, that is so small that if it is in C, you probably could do a fast machine code version in 8K.

http://www.divx-digest.com/software/huffyuv.html

Issues for people in general on the threads (not for you Jaun):

Still, the issue that people don't realise (not you Jaun) is just how much how your system is setup will affect how fast your code works (potentially dramactically). Customising code to the particular processor being used can also speed up the code).

Aswell as this, C uses only a limited machine code instruction set in compile, and the c routine stack and OS calls can slow things down. Improvements canbe made using machine code to access all the machine code instructions and streamline the code. This canbe dramtic improvements. People shouldn't trust MS, OS, or C compiler to think for them and do the right thing. If game programmers did this we would be struggling to run anything better than Unreal Tournament.

So if anybody runs something on their system and it doesn't perform well enough they shouldn't assume that is it, it could work much faster, that is what people like Rob are exploring for us.

I am wondering if I should not just get a cheap USB camera, rather than waiting around.

Juan M. M. Fiebelkorn · July 24th, 2004, 12:08 PM

RAM has a bandwith of around 6.4 GigaBytes/s against a normal disk, 7200 rpm which writes no more than 30 MegaBytes/s.
You can have a compression routine that outputs far more than what you can record on disk, thus limiting your output....

Wayne Morellini · July 24th, 2004, 12:59 PM

Modern disk systems work with handeling electronics on each end (IDE Integrated Drive Electronics, DMA transfers etc). So this hardware can block download to disk electoriics which can then dribble the informatyion ou to the disk. You can write to memory, send a request to the hardware system to pickup the information and write it out via DMA, while your program perfoms more compression. But what we are talking about is also passing the output through memory to the comrpession program (replacing raw disk read with half as many comrpessed reads). Leaving out all the stuff about buffers etc to optimise this process. So you get the request, and you get the a number of cycles per 64bit/s (preferably 2 cycles).

We have determine the max sustain data rate for a really good disk is something like 50mb/s, we also are looking at raid.

In old disk system you had to do it all manually as you said.

Juan M. M. Fiebelkorn · July 24th, 2004, 01:02 PM

What disk is that??
not a 7200 rpm one for sure.
The fastest ATA disk in the world is the Western Digital Raptor SATA 72 MB/s

No other ATA disk (SATA or PATA) writes more...

I don't get the rest of your post, sorry.

Wayne Morellini · July 24th, 2004, 01:14 PM

Let's do something useful.

I was wondering what is the cheapest cameralink frame grabber out there. I did a google search and came up with a mass of hits, there is a range of cameras and framegrabbers out there. So I found the cameralink standards homepage, and product links. In this I also found small industrial capture computers running embedded windows.

I am very tired and haven't done much pricing yet, but I am posting it here so you guys can have a look. Maybe we are barking up the wrong tree, it is so prolific maybe there are ready made systems and capture software for pro video capture out there allready, that we might be able to get volume discount on. I don't remember if anybody has looked at this site yet.

Cameralink
http://www.machinevisiononline.org/p...ex.cfm?cat=129

Registered Cameralink products
http://www.machinevisiononline.org/p...ls.cfm?id=1108

Gigbe
http://www.machinevisiononline.org/p...ex.cfm?cat=167

Product Search
http://www.machinevisiononline.org/buyers_guide/

Wayne Morellini · July 24th, 2004, 01:26 PM

We discussed a raid system that sustained around 200MB/s with four of the drives, in the threads. You can search them easily, by selecting the most posts per page in your site preferences and using ctrl-F, to search each page for "raid".

When you said raptor before I thought you meant one of the HD video raid systems I've heard of, sorry, but 72mb/s sustianed would be great.

The rest of the stuff I mentioned is just how modern drive electronics makes it possible to transfer 8 bytes at a time (64bit memory bus) for around 2-3 memory cycles. If your in cache most of the time than the amount of extra processing time taken up by the disk can be minimal (of course it is a bit more complex and the processor is also accessing the capture card and writting to memory and that stuffs up the free flow of memory and can slow it down a bit). But still the drive write time should not effect the processor too much (except it has to wait for the drive) if done rightly.

Thanks

Wayne.

Juan M. M. Fiebelkorn · July 24th, 2004, 02:25 PM

Wayne, no RAID system with normal disks will give you 200 MB/s if yopu use normal disks ( I mean 4 normal disks).
It would give you 200 MB/s if they were 4 Raptor or $ SCSI 10,000 rpm disks.
Please believe me.

July 22nd, 2004, 12:08 PM	#33
Wayne Morellini Inner Circle Join Date: May 2003 Location: Australia Posts: 2,762	<<<-- Originally posted by Juan M. M. Fiebelkorn : Wayne, Virtual memory is on disk, if it weren't the case it would disk writing. -->>> Sorry my fault, very late, what I was meaning is the pages in memory before it is written to disk. The HD-SDI could access them (as it is synchronised live) before they go to disk, saving a disk read. Ben, I am going to have to track that down, but I am sure that the most likey one that I have seen it in is the older ATI all in wonder cards (8500 and onwards though I don't know wether the 8500 had HD version) but I though equivalent top of the line Nvidia in/out cards have DVI in. It is interesting that ATI has comrpession hardware and uses the shaders/GPU to help, and clean up internet streams. It upscales 320H res WMV files very nicely. Well after 4am here, I am going to have to go to bed (unless I just have to take up your challenge before I go to sleep).

July 22nd, 2004, 12:12 PM	#34
Wayne Morellini Inner Circle Join Date: May 2003 Location: Australia Posts: 2,762	This must be a sign, my web browser is reporting that it can't find www.ati.com (there server must be down, good probably means some news release, better go to Toms and see if he has a review yet ;)

July 23rd, 2004, 02:25 AM	#35
Juan M. M. Fiebelkorn Major Player Join Date: Jun 2004 Location: Buenos Aires , Argentina Posts: 444	I've just made some more testing with Huffyuv (for the novice Huffyuv compresses but it is LOSSLESS, meaning that what you got compressed is the same as the original but takes half the storage space and half the bandwith) It was as follows: Source video 640x720 (the resolution of the Green channel of a 1280x720 sensor) YUY2 (because my disk isn't a WD Raptor and I'm reading and writing at the same time, and because YUY2 gives me an aproximate situation of the real task, because of datarate) Well, reading and writing from same disk, DEcompressing and REcompressing at the same time Huffyuv gives me no less than 28 fps on an Athlon XP 2000. Really nice...:) PS: if anybody here has the knowledge, please make a modification to take images directly from memory (eliminating Disk Reading) and test it,please. A note to any programmer: With this video things always remember to pay special attention to the CPU cache.Avoid cache misses as much as posible.

July 24th, 2004, 07:49 AM	#36
Wayne Morellini Inner Circle Join Date: May 2003 Location: Australia Posts: 2,762	<<<-- Originally posted by Ben Syverson : Show me -- I want to see spec sheets. -->>> OK Just finished my search, and I must applogise, I was wrong, there are no 3D cards with DVI input (yet) that I can find. I put this down to a mistake in reading an all in wonder 8500 reveiw a few years ago. Even though I found ads advertising DVI input on some 3D cards, it looks like these are all mistakes. I did find some HD cards that had DVI Inputs, one expressly to recieve a signal from a HD receiver, but these are not the low priced commodity cards we would need to look for. But still it was only a suggestion for camera manufactures to research and consider, and still is a cheap option, but when commodity cards/mainboards come out with the inputs. (I did also find 10bit DVI reference, and you can also pack pixels for greater than 10bits). I expect that this will happen sometime. Apparent not right DVI in: http://www.pricewatch.com/1/264/3234-1.htm http://www5.sharkyextreme.com/hardwa...211_1369251__3 Expert advice on DVI-in (if you sign up??) http://www.experts-exchange.com/Hard..._21039171.html Two actual solutions: http://www.cellarcinemas.com/cgi-bin/store/HD3.html http://www.primagraphics.co.uk/ds/pumapci.html Sorry Wayne.

July 24th, 2004, 08:21 AM	#37
Wayne Morellini Inner Circle Join Date: May 2003 Location: Australia Posts: 2,762	Re-edited, actual performance figures incorporated. Jaun, that is close to my estimates I did a longtime ago on either the viper, or 10 bit thread. I think I actually came closer to 1600Mhz, which is closer to the actual clock speed the 2000xp runs at. With use of mmx, SSE etc (they recon Huffy doesn't use) maybe we can bring that closer to 1ghz. This sounds strange though, because I remember a link that had huffy doing 38MB/s on a celeron 416, so 1.6Ghz should be more than enough for a single chip 720p. I think that there was a link to a even faster one. It would be good If we could find 3-5:1 true lossless, for not much more processing power. People seemed to get interested in wavelets, as a solution for both lossless, near lossless, and high compression ratios (I think they are right), but we can eventaully use any plugin comrpession codec we want (hopefully). People on these forums have a tendacy to not to reply to issues previously discussed (except for me). It canbe dissapionting to ask a question, or make a suggestion, and not get an answer. Unfortunately this means reading long threads first. Good programming suggestions Jaun, I've been in contact with the Rob's on these issues for a while, and did a post a while back. My estimates on I ghz VIA being enough for single chip 720p raw, is also because Rob is reporting 60FPS for his machine in his development blog at the wiki since he swapped to machine code routines. The Windows C slow down seems too much, but what do I know. Have fun. Wayne.

July 24th, 2004, 10:49 AM	#38
Juan M. M. Fiebelkorn Major Player Join Date: Jun 2004 Location: Buenos Aires , Argentina Posts: 444	Wayne, Huffyuv uses mmx, what it lacks is SSE, which would be better for processing 16 bits.. If you read my post again carefully, may be the 38 MB issue is true but when capturing. When in capture mode you are not reading and writing from disk at the same time (which is my situation).. About wavelets, I'm working on that.But remember that for a lossless compression you can use DCT which is faster and gives more or less the same gains (always around 4:1 with my tests)

July 24th, 2004, 12:01 PM	#39
Wayne Morellini Inner Circle Join Date: May 2003 Location: Australia Posts: 2,762	4:1 is perfect compared to 2:1 that huffy simular compressions get. Sorry, about the mmx mixup, I thought I read it didn't. If this is the case then potential speedups shouldn't be that dramatic from using SSE. I am surprised that there is that much of a difference between writing to memory and writing to disk, there must be something wrong, modern disk writing should not be very intesive compared to Huffy compression. The routine I saw is only 14.7k long, that is so small that if it is in C, you probably could do a fast machine code version in 8K. http://www.divx-digest.com/software/huffyuv.html Issues for people in general on the threads (not for you Jaun): Still, the issue that people don't realise (not you Jaun) is just how much how your system is setup will affect how fast your code works (potentially dramactically). Customising code to the particular processor being used can also speed up the code). Aswell as this, C uses only a limited machine code instruction set in compile, and the c routine stack and OS calls can slow things down. Improvements canbe made using machine code to access all the machine code instructions and streamline the code. This canbe dramtic improvements. People shouldn't trust MS, OS, or C compiler to think for them and do the right thing. If game programmers did this we would be struggling to run anything better than Unreal Tournament. So if anybody runs something on their system and it doesn't perform well enough they shouldn't assume that is it, it could work much faster, that is what people like Rob are exploring for us. I am wondering if I should not just get a cheap USB camera, rather than waiting around.

July 24th, 2004, 12:08 PM	#40
Juan M. M. Fiebelkorn Major Player Join Date: Jun 2004 Location: Buenos Aires , Argentina Posts: 444	RAM has a bandwith of around 6.4 GigaBytes/s against a normal disk, 7200 rpm which writes no more than 30 MegaBytes/s. You can have a compression routine that outputs far more than what you can record on disk, thus limiting your output....

July 24th, 2004, 12:59 PM	#41
Wayne Morellini Inner Circle Join Date: May 2003 Location: Australia Posts: 2,762	Modern disk systems work with handeling electronics on each end (IDE Integrated Drive Electronics, DMA transfers etc). So this hardware can block download to disk electoriics which can then dribble the informatyion ou to the disk. You can write to memory, send a request to the hardware system to pickup the information and write it out via DMA, while your program perfoms more compression. But what we are talking about is also passing the output through memory to the comrpession program (replacing raw disk read with half as many comrpessed reads). Leaving out all the stuff about buffers etc to optimise this process. So you get the request, and you get the a number of cycles per 64bit/s (preferably 2 cycles). We have determine the max sustain data rate for a really good disk is something like 50mb/s, we also are looking at raid. In old disk system you had to do it all manually as you said.

July 24th, 2004, 01:02 PM	#42
Juan M. M. Fiebelkorn Major Player Join Date: Jun 2004 Location: Buenos Aires , Argentina Posts: 444	What disk is that?? not a 7200 rpm one for sure. The fastest ATA disk in the world is the Western Digital Raptor SATA 72 MB/s No other ATA disk (SATA or PATA) writes more... I don't get the rest of your post, sorry.

July 24th, 2004, 01:14 PM	#43
Wayne Morellini Inner Circle Join Date: May 2003 Location: Australia Posts: 2,762	Let's do something useful. I was wondering what is the cheapest cameralink frame grabber out there. I did a google search and came up with a mass of hits, there is a range of cameras and framegrabbers out there. So I found the cameralink standards homepage, and product links. In this I also found small industrial capture computers running embedded windows. I am very tired and haven't done much pricing yet, but I am posting it here so you guys can have a look. Maybe we are barking up the wrong tree, it is so prolific maybe there are ready made systems and capture software for pro video capture out there allready, that we might be able to get volume discount on. I don't remember if anybody has looked at this site yet. Cameralink http://www.machinevisiononline.org/p...ex.cfm?cat=129 Registered Cameralink products http://www.machinevisiononline.org/p...ls.cfm?id=1108 Gigbe http://www.machinevisiononline.org/p...ex.cfm?cat=167 Product Search http://www.machinevisiononline.org/buyers_guide/

July 24th, 2004, 01:26 PM	#44
Wayne Morellini Inner Circle Join Date: May 2003 Location: Australia Posts: 2,762	We discussed a raid system that sustained around 200MB/s with four of the drives, in the threads. You can search them easily, by selecting the most posts per page in your site preferences and using ctrl-F, to search each page for "raid". When you said raptor before I thought you meant one of the HD video raid systems I've heard of, sorry, but 72mb/s sustianed would be great. The rest of the stuff I mentioned is just how modern drive electronics makes it possible to transfer 8 bytes at a time (64bit memory bus) for around 2-3 memory cycles. If your in cache most of the time than the amount of extra processing time taken up by the disk can be minimal (of course it is a bit more complex and the processor is also accessing the capture card and writting to memory and that stuffs up the free flow of memory and can slow it down a bit). But still the drive write time should not effect the processor too much (except it has to wait for the drive) if done rightly. Thanks Wayne.

July 24th, 2004, 02:25 PM	#45
Juan M. M. Fiebelkorn Major Player Join Date: Jun 2004 Location: Buenos Aires , Argentina Posts: 444	Wayne, no RAID system with normal disks will give you 200 MB/s if yopu use normal disks ( I mean 4 normal disks). It would give you 200 MB/s if they were 4 Raptor or $ SCSI 10,000 rpm disks. Please believe me.