Mark, given the CF interface, what would you record uncompressed HD stream to? We would have to wait for a CF device capable of sustained 120-150 MBytes/second, right?

after reading Piotr's posts I'm very curious to go out and watch my stuff on some really big TV's.

Mark, try recording some small water geyers at a nearby park for 10 seconds each at bitrates from 18-280 MBps, plugin them into to a TV via the HDMI port and sit back with one of those potent 9-10% ales, such as Sierra Nevada Bigfoot or Dogfish 90 Minute Pale Ale.

Mark, given the CF interface, what would you record uncompressed HD stream to? We would have to wait for a CF device capable of sustained 120-150 MBytes/second, right?...Hey Gints:
Not necessary. Try Raid 0. Problem solved.

Mark, try recording some small water geyers at a nearby park for 10 seconds each at bitrates from 18-280 MBps, plugin them into to a TV via the HDMI port and sit back with one of those potent 9-10% ales, such as Sierra Nevada Bigfoot or Dogfish 90 Minute Pale Ale.....Hey Gints:
I'll better that ! ;-) Try drinking 16 cups of fresh bean Moca Java slow percolated and sit down and watch the same playback. By then end of the playback you will understand the full meaning of 2001:A Space Odessy ;-) !

Mark, the most I've ever done is an Octo Espresso, which is two Quattros. How would you even order sixteen? "I'd like a HexaDeco Espresso, please?"

Not necessary. Try Raid 0. Problem solved.

How would the XDR or Nano record these? My Nano only has CF slots for recording, and I don't see the appropriate connectors for external storage in the XDR pictures.

Dear Mark,

We like to be very open with our communications.

However, our upper recording bit-rate, what is actually possible, is something that we prefer not to divulge for competitive reasons.

Dear Mark,

We like to be very open with our communications.

However, our upper recording bit-rate, what is actually possible, is something that we prefer not to divulge for competitive reasons.Dear Dan:
I was never fishing for private company internal information of this nature about CD's products, I was only requesting a higher data rate option for I-Frame recording for special high end digital cinema or special effects shooting. I don't understand how requesting 300 Mbps or 320 Mbps could trespass on this. Sorry.

How would the XDR or Nano record these? My Nano only has CF slots for recording, and I don't see the appropriate connectors for external storage in the XDR pictures. Hey Gints: No, no - You don't require any external storage for this feat. I'm not sure the Nano Flash can even do uncompressed, but I think the old XDR is built slightly differently internally (But I really do NOT know ???) You Raid 0 all 4 cards in an XDR or 2 cards in a Nano (If possible- I do NOT know). You use those newer 64 GB CF Cards which can do 60 and 90 MB per second playback from Sandisk (Forget their name - sorry), and you record in a striped configuration of 2 or 4 cards (Should be end user choice as option in menu). You shoot, but do NOT remove the striped CF card media from the Nano or XDR and you either....

1. Playout in realtime via HD-SDI from Nano or XDR.
2. Transfer files from XDR/Nano to computer via built in FireWire (Slow but do-able - Faster CF card media will transfer in Raid 0 config faster anyway)

You bypass the internal encoding engine with this option, thus you are inherently now up to 10 bits from 8.
* You could even stripe for *Compressed MPEG 2 recording ! Why not ? Then you can really boost your I-Frame encoding data rate

Sandisk Extreme does 60 MBytes/second write while Sandisk Extreme Pro does 90 MBytes/second . Now that I'm moving to 200 Mbps I-frame only, my wallet will become friends with the only fast 64 GB card Sandisk makes. I bought two spanking new 32 GByte Sandisk extreme cards eight months ago. How many times have I "re-promised" that I will never buy another $400+ flash card again ? Hey wait, there's no swearing at work.

1. Playout in realtime via HD-SDI from Nano or XDR.
2. Transfer files from XDR/Nano to computer via built in FireWire (Slow but do-able - Faster CF card media will transfer in Raid 0 config faster anyway)


CD would provide utility to interleave the files after specifying two directories from the CF cards that have been copied to your computer.

Hey Gints:
I have a hodge - podge of those Sandisk cards myself. I have 2 x 32 GB Sandisk Extreme III's, 1 x 16 GB Sandisk Extreme IV, and one 8 GB Sandisk Extreme III. I shove them all into my XDR and I can even record at 280 Mbps I-Frame across all 4 cards. The only eccentricity is that the fastest card (16 GB Extreme IV @ 45 MB per sec) has to be in the first slot, or else XDR will automatically down set everything to 220 Mbps.

It's good some humor, and relaxed comments, have been introduced to this somewhat delicate (and thus, tense) thread :)

But back to the topic: after Alister's comments, I don't feel isolated any more in my assessment of things, so I will tell you this (I wonder how many of you guys will agree):

- the EX1 is indeed inherently noisy; in fact (contrary to the generally adopted belief) this inherent noise is much more dependent on the scene nature than the level of lighting - in fact, many scenes shoot with gain at -3dB suffer from the shimmering more than some low-light scenes, shoot at +3dB, or even higher gain!

- unfortunately, nanoFlash is not the holy grail for curing this; at low bitrates (and I-Fo especially) it can be a complete mess, while at the highest bitrates (and in L-GoP especially) it augments this noise. I believe it's fair to make all the current and potential users aware of that.

Also, while I am sceptical about the nanoFlash potential to record with bitrates significantly higher than it's capable of currently (it's just not been designed to do so), I'm very much for the idea of Convergent Design considering to use the Sony encoder chip in the "constant quality, automatically variable max bitrate" mode (if at all possible).

What do you say, Dan and Mike?

I'd also suggest removing the options for I-Fo below 180 Mbps from the menu of available recording formats.

Dear Friends,

We love to be responsive to customer suggestions and implement them if possible.

We may be able to implement VBR (Variable Bit-Rate). However, we have to be cautious, in that the files have to work in the various NLE's.

(It would be best if the VBR files work in every NLE. If not, then we would be creating a "trap" in which a user could select an option, but then his or her files would not work in their NLE.)

We are trying to keep our menu options limited, so that the nanoFlash is not complicated to use or intimidating to first time users.

(But, this is not an issue, if all NLE's support VBR files).

We could use some information.

For example, if we just change our files to VBR from CBR (Constant Bit Rate), would these files work in your NLE's?

Please let us know which type of files will work in VBR mode.

Dear Dan,

Some CBR formats must be left for sure (e.g. the L-GoP, XDCAM HD 50/422), to ensure the widest NLE compatibility with the well established formats (there are sure more of them).

As to other formats (especially I-Fo), my knowledge is not deep enough - let's wait for our gurus' comments.

Thanks anyway, for your instant response!

Piotr

Dear Friends,

We love to be responsive to customer suggestions and implement them if possible.

We may be able to implement VBR (Variable Bit-Rate). However, we have to be cautious, in that the files have to work in the various NLE's.

(It would be best if the VBR files work in every NLE. If not, then we would be creating a "trap" in which a user could select an option, but then his or her files would not work in their NLE.)....Hi Dan: I don't understand your premise here. VBR only affects a file in the way it is encoded, but has nothing to do with the finally completed file being manipulated in any NLE.

are trying to keep our menu options limited, so that the nanoFlash is not complicated to use or intimidating to first time users....How would this complicate the menu system ?

But, this is not an issue, if all NLE's support VBR files).

We could use some information.

For example, if we just change our files to VBR from CBR (Constant Bit Rate), would these files work in your NLE's?

Please let us know which type of files will work in VBR mode....Dan, I think this needs to be additive and not one OR the other.

Dear Mark,

For Final Cut Pro, our files are identified as Sony XDCam 4:2:2 50 Mbps CBR, for our bit-rates 50 Mbps or higher.

I wonder if we encode the files as VBR and then indicate that they are VBR, whether the NLE's will recognize the files?

Or, whether if we encode in VBR, and specify that they are CBR, if the NLE's willl work with the files?

In response to your post: If we add another menu item, and there is a chance that it could be set wrong, then we are complicating things by adding the new menu item.

We will be discussing this.

Dan, I'm willing to test the VBR files for the Final Cut Pro, Sony Vegas 9 and provide comments with convenience software such as VLC Media Player, Windows Media Player, Western Digital TV and Sony PS3.

After stepping through my own and Piotr's footage, I'm biased against using 140+ Mbps LongGOP due to the quality drop from the I-frame, and the noise from that variation. There is a visible increase in quality almost every 6 frames when viewed on my Apple Cinema 23" display.

CD is creative enough to provide a default for Quality variation parameter. A single parameter could control both B-frame and P-frame "spoilage".

- the EX1 is indeed inherently noisy; in fact (contrary to the generally adopted belief) this inherent noise is much more dependent on the scene nature than the level of lighting - in fact, many scenes shoot with gain at -3dB suffer from the shimmering more than some low-light scenes, shoot at +3dB, or even higher gain!


I'm not sure if you're implying that the Sony noise level is less at Gain=-3dB, but there are a number of posts on the Sony XDCAM group that -3dB does not provide a higher SNR - only a signal attenuated by 3 dB.

Personally, I always use 0 dB gain. That is what comes out of the hardware, and I do not feel like messing with it. At least not until post.

Hi Gints & Adam:
In my visual testing & experimentation, I have noticed a slight decrease in overall noise if I shoot exteriors at -3 db gain.

(It would be best if the VBR files work in every NLE. If not, then we would be creating a "trap" in which a user could select an option, but then his or her files would not work in their NLE.)


This should be up to the user. We already have the trap of shimmering MPEG at 140-180 Mbps. I want 140 or 180 Mbps LongGOP VBR exceed the quality of 220-280 Mbps I-Frame only. Besides, after reckoning with the Force of the Flash, NLE's will be scrambling to add new Nano/XDR file compatibility to their editing suites ASAP. The Nano is unleashing a slew of scalable recording options.

It wasn't too long ago (pre-2008) that nearly all major news networks would refuse downloading my DV (.avi) and HDV (.m2t) files and instead insisted on my writing the clips/concatenations to tape to be sent via Overnight Fedex. Some then lost 1/2 to a full day sending my tapes to a conversion house so that they can be written to yet another tape.

If not, then we would be creating a "trap" in which a user could select an option, but then his or her files would not work in their NLE.

And since the user is most likely a pro, he would much rather have the "trappy" option available than be treated like someone who does not needs to be protected from himself.

I mean how is it different from having manual focus available? We know what we are doing, and when there is something we do not know, we are willing to learn it. If we were scared of such traps, we would be using $130 camcorders.

Dear Fiends,

My comment was not meant to be condescending.

My comment was directed to the fact that we want our nanoFlash to be a reliable recording device.

I hope everyone will realize that many crews do not own a nanoFlash, and they receive it the morning of the shoot, or the day before. Quite often they are not in a position to record files and test them in their NLE and workflow. Others, of course, have time to learn the nanoFlash.

Of course we do not want to "dumb down" the nanoFlash.

In an ideal world, we would just test VBR, and it would work, be better, and be accepted in every NLE without problems. We have not run these tests yet.

Another thing that we have considered is adding an "Advanced Settings" menu area, and a way to quickly set all "Advanced Settings" to the defaults.

We are also working on Profiles which will help in this area.

The current standard for Sony XDCam footage is 50 Mbps 4:2:2 CBR. This is what is accepted by the NLE's that support XDCAM footage.

CS5 officially supports other bit rates that we offer.

The Mainconcept Codec for Adobe CS3 and CS4 adds support for all of our modes and bit-rates and we greatly appreciate their efforts to support our products.

Final Cut Pro and others only suppport Sony XDCam footage at 50 Mbps 4:2:2 CBR.
Thus, we label our higher bit-rate footage, such as our 280 Mbps footage, as 50 Mbps so that FCP will recognize it.

Our engineers feel that we we cannot label VBR footage as CBR in the file header. They feel that the footage will not work if we do. We, at this moment, have not run this test.

In our next release, our MPG footage will be VBR. We just made that change.

I'm not sure if you're implying that the Sony noise level is less at Gain=-3dB, but there are a number of posts on the Sony XDCAM group that -3dB does not provide a higher SNR - only a signal attenuated by 3 dB.

That's what I mean, Gints - sorry if was unclear.

Another thing that we have considered is adding an "Advanced Settings" menu area, and a way to quickly set all "Advanced Settings" to the defaults.


Sounds like a good idea. Or you could offer an optional VBR firmware update that is not recommended for rental units along with an explanation. That way a pro who wanted VBRs could choose to download the VBR enabled firmware and would be well aware of the consequences.

That would protect the naive and enable the wise.

Dear Mike,

Thank you for your suggestion.

Sounds like a good idea. Or you could offer an optional VBR firmware update that is not recommended for rental units along with an explanation. That way a pro who wanted VBRs could choose to download the VBR enabled firmware and would be well aware of the consequences.

That would protect the naive and enable the wise.

I'll second that.
I suspect many here do not understand what it can be like for a rental company.

Dear Fiends,
In our next release, our MPG footage will be VBR. We just made that change....Hi Dan: Could we also have that advanced menu and VBR *Option* ported to the XDR ?

Dear Mark,

Yes.

I need to clarify. We have three file types: MOV, MXF, and MPG.

We have enabled VBR for MPG only.

I doubt that you are using MPG, but maybe you are.

Hi Dan:
Yes, I do use .MPG feature (Which was busted for playback in the XDR BTW - I haven't tested that long duration .MPG playback lately). I use .MPG recording to make quick industrial Kiosk Blu-ray DVD's. It works *very well btw.

I need to clarify. We have three file types: MOV, MXF, and MPG.

We have enabled VBR for MPG only.


Dan, the interest in Constant Quality MPEG2 and VBR is heating up. So, you already provide VBR for MPG (I haven't used that) recording ?

In our next release, our MPG footage will be VBR. We just made that change.

Missed this gem. Really ? I'm willing to test the alpha/beta/whatever.

Missed this gem. Really ? I'm willing to test the alpha/beta/whatever....Hey Gints: I thought the point of this exercise would be to offer the *option for us end users to be able to engage VBR encoding when shooting Long GOP and I - Frame ? In my opinion, this is what I think needs to happen. (Although it's always great to have this on .MPG setting, but let's be real - How many of us will do any real big work in .MPG setting ?)

Mark, agreed. I'm just anxious to find out if a VBR implementation will reduce the LongGOP shimmering that started this thread.

Hello Dan,
I agree with Mark VBR and CBR selection surely has to be the next feature that is enabled on the NanoFlash/XDR. I have used other Sony MPEG encoder boards in the past which have offered the selection of VBR or CBR and I found that I used the VBR option all time tailoring file size vs quality.

The only point of VBR is to optimize the files size. If you are not constrained about file size, VBR adds nothing. Aside of this, a proper VBR compression needs a multi-pass process.
Better than VBR would be to implement Open or Short GOPs (I don't know if is possible with this processor and in a RT process).
rafael
PS: If people is having problem choosing a proper data rate in CBR, wait for them to decide the Average and peak data rates for a VBR compression.

Dear Lance,

When you create VBR files, what NLE's are you using to read the files?

Could you please provide more info?

Bit-Rate, Long-GOP or I-Frame Only, 4:2:0 or 4:2:2, and what problems have you had reading the files?

The only point of VBR is to optimize the files size. If you are not constrained about file size, VBR adds nothing. Aside of this, a proper VBR compression needs a multi-pass process.
Better than VBR would be to implement Open or Short GOPs (I don't know if is possible with this processor and in a RT process).
rafael
PS: If people is having problem choosing a proper data rate in CBR, wait for them to decide the Average and peak data rates for a VBR compression.

AFAIK, single-pass VBR also has its advantages (I'm not sure how much less efficient datarate "distribution" is in this case, but the file size can be decreased for sure). But true - the most efficient is multiple pass, whereby one pass is needed to analyze the video quality needs, and another one to encode those demanding parts of it (motion etc.) at a higher, and the remaining part - at a lower bitrate, just to minimize the output file size.

As to the potential improvement in L-GoP quality arising from using shorter and more sophisticated GoP structure, I was mentioning this many times earlier in this thread.

I wouldn't worry about the added complexity - if some settings do indeed bring much better PQ than others, people would learn very fast how to use them ...

My interest in VBR is control of image quality variation from frame to frame. The Sony XDCAM 140 Mbps has a problem with quality variation, and we hope that VBR will address this problem . Of course, this will also reduce file size if I'm able to use 140 Mbps VBR to vary between 120-180 Mbps and yield the quality of 280 Mbps I-Frame only.

Gints, I hear what you're saying. In fact, we have both noticed and analyzed this quality variation that causes the noise shimmer, making it more distracting. BUT, I am not so sure VBR can take care of it - please try to convince me, and than we could try together and ask CD to try and implement it :).

What I - with my current, limited knowledge - believe to be a better cure for the intra-frame inconsistency, is some fine-tuning of the GoP length and structure. There's a number of parameters controlling this - what I'd expect CD to do is at least let us know which of them are available for their code to be used on the Sony encoding chip.

Who knows - perhaps the two approaches can complement each other?

I'd say the strategy should be:

- leave the I-Fo as is, but limit it to some really high bitrates where it shines (220 and up)
- try to improve the L-GoP by either optimizing the GoP structure/length, or adding VBR encodig, or both.

What do you say, Dan?

In an ideal world, we would just test VBR, and it would work, be better, and be accepted in every NLE without problems. We have not run these tests yet.

Dear Dan,

Of course my knowledge is limited, but frankly I don't see reasons why NLE should not recognized the VBR L-GoP files if nanoFlash had an option to record in this format.

With Vegas, one can encode either a CBR or a VBR MPEG-2 file from any source in the timeline, and such a file is always recognized, read-in, and played back in Vegas. Just tried it with Edius - same story.

Dear Dan,

Of course my knowledge is limited, but frankly I don't see reasons why NLE should not recognized the VBR L-GoP files if nanoFlash had an option to record in this format.

With Vegas, one can encode either a CBR or a VBR MPEG-2 file from any source in the timeline, and such a file is always recognized, read-in, and played back in Vegas. Just tried it with Edius - same story. Hi Piotr & Gints: Once a Sony XDCAM HD 4:2:2 MPEG 2 file is created, then the NLE should treat it as any other completed and ready to play XDCAM HD file. How the file is made makes no difference as long as the finished XDCAM HD file meets the editing specification of the NLE - In the case of Avid Media Composer, the known standard is Long GOP 50 Mbps or I-Frame at any data rate you please.

* As long as the file header information remains in a form the NLE was previously able to read, then, in theory, there shouldn't be any issues.

It would be good to find a manual on the Sony XDCAM encoder, and admittedly, I don't know what I'm searching for. I see a qpweighting parameter mentioned in the AVC encoder specs. I wonder if this is the parameter for setting the relative quality of the I, B and P frames which are often specified as IQSCALE, PQSCALE and BQSCALE in MPEG encoder parameter files :

http://www.sonic.com/images/products/sonyavcencoder/sonyavc_parameters.jpg

Piotr,

Followed your thread only incidently, but was triggered by it in the first place because I also wonderd why at some circunstances noice was moving through my images like it was 'snowing'. Specially at lower light circumstances. I did not fully check the whole thread now, and discussion seems sometimes to move towards other directions. However, I have been playing with my gamma settings and noticed that this terrible noise out of the EX3 into the nano was very pronounced with gamma STD3 (which I used before), and is almost not there with CINE2. Maybe basic knowledge you know since long, but a revelation for me.

Cees, of course some EX's PP settings are better than others as far as noise is concerned. Usually however, no "best" setting exists - it's always a trade-off.

Let me stress it however that this thread is not about how noisy the EX cameras can be (some even say they are not that noisy at all, considering chip size and price tag). What we're trying to find out here is the best setting for the nanoFlash (or even modifying the way it encodes, if possible and viable to Convergent Design), so that the advantages of this marvellous device always outweigh the main disadvantage: the tendency to augment the noise, by exaggerating its shimmering nature.

Everybody participating in this thread realize perfectly that camera noise cannot be eliminated by the nanoFlash, as the device cannot tell noise from detail, and enhancing detail is one of its very purposes (along with minimizing compression artifacts). However, we've achieved some consensus on that the problem is aggravated by the L-GoP-inherent, frame quality fluctuation. Thus, minimizing this fluctuation has become the goal of our debate here :)

Thanks for your input, anyway. If in response to it, I have taken the liberty to summarize the essence of this (somehat convoluted) thread , isn't because I'm patronizing on you at all (please forgive me if it sounds that way). I simply wanted to help those new ones lurking into it, to understand what is being talked here...

:)

Piotr

Hi Piotr & Gints: Once a Sony XDCAM HD 4:2:2 MPEG 2 file is created, then the NLE should treat it as any other completed and ready to play XDCAM HD file. How the file is made makes no difference as long as the finished XDCAM HD file meets the editing specification of the NLE

* As long as the file header information remains in a form the NLE was previously able to read, then, in theory, there shouldn't be any issues.
By definition XDCAM HD 422 is CBR.
rafael

By definition XDCAM HD 422 is CBR.
rafaelHi Rafael: Yes I understood this was so, all I was wondering if there is room in the Sony XDCAM HD specification for the utilization of a VBR encoding scheme. Does anyone know or have a link to any possible web page which would display the full Sony XDCAM HD 4:2:2 spec so we could check ?

Sony's pro video british site has a lot of detailed docs relating to their pro gear. I would start there if you are looking for XDCAM specification related material.

Hi Andrew:
OK. Will do :-)