Noise comparison: 35/4:2:0 vs. 180/4:2:2

[Cees van Kempen]

Piotr, it is an EX3 I use.

Dan, Indeed I won't have access to AC power. I do have extra external batteries with my nextos. If it turns out I can keep up with charging my batteries (nextos, EX3 batteries, SLR equipment), I will use 100Mbps. As soon as power consumption seems to go faster than recharging all the batteries (I will use portable solar panel and the outboard engine of our boat), I will move to 50 Mbps LongGop, at least knowing this still is broadcast quality. So I have to care less about the nexto batteries. There will not be a lot of camera movement, not a lot of movement in the image (if the tribal dancing starts I will change to higher bitrates). Excessive detail may occur when shooting close ups of birds (feathers).

It is indeed the kind of challenge that makes me very happy with my nano!!

Thanks to you both for your prompt reply.

Cees

[Gints Klimanis]

Originally Posted by Piotr Wozniacki

- for L-GoP: 100 Mbps max if you're planning extensive post-processing, 50 Mbps if you're just cutting
- for I-Frame Only: as high as possible (i.e. 280 Mbps if your cards allow, otherwise 220 minimum).

The reason being that the EX series cameras are indeed noisy, and L-GoP at bitrates higher than 50 Mbps actually augment this noise (or strictly speaking, do not mask it enough, like the more compressed, i.e. lower bitrate, formats do).

Piotr, You started this thread and the lengthy discussion. The above does not represent the entire issue, as you already know, but I'll write it again. LongGOP above 100 MBps has excessive variation between I and B-frames, and this variation grows with increasing bitrate resulting in nearly no benefit between 100 MBps and 180 Mbps LongGOP. This detail variation is seen as shimmering seen in image including detailed subjects as grass or wet sand - not just noise from Sony EX cameras. I-Frame Only solves the inter-frame problem at 220+ MBps, but I suspect this is an I/B frame quality balancing issue in LongGOP more than inadequate bitrate.

[Piotr Wozniacki]

Originally Posted by Gints Klimanis

Piotr, You started this thread and the lengthy discussion. The above does not represent the entire issue, as you already know, but I'll write it again. LongGOP above 100 MBps has excessive variation between I and B-frames, and this variation grows with increasing bitrate resulting in nearly no benefit between 100 MBps and 180 Mbps LongGOP. This detail variation is seen as shimmering seen in image including detailed subjects as grass or wet sand - not just noise from Sony EX cameras. I-Frame Only solves the inter-frame problem at 220+ MBps, but I suspect this is an I/B frame quality balancing issue in LongGOP more than inadequate bitrate.

But Gints, I fully agree - I never backed out of this conclusion of ours. However, considering CD does not confirm it (and obviously, isn't going to do anything about L-GoP fine tuning), to questions like that of Cees I am answering with what the work-around is, rather than beat the dead horse again and again.

Have you noticed, Gints, the we are the only two persons in this thread who can see the obvious imperfections in the nanoFlash L-GoP structure? Well - 2 users is not enough to make CD admit it, not to mention fix it, for us :(

[Dan Keaton]

Originally Posted by Piotr Wozniacki

But Gints, I fully agree - I never backed out of this conclusion of ours. However, considering CD does not confirm it (and obviously, isn't going to do anything about L-GoP fine tuning), to questions like that of Cees I am answering with what the work-around is, rather than beat the dead horse again and again.

Have you noticed, Gints, the we are the only two persons in this thread who can see the obvious imperfections in the nanoFlash L-GoP structure? Well - 2 users is not enough to make CD admit it, not to mention fix it, for us :(

Dear Piotr,

I am going to step out of character and respond to your accusations:

On Post 165 of this thread I posted:

Dear Piotr,

The Sony Long-GOP consists of I Frames, B-Frames, and P-Frames.

They are not all created equal in terms of quality.

If you want the highest quality, we recommend 220 Mbps or 280 Mbps I-Frames Only.

Our 280 Mbps I-Frame Only has been thoroughly tested by a major network for one of their high production value prime-time shows using the most sophisticated of test equipment.

The quality met their specifications and needs.

As I calculate it, you are looking at two frames, zoomed in 310%.

Please feel free to ship us the DVD, if you wish.

Thus your comment that Convergent Design "Does not confirm it" is false.


In Post 1 of this thread you stated:

...Now, I have deliberately chosen a back-lit scene like this, and a picture profile not designed to minimize the chroma noise, so that the results are more readily visible. But frankly, I never suspected the difference would be so great, and not in favour of the nanoFlash...

And you have continued presenting images throughout this thread where your camera or lighting conditions were not optimized to produce a quality image, and then faulted the nanoFlash for not removing the noise from the images.

Then, after 25 pages of posts, on Post 361, you posted:

Ron,

Assuming you're shooting with an EX1/R/3, yes - that's right.

However it all depends on the actual S/N of your source camera. For instance, I've just run a series of new experiments - and with PP OFF (yes - not just DETAIL, but also the matrix, etc at the factory settings), even with my EX1 as the source, the 100 Mbps nanoFlash L-GoP looks gorgeous!

Well - no wonder; just the GIGO formula at work... The goal now being how to possibly tune the L-GoP nanoFlash encoding so that it does NOT augment the noise.

Piotr

Previously, I asked for details on how you were creating these images and you refused.

Others advised you to turn off your picture profile.

In your most recent post you state:

(and obviously, isn't going to do anything about L-GoP fine tuning)

I am going to politely point out that you are not fully aware of our internal developmental plans, nor do you fully know what we are currently working on, nor are knowledgeable about what we are going to do next.

Your accusations are unfounded, unwarranted, and unwelcome.

[Piotr Wozniacki]

Dear Dan,

I'm very sorry that - unintentionally - I made you "step out of character". Really, didn't mean to...

But I must defend myself, as I don't want anyone following this thread to think that what I have been writing in it, has been aimed against CD or you personally.

So, first of all, I must say I got very sad reading that "my accusations are unfounded, unwarranted, and unwelcome". The reasons you made me so sad are:

1. First of all: nothing written in this thread has been meant as "accusations".

2. Secondly, even though in our latest e-mail exchange you agreed that my footage at 180 Mbps

shows defects in the image

- at the same time you kept blaming the lighting conditions and other factors, not related to how nanoFlash encodes L-GoP. Even though the very same scene I sent you recorded at 50 Mbps was clean!

This made me believe CD is not acknowledging the issue as described by myself and Gints in this thread.

So, even though indeed, as you put it:

"I am not fully aware of CD's internal developmental plans, nor do I fully know what CD are currently working on, nor am knowledgeable about what CD are going to do next"

- I simply thought the message from you was clear that the excessive noise has been my camera fault. If however CD is seeing the problem, and plan to take a closer look at it - this is great news, and I'll be the first to thank you for this extra effort.

3. Last but not least, what makes me sad is that you didn't appreciate the fact I moved this discussion from the public thread to our private email exchange, in order to avoid anything that could be perceived as criticism towards Convergent Desing, and/or the nanoFlash. As a loyal customer, I though it was important in the specific situation we're facing, with the challenge from CD's competitors... In one of my emails, I wrote:

"I hope you agree that - especially with the current challenge by the competition - it's important fro Convergent Design that my lengthy thread on DVInfo was concluded with some constructive outcome...

By "constructive outcome", I meant some well grounded consensus between users like myself and Gints, and Convergent Design, on the nature of the issue and how it can be resolved or worked around.


If - even with the above clarification - you still feel personally offended, please accept my sincere apologies.

Piotr

[Rafael Amador]

This has been a very interesting and productive thread.
After almost 400 posts, my conclusion is that would be desirable to try to fine tune the LGOP structure of the NANO.
CD has been making an incredible effort trying to give to the NANO all the operational options needed for a wide range of cameras and NLEs and for a variety of scenarios and jobs.
Once those operational needs are satisfied, it would be great if we could pay a bit of attention yo the NANO encoding.
We know how to improve the MPEG-2 compression, but we don't know if the NANO meet the options to do so.
In the end what we want is the NANO to be the best 8b recording solution.
With most affordable cameras to come outputting 8b Uncompress, the NANO will still being the perfect recording companion.
Best,
Rafael

[Cees van Kempen]

To finalize the question I posted in this thread. Yes: I have wishes that I would like to be optimized on the nano. And yes, I find it ridiculously valuable as it is. IT IS GREAT DEVICE. And so is the support from CD, answering my questions early mornings, late at night and during the whole of the weekend. Never experienced anything like that before. I am sure Poitr also realizes that. He himself is by the way also very helpful at any moment and obviously having very valuable knowledge. So both of you, I learned a lot this weekend. Thanks a lot!

Cees

[Piotr Wozniacki]

Originally Posted by Cees van Kempen

IT IS GREAT DEVICE.

Of course it is, Cees - mine is permanently attached to the camera (using Olof's plate and bracket), and I never shoot without it!

Originally Posted by Cees van Kempen

And so is the support from CD, answering my questions early mornings, late at night and during the whole of the weekend. Never experienced anything like that before. I am sure Poitr also realizes that.

Yes, I do realize that, and have always been very grateful to CD and Dan in particular...

Cheers,

Piotr

[Gints Klimanis]

Originally Posted by Piotr Wozniacki

Have you noticed, Gints, the we are the only two persons in this thread who can see the obvious imperfections in the nanoFlash L-GoP structure? Well - 2 users is not enough to make CD admit it, not to mention fix it, for us :(

This is probably because we represent the operational cost-conscious segment of CD's market, and I hope our comments make a difference. I thought 80 GBytes/hour of footage was a lot at 140 Mbps (shoot SxS 35 Mbps in parallel.) , and this footage needs at least two backups. The rest of CDs customers will just use 220-280 MBps.

[Gints Klimanis]

Originally Posted by Piotr Wozniacki

However, considering CD does not confirm it (and obviously, isn't going to do anything about L-GoP fine tuning),

There is a possibility that CD may not be able to do anything about it as they do not make the Sony codec. MPEG shimmering is a common problem and usually indicates inadequate bitrate. After tuning for constant quality, it is also possible that LongGOP may need to operate at double the bitrate, anyway. We continue this discussion because we *suspect* that LongGOP should be at least twice as efficient as I-Frame only.

[Piotr Wozniacki]

Any news on L-GoP optimization, Dan?

Thanks,

Piotr

[Gints Klimanis]

Piotr, It's taken me this long to accept that recording I-Frame only at 220-280 Mbps is the way to go. Yes, I was disappointed that 6-8x data rate is needed to exceed the 35 Mbps Sony EX1, but that is the best portable high quality video system available. We did not expect that the higher data rates would expose sensor noise on the Sony EX1 and that much of the high bitrate is channeled to code this noise as accurately as possible. Still, better inter-frame data balancing improvements would be welcomed in a future release. Nano could be that ultimate MPEG-2 LongGOP recording machine.

[Mark Job]

Hi Gints & Piotr:
This is where one truly hits the wall with MPEG 2. MPEG 4 is actually much more efficient at encoding at lower data rates. MPEG 2 has to use high data encoding rates to obtain even slight improvements in quality, whereas MPEG 4, Quicktime, MJEPEG, Cineform and Cineon do not. Quicktime allows for 8,10 & 12 bit color precision encoding. MJEPEG will do 8 or 10 bit encoding precision, and Cineform will do anything. The only way to get SD & HD MPEG 2 to look really great is to use those famous $50,000.00 Zoran hardware, realtime MPEG 2 encoders, which use special chip optimizations for CBR & VBR encoding @ key DVD authoring data rates. (Usually, 4, 6 and 8 Mbps for SD) (18 to 24 Mbps data rates for Blu-ray) These boards can give you the big screen HD quality you seek @ 18 Mbps data rates ! I have no idea of what *Optimization settings* the Sony hardware MPEG 2 XDCAM 4:2:2 encoder is set at to give you the optimized MPEG 2 results you are looking for ? (Probably Long GOP 50 Mbps would be one of the targets this chip in the XDR & Nano is optimized at to give you a really great picture) I have observed Long GOP 100 to be no sweet spot as far as I am concerned in regard to a very clean BIG screen or projected picture, yet I have seen XDCAM Long GOP 50 projected , as well as on a 50 plus inch flat screen, and you see a remarkably clean image without the granularity & flashing you guys are seeing.

A word to the wise; If you are using FCP to post your Nano/XDR clips, then please be ware you are at a certain disadvantage, which can actually cause the anomalies you are seeing on large screens. What do I mean ? Simple. FCP is processing all of your effects @ 8 freaking bits ! Now with Avid (And **MANY** Avid Media Composer editors simply do not know this as well !) - AMC gives you the *Option to process and render @ 16 bit precision !!! Yup !!! This is what I wrote ! Many folks don't even know how to turn it on !!!

**This setting is in the >Media Creation>Render Settings Tab !

1. Don't check the Same as Source box, because your source is only a sucky 8 bits !
2. Don't select *Automatic* because this tells Avid Media Composer to analyse the video clips and automatically process all effects at the same color precision as your clips (Sucky 8 again !)
3. DO SELECT 16 BIT PROCESSING ! Even Avid goes into some detail in their instruction manual as to how 8 bit sources can benefit from turning on 16 bit processing.

Two things will now happen to what you post:

A) Your processing (Read Rendering) times will take 3 times longer than before.
B) Your resulting output will look demonstrably smoother, there will be *MORE* color ! There will be *LESS GRANULARITY* ! You guys *WILL NOT* see those anomalies you are seeing anymore (Providing you shoot @ Long GOP 50 Mbps) You simply won't believe your eyes. - And what you observed for I-Frame recording @ 220 and 280 Mbps will still apply, but wait until you see what that now looks like !

Try my solution - You will like it !

[Joe Batt]

Mark, this page shows info that is different than what you posted.
Apple - Final Cut Studio - Tech Specs and System Requirements
I'm not trying to challenge your info I'm just trying to learn, Am I reading what they are listing wrong? I would like to know because I just got the Nanoflash and am trying to get the most out of it.

[Piotr Wozniacki]

Originally Posted by Gints Klimanis

Piotr, It's taken me this long to accept that recording I-Frame only at 220-280 Mbps is the way to go. Yes, I was disappointed that 6-8x data rate is needed to exceed the 35 Mbps Sony EX1, but that is the best portable high quality video system available. We did not expect that the higher data rates would expose sensor noise on the Sony EX1 and that much of the high bitrate is channeled to code this noise as accurately as possible. Still, better inter-frame data balancing improvements would be welcomed in a future release. Nano could be that ultimate MPEG-2 LongGOP recording machine.

Hi Gints,

Same here - I invested in a couple more 64 GB cards, and whenever shooting something "serious enough" (meaning intended for heavy grading etc.), I'm using I-Frame Only 220 Mbps.

At other occasions when L-GoP is enough, I almost exclusively use the 50 Mbps bitrate.

Mark,

I'm not a Final Cut or Avid user, but you're right that a lot can be improved in post. Vegas Pro has an option of 32bit floating point pixel processing, and I'm using it whenever my footage needs de-noising. For that, I'm using the Neat Video plugin, which gives excellent results (not just de-noising, but also sharpening the footage to my liking). The price is much longer rendering times, so - having invested in the nanoFlash - I'd rather get clean pictures to start with, and of more manageable sizes than the 220 Mbps is producing...