Sometimes I used It To Make Quick Blu_ray DVD's
 

Hi Dan:
Yes, I do use .MPG feature (Which was busted for playback in the XDR BTW - I haven't tested that long duration .MPG playback lately). I use .MPG recording to make quick industrial Kiosk Blu-ray DVD's. It works *very well btw.

Dan, the interest in Constant Quality MPEG2 and VBR is heating up. So, you already provide VBR for MPG (I haven't used that) recording ?

Missed this gem. Really ? I'm willing to test the alpha/beta/whatever.

What about the I-Frame & Long GOP Footage being in VBR ?
 

...Hey Gints: I thought the point of this exercise would be to offer the *option for us end users to be able to engage VBR encoding when shooting Long GOP and I - Frame ? In my opinion, this is what I think needs to happen. (Although it's always great to have this on .MPG setting, but let's be real - How many of us will do any real big work in .MPG setting ?)

Mark, agreed. I'm just anxious to find out if a VBR implementation will reduce the LongGOP shimmering that started this thread.

Hello Dan,
I agree with Mark VBR and CBR selection surely has to be the next feature that is enabled on the NanoFlash/XDR. I have used other Sony MPEG encoder boards in the past which have offered the selection of VBR or CBR and I found that I used the VBR option all time tailoring file size vs quality.

The only point of VBR is to optimize the files size. If you are not constrained about file size, VBR adds nothing. Aside of this, a proper VBR compression needs a multi-pass process.
Better than VBR would be to implement Open or Short GOPs (I don't know if is possible with this processor and in a RT process).
rafael
PS: If people is having problem choosing a proper data rate in CBR, wait for them to decide the Average and peak data rates for a VBR compression.

Dear Lance,

When you create VBR files, what NLE's are you using to read the files?

Could you please provide more info?

Bit-Rate, Long-GOP or I-Frame Only, 4:2:0 or 4:2:2, and what problems have you had reading the files?

AFAIK, single-pass VBR also has its advantages (I'm not sure how much less efficient datarate "distribution" is in this case, but the file size can be decreased for sure). But true - the most efficient is multiple pass, whereby one pass is needed to analyze the video quality needs, and another one to encode those demanding parts of it (motion etc.) at a higher, and the remaining part - at a lower bitrate, just to minimize the output file size.

As to the potential improvement in L-GoP quality arising from using shorter and more sophisticated GoP structure, I was mentioning this many times earlier in this thread.

I wouldn't worry about the added complexity - if some settings do indeed bring much better PQ than others, people would learn very fast how to use them ...

My interest in VBR is control of image quality variation from frame to frame. The Sony XDCAM 140 Mbps has a problem with quality variation, and we hope that VBR will address this problem . Of course, this will also reduce file size if I'm able to use 140 Mbps VBR to vary between 120-180 Mbps and yield the quality of 280 Mbps I-Frame only.

Gints, I hear what you're saying. In fact, we have both noticed and analyzed this quality variation that causes the noise shimmer, making it more distracting. BUT, I am not so sure VBR can take care of it - please try to convince me, and than we could try together and ask CD to try and implement it :).

What I - with my current, limited knowledge - believe to be a better cure for the intra-frame inconsistency, is some fine-tuning of the GoP length and structure. There's a number of parameters controlling this - what I'd expect CD to do is at least let us know which of them are available for their code to be used on the Sony encoding chip.

Who knows - perhaps the two approaches can complement each other?

I'd say the strategy should be:

- leave the I-Fo as is, but limit it to some really high bitrates where it shines (220 and up)
- try to improve the L-GoP by either optimizing the GoP structure/length, or adding VBR encodig, or both.

What do you say, Dan?

Dear Dan,

Of course my knowledge is limited, but frankly I don't see reasons why NLE should not recognized the VBR L-GoP files if nanoFlash had an option to record in this format.

With Vegas, one can encode either a CBR or a VBR MPEG-2 file from any source in the timeline, and such a file is always recognized, read-in, and played back in Vegas. Just tried it with Edius - same story.

How an MPEG 2 File is Encoded Should Not Affect N:E
 

Hi Piotr & Gints: Once a Sony XDCAM HD 4:2:2 MPEG 2 file is created, then the NLE should treat it as any other completed and ready to play XDCAM HD file. How the file is made makes no difference as long as the finished XDCAM HD file meets the editing specification of the NLE - In the case of Avid Media Composer, the known standard is Long GOP 50 Mbps or I-Frame at any data rate you please.

* As long as the file header information remains in a form the NLE was previously able to read, then, in theory, there shouldn't be any issues.

It would be good to find a manual on the Sony XDCAM encoder, and admittedly, I don't know what I'm searching for. I see a qpweighting parameter mentioned in the AVC encoder specs. I wonder if this is the parameter for setting the relative quality of the I, B and P frames which are often specified as IQSCALE, PQSCALE and BQSCALE in MPEG encoder parameter files :

http://www.sonic.com/images/products...parameters.jpg

Piotr,

Followed your thread only incidently, but was triggered by it in the first place because I also wonderd why at some circunstances noice was moving through my images like it was 'snowing'. Specially at lower light circumstances. I did not fully check the whole thread now, and discussion seems sometimes to move towards other directions. However, I have been playing with my gamma settings and noticed that this terrible noise out of the EX3 into the nano was very pronounced with gamma STD3 (which I used before), and is almost not there with CINE2. Maybe basic knowledge you know since long, but a revelation for me.

Cees, of course some EX's PP settings are better than others as far as noise is concerned. Usually however, no "best" setting exists - it's always a trade-off.

Let me stress it however that this thread is not about how noisy the EX cameras can be (some even say they are not that noisy at all, considering chip size and price tag). What we're trying to find out here is the best setting for the nanoFlash (or even modifying the way it encodes, if possible and viable to Convergent Design), so that the advantages of this marvellous device always outweigh the main disadvantage: the tendency to augment the noise, by exaggerating its shimmering nature.

Everybody participating in this thread realize perfectly that camera noise cannot be eliminated by the nanoFlash, as the device cannot tell noise from detail, and enhancing detail is one of its very purposes (along with minimizing compression artifacts). However, we've achieved some consensus on that the problem is aggravated by the L-GoP-inherent, frame quality fluctuation. Thus, minimizing this fluctuation has become the goal of our debate here :)

Thanks for your input, anyway. If in response to it, I have taken the liberty to summarize the essence of this (somehat convoluted) thread , isn't because I'm patronizing on you at all (please forgive me if it sounds that way). I simply wanted to help those new ones lurking into it, to understand what is being talked here...

:)

Piotr

By definition XDCAM HD 422 is CBR.
rafael

XDCAM HD File Encoding
 

Hi Rafael: Yes I understood this was so, all I was wondering if there is room in the Sony XDCAM HD specification for the utilization of a VBR encoding scheme. Does anyone know or have a link to any possible web page which would display the full Sony XDCAM HD 4:2:2 spec so we could check ?

Sony's pro video british site has a lot of detailed docs relating to their pro gear. I would start there if you are looking for XDCAM specification related material.

Thanks
 

Hi Andrew:
OK. Will do :-)

Sorry guys I did not intend to send this thread into a different direction.
DAN
For your info the NLE was M100 iFinish which came with the real time Sony MPEG encoder SD the menu option allowed the use of 15 to 50 Mb I-frame 4:2:2 in CBR. It also allowed the selection of MPEG-1, MPEG 2 4.2.0 or MPEG 4.2.2. This system was only used to create archival copies of finished projects and not used as an editing medium. I used the VBR option for storage allowing control of file size vs quality. I now use the NanoFlash to store my finished projects.

Dear Lance,

Thanks for the extra information.

Hi Mark,

To cheat the NLEs shouldn't be a problem. In fact the NANO files are off-standard (data rate, I-frames,..) but they work in most of the applications. At least FC can manage ot of stuff that in theory is not supported.
rafael

I agree that "cheating" NLE's should not be a problem. After all, all we need is the highest possible quality - and NOT a strict compliance with any particular "industry standard"...

Why?

Well - simply because it's our delivery formats that ultimately counts; this must be absolutely compliant with some industry standard (be it HDTV broadcast, BluRay disk, ....).

So, does it make sense to bother Convergent Design an push them to upgrade their encoding schemes from the "constant bitrate" to the "constant quality"?

Well, I think no, it does not (strange does it sound from somebody who initiated this thread, doesn't it).

I'll explain: given that all cameras will inevitably feed the nanoFlash with some noise (XDCAM EX in particular, as this is their Ahilles heel when compared to other great specs) , the only solution seems to be getting rid of it in post.

I'm editing in Vegas Pro, and I've just purchased the Neat Video Pro plug-in. I have run it on both the SxS clips, and the simultaneously recorded, 220 Mbps I-Fo nanoFlash clips...

And I can tell you the results from the nanoFlash clips (not just I-Fo 220, but also L-GoP 100) are much, muuuuch better than those from the native XDCAM EX. The obvious reason being that with much less compression artifacts, the ingenious Neat Video filter can really show all its power on what it's supposed to deal with: the noise.

Add to it that Neat Video can also sharpen your video at the same rendering run, and you can shoot with ALL detail enhancement OFF - thus further relieving the nanoFlash codec from dealing with detail enhancement, and allowing it to encode even more efficiently. The motion, as well as static high frequency details, can be better handled by the encoder - resulting with even less compression artifacts.

Do you hate noise, that looks augmented along with detail in the nanoFlash files? I do. Do you want your video sharper? Well, I do too.

The answer for the time being is:

- record in L-GoP, at 100 Mbps max, or
- record in I-Fo, at 220 Mbps min, and

- render your material with the Neat Video (or any other, equally capable, tool)!

Well, of course Convergent Design may come with a better solution any time now, but frankly, I'd rather they spent their man-hours on finally implementing CF card hot-swapping, SDI embedded + analog audio mixing, and alike...

Just my opinion, YMMV.

Piotr

Neat video ...
 

I've been using this software and it does very good noise reduction. I use it on everything out of the EX3.

Any news from testing Kris's NF?
 

Hello Piotr, I would love to know what is the outcome of the NF trade/test with Kris? You never come back to clear up the suspicion of faulty NF or camera ensue?
Cheers

Hi Luben,

My meeting with Kris is planned as soon as he comes to Poland (he's in UK now, and supposed to give me a phone call when he arrives here). Of course I will update this thread accordingly - but frankly, I'm 99.99% positive both my EX1 and my nanoFlash are working up to specs.

It's just that - unlike most "real pros" here who assess their material using super expensive production monitors with HD-SDI input - I always base my opinions on the results I'm getting with my audience in mind, and this means a really big display. Not necessarily with professional inputs (just HDMI), but of 50" and up in size - while most persons trying to convince me there is no noise problem are using something like a 20" incher... What they don't realize is that it's not the type and quality of the input, but the very size of a pixel which decides whether or or not some artifacts will be noticeable or not.

I have almost come to terms with it, and am right now working a new workflow involving obligatory use of the Neat Video filter in Vegas. The additional benefit is that it can de-noise and sharpen at a single go, thus making it the obvious choice to shoot with in-camera sharpening (detail) off. This further enhances results by making the nanoFlash encoder less burden (no sharpening artifacts), while the noise can be removed in post and the picture sharpened to taste.

It's a pity though that all those ideas myself and other people presented in this thread (like fine tuning the Long GoP structure, or even introducing VBR) have not even been commented or considered seriously by CD...

VBR Encoding *is* part of Sony's *OFFICIAL* XDCAM HD 4:2:2 Spec !D
 

Hi Piotr & Luben:
I can now confirm that VBR encoding *is* part of the *Official Sony XDCAM HD 4:2:2 SPEC ! I have a copy of the spec straight from Sony entitled,

"Codec Technology for XDCAM Tapeless Products and Systems
By Hugo Gaggioni, Chief Technology Officer, Sony Broadcast & Production Systems Division"

On Page 5 it States:

"3. Adoption of VBR
For HQ and LP mode, a Variable Bit Rate compression method was adopted in order to take advantage of the random access nature of tapeless media. It is different from the 2 pass* VBR compression technique used in DVD authoring. This codec was specifically developed for MPEG to perform high-speed efficient picture coding.
The XDCAM product line is based on the MXF file format and MPEG-2 Long GOP for exchanging and transmitting Video, Audio and Metadata. In the VBR (Variable Bit Rate) encoding method adopted for XDCAM HD products, the maximum bit rate that can be processed in real-time is fixed. This results in the high level of picture quality exhibited by XDCAM HD products over different picture material while minimizing the file size and maximizing transfer speed."

Apparently, VBR encoding was introduced by Sony as soon as the XDCAM *HD* version of cameras came out.

Dear Piotr,

We have to maintain compatibilty with a wide range of Non-Linear Editors.

Using a non-standard number of frames in a Group of Pictures would require substantial amount of testing to ensure compatibility.

We use a "Closed GOP". Switching to an "Open GOP" has ramifications when editing. With Closed GOP, the I-Frame that starts each group does not depend on or need information from the preceeding GOP.

Dear Mark,

There are three flavors of XDCam:

XDCam HD
XDCam EX
XDCam

We support both XDCam EX (35 Mbps, 4:2:0 35 Mbps VBR) and XDCam (50 Mbps, 4:2:2, CBR).

XDCam HD is different.

XDCAM - Wikipedia, the free encyclopedia

Any chances of supporting XDCAM HD format - at least with the higher bit rates, Dan?

Dear Piotr,

I have not been able to determine if XDCam HD is 1440 x 1080. The PDW-350 is a 1440 x 1080 camera and it is a XDCam HD camera.

We want to continue to record in 1920 x 1080, thus we are using XDCam format and not XDCam HD format.

Dear Dan,

I'm a bit confused with what you're saying. Even in the Wikipedia article you linked to, it reads:
"In August 2009, Convergent Design began shipping the nanoFlash Portable Recorder, which uses the Sony XDCAM HD422 codec at bit rates ranging from 18 Mbit/ second to 35 Mbit / second (in the 4:2:0 colorspace), and bit rates ranging from 50 Mbit/ second to 280 Mbit/ second (in the 4:2:2 colorspace).

So, why are you saying nanoFlash is only capable of XDCAM EX and XDCAM, but not XDCAM HD?

As to your other point: of course I'd be for the full raster - just VBR option for the high bitrates. From what Mark quoted, the Sony chip you're using is capable of it...

Thanks,

Piotr

Huh ?
 

Hi Dan:

Uhh, I understood the Nano Flash & Flash XDR are using the Sony XDCAM HD 4:2:2 codec, therefore, VBR is also possible. (??) So what codec are you using ? XDCAM or XDCAM HD ? XDCAM is an SD only codec according to the Sony White Paper on the spec.

Dear Dan,

Could you please explain your confusing post? I'm sure it's not just myself or Mark who are waiting...

Thanks,

Piotr

Dear Friends,

As I understand it, there are three flavors of Sony XDCam:

XDCam HD, 1440 x 1080 4:2:0
XDCam, 1920 x 1080 4:2:2
XDCam EX, 1920 x 1080 4:2:0

Here is an excerpt from an article on Wikipedia:

XDCAM is a tapeless professional video system introduced by Sony in 2003.

The first two generations, XDCAM and XDCAM HD, use the Professional Disc as recording media. This disc is similar to Blu-ray Disc and holds either 23 GB of data (PFD23, single-sided) or 50 GB (PFD50, double-sided).

The third generation, XDCAM EX, uses solid-state SxS cards instead.

XDCAM - Wikipedia, the free encyclopedia

--------------------------------------------------------------------------------------------------

The nanoFlash uses XDCam, 1920 x 1080, 4:2:2 at all bit-rates 50 Mbps and higher.

And XDCam EX, 1920 x 1080, 4:2:0 at bit-rates of 18 Mbps and 35 Mbps.

And we have an option to record in 1440 x 1080, 4:2:0 at 35 Mbps.

--------------------------------------------------------------------------------------------------

1. The Sony PDW-F330, PDW-F350, PDW-F355 are "XDCam HD".

Here is some text from an offical Sony site discussing the Sony PDW-F330:

HD 1080 Recording at Selectable Frame & Bit Rates
XDCAM HD products can record video signals in 1080/59.94i, 50i, 29.97P, 25P and native 23.98P - using a "MPEG HD" codec with industry standard MPEG-2 MP@HL compression.

The user can also select the following bit rates; 35, 25 or 18Mb/s depending on picture quality and recording length requirements. Choosing the highest bit rate of 35Mb/s results in a recording time of approximately 60 minutes, while choosing 18Mb/s provides a recording time of approximately 120 minutes - the longest recording time offered by any current HD camcorder.

MPEG HD images are recorded at 1440x1080 pixels with 4:2:0 sampling.

Sony : PDW-F330K (PDWF330K) : Features : United Kingdom


--------------------------------------------------------------------------------------------------

2. The Sony PDW-700 and PDW-F800 are "XDCam".

The Sony PDW-700 is titled:

"PDW700 Sony Professional XDCAM®
HD Camcorder"

The Sony PDW-F800 is titled:

"PDWF800 Sony Professional XDCAM®
HD422 Camcorder".

While the title for the Sony PDW-700 camera includes "HD" on the next line, it is in the same class as the PDW-F800 camera which uses "HD422" on the next line. I interpret these to indicate that these two cameras are "XDCam" cameras as opposed to the "XDCam HD" cameras that are 1440 x 1080.

The following is from an offical Sony website for the PDW-700:

MPEG HD422 (MPEG-2 422P@HL) (CBR: 50 Mb/s)
MPEG HD (MPEG-2 MP@HL):
HQ mode (VBR, maximum bit rate: 35 Mb/s)
SP mode (CBR, 25 Mb/s)
LP mode (VBR, maximum bit rate: 18 Mb/s) (Playback only)
MPEG IMX (MPEG-2 422P@ML) (50/40/30 Mb/s)
DVCAM (CBR,25 Mb/s)

Sony Product Detail Page - PDW700


The follow is from an official Sony website for the PDW-F800

MPEG HD422 (MPEG-2 422P@HL) (CBR: 50 Mb/s)
MPEG HD (MPEG-2 MP@HL):
HQ mode (VBR, maximum bit rate: 35 Mb/s)
SP mode (CBR, 25 Mb/s)
LP mode (VBR, maximum bit rate: 18 Mb/s) (Playback only)
MPEG IMX (MPEG-2 422P@ML) (50/40/30 Mb/s)
DVCAM (CBR,25 Mb/s)

Sony Product Detail Page - PDWF800

--------------------------------------

3. The Sony PMW-EX1, PMW-EX1R, PMW-EX3, PMW-320 and PMW-350 use "XDCam EX".

The following is from an offical Sony site, and describes the Sony EX1R:

Video:
MPEG-2 Long GOP HD HQ mode: VBR, maximum bit rate: 35 Mb/s,
MPEG-2 MP@HL HD SP mode: CBR, 25 Mb/s,
MPEG-2 MP@H14 SD mode: DVCAM
Audio: Linear PCM (2ch, 16-bit, 48-kHz)

Sony Product Detail Page - PMWEX1R

Don't Always Go By Wikipedia - It's Often Erroneous
 

.....Dear Dan: I have to tell you that no High School, Community College, or especially ANY University will accept written papers or essays with citations from Wikipedia, because Wikipedia is often presenting erroneous data as factual and accurate.

According to Sony's White Paper data: XDCAM HD is 1440 x 1080 AND 1920 x 1080
**XDCAM = Standard NTSC Definition @ 720 x 480, or 486 and PAL Spec ONLY.
The Nano Flash therefore *must possess a Sony hardware encoder of the XDCAM HD 4:2:2 persuasion. The XDCAM HD spec can ALSO do Standard NTSC Definition. EDIT: XDCAM is 4:2:0 and XDCAM HD is 4:2:2
-------------------------------------------------------------------------------------------------- [/QUOTE]

Hi Dan,

I am out of my league here, but as I understand the principle, the nanoflash pulls the signal from the HD SDI stage and so circumvents the compression by the camera, thus giving the option of compression to the nanoflash. I realize this is a bit of an oversimplification because not all HD SDI signals are the same.

The key to the puzzle would presumably be to compare the HD SDI signals from the XDCAM EX and the XDCAM HD cameras (not the final codecs). If the HD SDI signals are the same, then the request seems to be to have the nanoflash compress the signal from 1920x1080 to 1440x1080.

At that point I am lost because I am not sure why you would do that....???

Alan

Addendum
 

Hi Dan:
Another reason why I am thinking the Sony hardware encodrer in the Nano/XDR is of the XDCAM HD 4:2:2 verity, is because the XDCAM regular encoder *cannot ALSO do XDCAM EX. Only the more modern XDCAM HD 4:2:2 encoder can do the extra XDCAM EX 4:2:0 codec as well.

Dear Mark,

For each quote, I referrenced the source, Many of these were from official Sony websites.

I am well aware that Wikipedia has errors.

Sony's labeling is confusing.

In any case, I listed what the nanoFlash can do.

Whether you want to call it Sony XDCam, XDCam HD, or EXCam EX does not really matter.

I have listed the modes that we can do, which includes the formats used by the Sony PDW-330, 350, and 355, and the modes used by Sony XDCam EX (EX1, EX1R, EX3, PMW-320 and PDW-350), and the modes used by the Sony PDW-700 and PDW-F800.

Thus we handle the 1920 x 1080 and 1440 x 1080 as well has 720p 1280 x 720.

Would you like to include a link to the Sony White Paper you reference?

Dear Alan,

We added the 1440 x 1080 mode, to match the output of the PDW-330, PDW-350, and PDW-355 cameras at the request of a major Hollywood studio.