EX1/EX3 line twitter and shimmering

[Steve McClean]

I seem to get an awful lot of line twitter and background shimmering when viewing my EX1 and EX3 footage on a 1080p LCD TV. Its the same effect whether I am watching raw footage straight from the camera or the edited version on DVD. I've read a few posts here suggesting that this is probably a monitoring issue but the fact is that my end clients are watching the end output on similar consumer TVs so I'd love to find a solution that eliminates this problem as much as possible (or entirely!).

Turning down the detail when shooting helps a bit but doesn't get rid of the problem and gives the extra problem of my beautifully sharp HD imagery getting softer and softer. Any conclusions on how to best deal with this problem guys? It is the same on both my EX1 and EX3 suggesting it is not a problem unique to my camera - I tend to shoot PAL 1080 25p and use Phil Bloom's recommended picture profile settings...

Thanks
Steve

[William Urschel]

Dear Steve:

My title above says it all, and I MEAN it. I have the same issue with the EX-1 as you are discussing. It appears that there are two divergent camps in this industry - one which has the same very negative issue as you and I. And then what may be the majority stating, "What Problem?". The latter appear to just edit normally with no issue at all (a prime example would be Doug Jensen, who I respect immensely - I have purchased, with good reason, just about every video or program device he has released to market, as well as his "warm" cards, etc.). If I look at one of his videos shot with the EX-1 (his superb intro to the EX-1 was shot mostly with something else, as the EX-1 was the subject!) I see the same issue which you are discussing on a Samsung 42 inch flat panel LCD (with Sony BD Player) and likewise on a 109 inch Stewart Firehawk Screen (Projector Panasonic AE 3000U, Denon receiver with 1080 30p straight signal feed through with no processing, and a very capable Panasonic BD Player). Some of his material is 1920x1080 60i, some is 720x480, and then tere are a number of other presentations. But when shown on the two
referenced systems, what shows up is exactly what you are talking about. And both systems are thoroughly calibrated, and neither system shows ANY such issues with most Hollywood DVDs and BDs. And when his EX-1 mateiral is shown on a much lower reolution computer screen, everything looks fine.

Before I go further, and really aside from my agreement with your issue, and since you're looking for a solution, I suggest that you search this and some other related forums for the answer to the issue which has been presented by so many! Others have found the problem to be in the generation of, for instance, the MPEG2 rendition for both BD and DVD, and have used such solutions as ProCoder 3, and a myriad of others, including a considerable number of freeware programs! Some have found that the issue for DVD is downrezzing to SD from HD (that really is not my issue, since I use Cineform's PHD 4, which is considered as superb as anything out there for downrezzing to SD from HD).

You made no mention if you're on PC or Mac (or did I just miss that, or what editor you're using, which may help explain why you've had no response), but no matter what you're on, you will find a myriad of solutions out there that have worked for so many - I have tried many of them and NONE have worked for me. There follows some of the solutions I've tried, and then ONE solution I've found for myself that works for Blu-Ray.

Last year I spent over 1700 hours attempting to get something that worked for me and my customers, and I failed utterly. And by the way, I have the FX-1 and three other HDV cameras with great pictures for BD and DVD! I started with a $7,500 Boxx Computer (obviously running Windows XP at that time), Adobe CS3, and a bunch of ancillary programs - as stated, this all worked great for everything I threw at it, except EX-1 at 1920x1080 60i or 30p. Soooo, I purchased another $7,000 Boxx i7 overclocked, 6 Gig Ram, 64 bit Vista, and purchased Adobe CS4 (ouch), etc. And that took everything I threw at it (except Premiere CS4 glitches till they fixed it), except the EX-1. And I spent another $2,500 on programs to handle MPEG2 encoding - and I followed to the letter the direction given by others who had no problem. And to no avail. So, being curiouser and curiouser I decided to puchase a cheap but capable consumer camcorder (a $1,300 Canon hi-def, with hard drive and utilizing the dreaded AVCHD. Well, as you can imagine, with one smaller CMOS sensor and a cheap lens, the picture wasn't great, but it was hi def - and the interline twitter and other arifacts I seen with the EX-1 were EXACTLY the same!

SO I QUIT! I am currently making very presentable 1920x1080 30p shots which show up with considerable detail on almost all screens. As you have, I had previously tried various detail reduction (in the camera and external as such efforts as Premiere Gaussian Blur -both horizontal and vertical), and it had reduced the problem, but as with you, the problem persisted, and I was sickened by blurry photos that couldn't match ANY of my HDV cameras.

I currently produce fairly sharp, artifact twitter free Blu-Ray discs by finally rendereing out the timeline with Adobes anti flicker filter. That's all, and it works for me and my customers, and it IS sharper than any of my HDV cameras.

I no longer produce ANY DVDs resulting from the EX-1 - they always just turned out as Garbage, no matter what.

Today when a customer insists on both DVD and BD, I use one of the HDV Cameras. And
everyone's quite pleased

But again, look at all the solutions out there, and if you find one that works for you (most people do!) be sure to post it here!

Goodnight and Good Luck!

Bill Urschel

[Perrone Ford]

Bill,

Out of curiousity, could I ask you two questions:

1. How is your EX1 set up? I am particularly interested in the picture profile settings you use.

2. What was your workflow? How did you get your video into your computer, and from there what was the path to how you got it to Blu-Ray.

I am not doubting your issue, I am just curious as to what might be causing it.

[Piotr Wozniacki]

The OP's problem is probably not line twitter, but the mosquito noise that the EX camera are notorious of when under-exposing.

[Perrone Ford]

Would be extremely helpful for the OP to post a still frame of his video that exhibits the problem.

[William Urschel]

Perrone:

After trying a whole myriad of PP settings (including the one suggested by Doug Jensen),
and after, in particular, decreasing the picture detail in 15 different steps and carefully monitoring the very disappointing results, PP has been set to off, ever since. What I find
necessary to modify is taken care of in post (yes, I know, there are some things it is too late for that!).

As to workflow for EX-1 1920x1080 30p ending as BD-R:
1. Update OS, security, video driver, all Adobe CS3 and CS4 programs, and Cineform, turn off anti-virus, unless then connected to Internet. Conform separate 7.2krpm 850 GB hard drive, and then set up folder structure for all edit functions.

2) Place SxS Card in Sony SBAC-U510 Memory Card USB Reader/Writer

3) Set up Save Folder on BCDV-01 Data Hard Drive for placement of BPAV folders from Memory Card Reader - folder may be named in any way determined, except it MUST NOT include the letters “BPAV“!

4) Open "Computer" File on Boxx, and then transfer (copy and paste) BPAV folder from Memory Card Reader to appropriate Save Folder - multiple Save Folders titles should fully identify unique card content, as the copied BPAV file title itself cannot have deletions or additions of any kind, including title, deletions, or additions.

5) After transfer of BPAV files, if necessary, open up XDCAM EX ClipBrowser to identify individual BPAV EX-1 / Camcorder files, and, as appropriate, delete unnecessary files

6a) Convert individual BPAV files to AVI files for editing by setting up Cineform Prospect 4k HD LINK. DO NOT UTILIZE ADOBE PREMIERE PRO CS3 OR CS4 SETTINGS FOR DIRECT IMPORT OF BPAVE FILES! Doing so, instead of using HD LINK conversion will be MUCH quicker and apparently as effective, but will result in final production with much more distortion, no matter what the popular press and commentators have to say to the contrary!

Open Cineform Prospect 4K "HD LINK" and then click "Preferences" on the Cineform Window,
and then in the window that opens:
- avi - Automatically convert to Cinform Intermediate
- Capture Location - specify a location, if not automatically done so
- Check "Use Capture Location for Converted Fields"
- Make up an identifying "Base Capture Name"
- Under "Destination File Format For Capturing and Conversion", check "AVI"
- Under "Cineform Encoder Options" select "Film 1"
- Enable "threading" to make maximum use of computer capacities
- Under "Frame Format", select "Progressive"
- Select "Convert to CFHD

Then, in the initial window, select "Folder" and select the BPAV file folder for conversion and run the program.

6b) Initiate and set up Adobe Premiere Pro CS3 Program Format: XDCAM EX 1080p 30p (HD) (Cineform), provide name for project folder, select and load Premier.

7) As soon as Premiere is loaded, it is VERY IMPORTANT to immediately select "Preferences" and assure that data is saved to the S Drive Adobe Premeire CS3 Project Folder, to AVOID filing data on C Drive

8) Edit video in timeline, run normal review, including audio, to be no higher volume than -6 dB

9) “Export Movie” from work area bar, utilizing Cineform Prospect 4k program as: Cineform AVI Export, Film Scan 2, Enable Threading, 1920x1080, Square Pixels, No Fields (Progressive Scan). Identify target folder, name specific project and format, and Save, to start transcode

10) Then, in the previously set-up Premiere project, import the just exported Cineform AVI, place in timeline, and view and test for validity. Then delete original edit segments. For final AVI, as single segment, right click the segment, open "Field Options", and check "Flicker Removal, and render within Premier.

11) Again “Export Movie” from workbar, as Cineform Prospect 4k, formed as 1920x1080 30p, with same settings as previously, except render as "High", instead of "Film 1"

12) Copy and paste resulting file to "My Book" (for My Book, double copy to two separate drives, for safety) backup for semi-permanent save

13) Following rendering, set up Adobe Media Encoder CS4 as MPEG2 Blu-Ray, 1080i 29.97 High Quality, Quality-5, 1920x1080, Upper, Constant Bit Rate-30, and render MPEG2 codec.

14) Initiate and set up Adobe Premiere Encore CS4 as BD-R MPEG application: bring AME file into timeline, and set up menu and timeline processing, test (for test procedure only, use BD-RE), and following successful BD-RE, prepare required number of BD-Rs.

I have never had any particular anomalies, artifacts, etc., resulting from low light shots; compared to any of my other cameras the production from this camera is superb.

Except for ingestion via firewire, HDV material is handled identically (without any flicker removal), except for conversion of the HDV to square pixels - HDV material does NOT demonstrate any significant flicker, twitter, etc..

I have been invited from time to time on this and other boards to give screen shots - these have been useless in exhibiting the problem I see on the big screen, as single frame grabs are usually entirely pristine within the confines of the architechture. The twitter does not show up until it goes up on the screen. And finally, as I have said elsewhere, I have been
designing, constructing, and maintaining what is now known as home theater since 1984 (with 35 inch CRTs until Barcovision and other front projectors starting in 1991 - in the commercial realm, except for a few shoddy or very early DVDs, I have not seen the kind of distractions from disc technology until I first attempted disc production from the EX-1! And as the rabbit said, "That's All Folks!"

[Perrone Ford]

I have a couple of disparate thoughts here looking at the workflow, and at your careful observations.

1. Is it possible for you to simply open up a file straight as it comes out of the ClipBrowser into Premiere and bypass the Cineform conversion?

2. Your note that you do not see this with HDV is interesting. HDV, shooting at 1440x1080 is not capturing anything close to the same detail and consequently, I wonder if we are seeing that lack of resolution (and perhaps aliasing) solving the issue. Also, HDV is typically shot interlaced. Do you normally shoot interlaced on the EX1?

3. If you export your final file to something other than Mpeg2, say AVC for BluRay, do you see the same issue?

This is a curious problem to me, and your workflow seems unnecessarily convoluted, but I have never done HD through Premiere so perhaps it is all necessary. I would appreciate it if you could expound a bit on the questions I've asked here.

I would also like to get a 5 second clip of your problematic raw footage to see if taking it through my workflow would produce the same results.

-P

[Bob Grant]

What the camera is shooting is another factor to consider. If there's no fine detail in the scene then you're unlikely to see a problem. I made a series of DVDs from hi res stills. Most were OK, some required a bit of wrangling, one was so bad the entire image appeared to blink at several fps.
We have a plant down here referred to as a "Black Boy", not very PC! This plant has very long, very fine leaves growing from the top of a black trunk. It dislike this plant for the problems it creates with line twitter.

Monica's Site - Xanthorrhoeas

Get those in a shot and you can have some issues.

I have to admire people who buy a new PC in the hope of wrangling problems such as these, I've yet to find a computer that changes binary data differently to another one. What would be more fruitfull as well as a screenshot or two is knowing what mode the camera was recording in and what was the output encoded to e.g. 24p, 25p, 50i or 60i?

[William Urschel]

Dear Perrone:

Thank you for your interest! I have often read your posts which are usually helpful.

1. In the some 1,700 hours I spent last year attempting to secure decent video productions from the EX-1, one of the many permutations and combinations I attempted (more than once) was to open the file straight as it comes out of the ClipBrowser into Premiere and bypass the Cineform conversion - the results were decidedly poor, compared to using the Cineform "Visually Lossless" TM Premiere plugin and programs.

2. Your observation about the significantly lower resolution with HDV probably gets at something of the crux o the matter.

I have used the EX-1 in the 1920x1080 60i mode as well as progressive. For Blu-Ray discs, I obtain results which show substantially less interline twitter and artifacting when I shoot interlaced than when I shoot progressive, but I have continued to shoot progressive for the last year, as now and then a customer will change his mind at the last moment, indicating a desire for DVD as well as Blu Ray - if I shoot progressive, I can make marginally effective DVDs, but if I shoot interlaced, the resulting DVDs are totally hopeless. By the way, I have also made DVDs (as is possible with Adobe) both 30p and 60i, with only marginal results at best.

3. I make everything, BD and DVD, MPEG2 now, but in ALL the testing I did during the last year, I'll bet I tried out some AVC at one time or another, but having finally given up the nightmare of all that testing and hundreds of pages of notes, I destroyed all the records except for the results that worked best. Taking a stab at AVC at least for a short sample is certainly worth it, and I'll give it a try sometime in the next few weeks.


Finally, you said, "I would also like to get a 5 second clip of your problematic raw footage to see if taking it through my workflow would produce the same results." --- Yes, I'd be very interested and most appreciative if that could be done, and if by raw footage you truly mean a BPAV file, I could shoot just a five second segment of a slow pan with fine detail in the frame, but I am not at all acquainted with how I could get a file of that size to you other than paste it to a DVD and mail it, or what? Or by raw footage do you mean an avi file, or what? What are your thoughts about this?

Best regards,

Bill Urschel

[Perrone Ford]

I am learning more each time you post.

1. This is a significant development. And it prompts a question. Have you taken one of the discs you've created to 2-3 other BluRay/HDTV setups and observed your work on those? The reason I ask this, is that from your description, it sounds as if your problem lies in the display of your footage. As we begin to reduce detail, either by shooting HDV, or by taking the footage through Cineform which does lose some detail, your footage appears to look "better". This is the opposite of what SHOULD happen. Your footage should look best right out of the camera.

To that end, I would ask you to connect the component cable to your EX1, and to your HDTV and point it at some fine print in a magazine or print out a test chart. See how that displays on your monitor if you set the camera in 30p, 60i, and HDV 1440x1080 mode. If your footage straight from the camera looks like it's shimmering and aliasing, then the problem is the display, and not the workflow. This is assuming that you've got the camera set up well.


2. The EX1 has less resolution when shooting interlaced. Again this leads me to believe that your display may be at fault since the results seem to improve as we degrade the resolution of the source.

3. AVCHD may not make sense based on what you've given me here. It's ability to reproduce finer detail at these kinds of rates may well exacerbate the issue. And this may be why you tossed the idea out in your notes.

And finally, yes, I mean a raw BPAV folder structure. But honestly, I only need the .mp4 file from within that structure. If you shoot the recording, I'll handle the upload.

-P

[William Urschel]

Dear Perrone:

Quickly, as I'm getting ready to wind things up to take my wife to hospital tomorrow for outpatient procedure, and then almost immediately the relatives show up for a week (!),
I'll try out your suggestions re component out hook-up. Though I wonder - the 42 inch LCD flat panel and 109 inch projected screen I use for testing both show up everything else well, except for the results from the EX-1, but also from a new Canon hi def consumer camera - further, I have the same distressing results on ALL of my customers' LCD and plasma flat panels and front projectors, unless I render out the flicker filter, which does contribute to my having to play out the timeline twice through Cineform. And by the way,
particularly on that 109 inch screen, multiple processing with Cineform does NOT reduce the resolution according to 1) the resolution charts I've shot, with and without Cineform, and 2) visual observation of resolution according to independent testing - that's why I use Cineform to process and to back-up everything I do! Or have I missed something here also?

Soooo, I'm off this board for a while! But you'll see me when the subject title comes to the head of the list again. And again, thanks so much for your thoughts and interest.

Best regards,

Bill Urschel

[Steve McClean]

Thanks for the in depth reply Bill. You're right, I didn't mention that I am currently editing on a Mac with Final Cut Pro 6 and I've read a ton of postings on different ways to get the best looking DVDs from the EX1/3 footage but my concern is more to do with the line twitter being on the raw footage.

I can understand problems with artifacts being introduced as a result of compression methods and mixing interlaced and progressive footage etc but what I am really frustrated about is that these twittering and shimmering problems appear to be on raw footage played directly from the camera onto my LCD TV before I ingest anything into my editing suite - I'm not sure there have been many suggested in-camera solutions to this other than a) turn down the detail (which sounds like it is not a favoured solution for either you or me) or b) be more selective in what you film (which makes sense but isn't always controllable if you are covering an event for example where you have little choice over what is in the background).

Given your experiment with the cheap camera, have you basically concluded then that this is just an unavoidable problem that comes with the EX1's CMOS chip? Does anyone else have any other suggestions on how to help avoid these problems with the raw footage or do you all just have the same issues with the raw camera footage, accept it and try to fix it in post?

Thanks for your help
Steve

[Perrone Ford]

It makes no sense that it would be the CMOS chip. It would make a LOT more sense if it was the codec, the lens, or something else.

So let me ask this. Is the problem also evident if you shoot in 720p? I would ask if you could use an external recorder out of the SDI spigot to eliminate the codec, but that may not be an option available to you. So Steve, do YOU see this issue if you hook your component cable up to your display straight out of the camera? This output should be pre-codec, but will come after the sensor. If that's clean, then you have an issue with the codec most likely, and not the chip or the lens.

[Steve McClean]

Hi Perrone, yes I do see these issues coming straight out of the camera via component cable straight into the LCD TV - and 1080p and 720p both have the same issues as far as I can see. I've tried taking feeds from the camera to a couple of different LCD screens and its shows up on both. Maybe I have just been unlucky of course with these particular displays and a more extensive trial of different TVs would show different results but therein lies the frustration - when you hand over your final footage to the client you obviously have no control over what they watch it on which makes quality control rather difficult... To be honest, I really hope that it is just my display(s) but it certainly sounds like I am not alone with this problem which is why I was wondering if I was missing something (other than reducing the detail) that might help me improve the results in-camera...

Steve

[Perrone Ford]

Do you have a sample of your footage anywhere? I'm curious as to what it is you're seeing. Also, would you share your picture profile if you are using one? I've posted one of my "Flat" one's around here somewhere, maybe try that one if you are so inclined so we can compare apples to apples.