Stereo3D Toolbox with FCE

[Pavel Houda]

I am very hopeful about the potential of this tool. I am trying to use it with FCE and am struggling. Could someone describe quick workflow so I don't have to struggle too much? I am capturing the video with two AVCDH camcorders and will be outputting to DLP TV. I can use FCE for 2D work, so just a little push would make the whole experience satsfying a bit faster. Thank you in advance. The original installation and startup demo dissapeared (I realize that the installation is little different now, and I have the software installed). This is really exciting! Thank you Tim.

[Tim Dashwood]

Hi Pavel.

I'll repost what I had emailed you earlier today but I will also describe a very basic process if you are just experimenting with some shots (a workflow with more steps or additional software like neo3d would be required for editing a much larger project.)

First off let me say that I do not own Final Cut Express so I'm not familiar with what features are missing that I would normally use in Final Cut Pro. For example, does Final Cut Express have the ability to subclip or does it have the media manager? I'm not sure.

Here's some background on DLP TVs.
We added checkerboard mode specifically for Mitsubishi's DLP 3D mode. For those interested here's the white paper on DLP's 3D mode.
Most of the DLP 3D-capable monitors have a native resolution of 1920 x1080 so it is best if you shoot in that mode or uprez your footage to that resolution. (nesting can also be used to do this.) I'm not sure if Final Cut Express can output 1920x1080 to the second monitor (via "Digital Cinema Desktop Preview - RAW" or a capture card.) If it does then connect your DLP and make sure the resolution is set for 1920x1080. If you are using your graphics card then you may have to go into display settings and check the resolution.
If FCE doesn't have the ability to output a live signal then you can still master a file in 1920x1080 checkerboard and then burn a Blu Ray (adobe encore cs3 or cs4 seems to work well for bluray.)
The shutterglasses are used with DLP 3DTV to sync to the 120Hz output. (60Hz per eye)
There is a little 3-pin "3D" sync port on the back of the DLP 3DTV that sends the signal to a set-top IR transmitter which in-turn sends the sync signal to the active shutterglasses being worn by the viewers.

So now onto the "basics for beginners of shooting footage for 3D with HD cameras" (preferably 1920x1080) and processing it in Stereo3D Toolbox and FCE (or FCP.) [This may seem long but most of it is just common sense stuff I'm including for absolute beginners.]

1. Set both cameras to the exact same settings and turn off image stabilization (this will screw up your convergence later on. Trust me... I've learned the hard way!)
2. Mount the two cameras as close together as you can and always use a wider field of view than you need, you will be zooming in a little bit anyway when you converge later. I recommend that beginner's use a parallel configuration, meaning both cameras are pointing directly straight ahead. Do not "toe-in" to converge on your subject. It will cause you more headaches later.
3. If your screen is big (42" or bigger) then use the 1/60th rule to determine how close your closest subject can be to your cameras. All this means is that you measure the distance between the centre of your two lenses (interaxial distance) and multiply that by 60. If the the distance is 3 inches then you should stay at least 180inches (15 feet) away from your closest subject. (For those shooting for YouTube 3D or the small screen you can use a smaller 1/30 factor.)
4. Use the IR remote for your cameras to trigger them to record and to stop. This may not be perfect but it will get you close.
5. Use a clapper slate or simply stand in front of your shot and clap your hands once. You will use this to confirm sync of your cameras. If you are using consumer cameras that don't have genlock don't expect perfect sync and don't shoot any fast action.
6. Ingest your clips into FCE or FCP. If you are using log and capture I recommend you log everything first and use the clapper as your in point on all clips. Be sure to clearly mark which are left and which are right!!!! Then batch capture.
7. If you only have FCE and not Motion or AE it is important that your clips are starting at the same moment in time - to the frame. You can subclip them to the clapper (CMD + U on FCP) or mark ins to the clapper and use media manage to trim the excess. Also if you didn't record in 1920x1080 and your DLP monitor is 1920x1080 then now would be the time to uprez the footage with media manager.
8. Assuming you've worked out that your left and right clips actually start at the same time it is now time to create a sequence (using the same settings as your clips) and insert the left clip or left subclip.
9. Apply the Stereo3D Toolbox filter to the left clip and open it (by double-clicking on your clip or selecting it and hitting RETURN.)
10. Now drag the right clip into the "right" image well. The default mode will be side by side. You have just successfully merged your right and left clips. It is a good time to scroll to the clapper and make sure both sides are in sync. If they aren't then go back a few steps and check the sync. (BTW this whole sync thing is much simpler with professional systems that can genlock and sync TC!)
11. Now switch the output mode to superimpose and then turn on the convergence tools.
12. Use the convergence tools to first check that the vertical alignment is perfect. This is a must to avoid eye strain. Use something level in the frame for reference. If one camera was tilted you can use the Z or individual rotation controls to fix it. If you had tried to zoom in a bit and one camera was zoomed in a little more than the other use the left and right zoom controls to compensate.
13. Once the vertical alignment is good you can decide what is going to be converged. This is where you use the X axis convergence controller. Whatever object is perfectly converged will appear to the viewer to be at the screen plane, so choose wisely. Also keep in mind that for good 3D any objects between the converged subject and the camera should not be occluded by any edge of the screen. This will cause your brain to go "wait a sec...why are my eyes telling me this object is closer to me than the screen but it is being cut off by the screen?" This is why it is OK to have an object such as a sword, knife, stick, gun, fist, whatever appear to come out of the screen as long as it doesn't touch the edge.
14. When you are happy with the convergence settings you can turn on "autoscale" to get rid of any black borders created by your convergence settings and it is time to test it out. Set your output mode to checkerboard and make sure FCE or FCP is outputting Digital Cinema Desktop Preview in RAW mode to the DLP monitor, which should also be in its raw native resolution.
15. Put on your shutterglasses and enjoy!

By the way I skipped over the ability to colour match the left and right eyes if your cameras weren't set exactly the same but I'll assume they were.

If you want to edit your clips down into a new sequence then use the above methods to make a new sequence for each clip, but set your output mode on each to side by side, over/under or interlace. This will be the intermediate 3D mode used for editing your clips. Drag the sequences (which are stereo clips now) into a new timeline for editing. This is called nesting. Apply another instance of Stereo3D Toolbox to those nested sequences and set the input mode to whatever you used as an output mode before. At this point you can use checkerboard for your output and edit away.

[Pavel Houda]

Thank you very much for posting the information and instructions. I used it to do some experimenting, and it was of tremendous help. I can say that the tool works great on top of the FCE. I took just about all the missteps you described, because I started before your post, so all the hints are really important to follow. I started with mismatched camcorders (I was just to try the concept, understand the fallout of it, and know how to correct that), and I left one of them on optical stabilization by mistake. That caused a major problem, easily fixeable with new footage. To the l/r alligned, I asked my subjects to clapp their hands, and alligned to that event. The FCE can generate subclips (cmd-U), so I could sync the pairs quite well, considering that I couldn't genlock the two camcorders. I did learn about the minimum distance required the hard way. Otherewise I had really no big issues with the Stereo3D Toolbox plug-in. FCP with it's ability to match the colors, etc., would have been great (vectoroscope, etc.) but as I said, I can tolerate the mismatch problem for this experiment. I could check the results only with anaglyph glasses so far. That provides much less quality than the DLP or some other polarization method can provide, having color and other annoying side effects, but I achieved excellent 3D results for sure. My Mac will need to be moved close enough to the TV's to get the checkersoard at 60p to the DLP-TV's, I didn't have the time and energy to do that yet. I did try to burn blu-ray discs, but I only have Toast to do it with, and it modifies the checkerboard a lot, when it compresses the .mov files, so that the pattern gets modified and the effect just doesn't work. It displays something that looks like SD scan pattern. It does it with several blu-ray players as well as with PS-3, so I am pretty sure that the problem is with the blu-ray authoring. I don't have CS3 or CS4, so I will have to come with some other way to try this. So far I only saw the DLP 3D demos run by HDMI outputs of computers, generally with nVidia graphic adapters, I never saw a blu-ray player output done with the DLP patter, so I am not sure if it can be done. If someone did it, I am interested in the "how", please.

So to summarize, I am almost there and I am sure it will work well. I am very thankful for your patience, instructions and help.

[Tim Dashwood]

Blu-Ray uses two primary types of compression: H.264 and Mpeg2. I haven't done extensive experimenting in this area but I would imagine H264 at the highest bit rate would work best. Mpeg2 in Blu-ray seems to still be 4:2:0 chroma-subsampled so a lot of data will be thrown away from the checkerboard pattern and cause ghosting. I believe H264 can maintain 4:2:2 chroma subsampling, which still isn't perfect, but should perform better.

You will also have to make sure no scaling happens (1440x1080 vs 1920x1080) because it will also ruin the pattern.

Unfortunately I don't have access to a DLP 3DTV to test this complete workflow and have relied on beta testers such as yourself to provide feedback.

I will probably do a blu-ray test output soon and upload it here.

[Gregory Stoner]

Follow up on Tim note, currently the Blu-ray only support Rec 709 color space, 4:2:0 Chroma Subsampling, 8 bit luminance encoding for all the currently support codec MPEG-2, H.264 and VC-1.

PS Tim you developed a very nice S3D plug-in.

[Pavel Houda]

Yes, I tried to generate several MPEG-2 and AVC files on the blu-ray discs, at different settings, but no luck. Some more data. Tonight I connected the iMac to the DLP-TV via HDMI at home and got the same results. I tried both QT and FCE outputs to the TV. QT as: View>Present Movie> Full Screen and Actual Size, overscan on/off, different rates. First I double checked the System Settings; the output said 1920x1080 @ 60Hz. Tried overscan and no overscan, 30 Hz. Still no results. I have a PC with nVidia card and 3D test pattern running on that, all is well, but the iMac is outputting some sort of scan line like faint pattern, not smooth vertically like the nVidia raw pattern. Actually the same results as the Blu-ray discs exhibited. I am afraid something is wrong with the checkerboard pixel delivery to the TV. It is quite low end, unforgiving requirement, synchronized to the micro-mirrors. I see both left and right images and those scan like lines. Like I said, the PC output works fine, so I am sure there is no issue with the TV, glasses, etc. Something is wrong with the pattern or timing, even though the timing setting is fine. The setup would had the 59.9Hz option grayed out, so I couldn't try that. I originally didn't have time to check direct connect (yesterday), so I thought the problem was due to the disc compression, but I doubt that conclusion now. I am not sure what I can try now. Please let me know what you think.

[Tim Dashwood]

Thanks for the clarification on 4:2:0 in Blu-Ray Greg.

Pavel, is it possible for you select RAW instead of Full Screen in the FCE output?

Overscan should definitely be on and you should be able to select 60Hz (or possibly 120Hz) as the sync rate.

[Pavel Houda]

Tim, the FCE "Video Out" options are: Canvas Playback, Main, Preview and Full screen. No Raw. All the other settings are just like your screen shots.

[Tim Dashwood]

Hmmm. I'm concerned that if "full screen" tries to do any scaling at all it will throw off the pattern.

BTW, there is a simple way to create a test pattern with the plugin. Just apply it to any footage or slug and select "Left & Right Image Wells." You will see the words LEFT and RIGHT (one black on white and the other white on black.) You can then select checkerboard output and you should literally see a black and white checkerboard pattern (when viewing at 100%.)
These placeholders can be used to check the left/right sync for any output.


One last thing I just thought of... you used the same HDMI input on the DLP TV that you used when testing with a PC right? I think on most of these 3DTVs only one particular output will send sync to the 3-pin output.

[Pavel Houda]

I did try the same port. The IR sync was getting out. You can see it with the glasses, since if the IR is missing, the glasses become more transparent, because the IR shuts one eye off alternately, so the display goes dimmer and more contrasty with the IR present. On this particular model (73833) any HDMI port can be used for 3D. It is selectable by menu, and the port has to be set as "PC" or "Game" and 3D needs to be turned on and the frames going at it must be 1920:1080x60p, with the checkerboard pixel arrangement for L/R eye. I agree that any scaling or processing will through the pattern off. Maybe it is not in the cards.... Thank you.

[Pavel Houda]

Tim, finally I did a clip end to end. I had to convert side-by-side output into windows stream, where I used a player to output the checkerboard to the DLP TV. It now all works great, way better than any anaglyph. It retains color black is blacker and the contrast is actually better. At 120Hz the output is just as smooth as any normal viewing. Your plugin is just awesome! Had some problem with over/under, but not a priority at this time. Just that the watermark covers 1/3rd of the frame. Thank you for your support.

[Tim Dashwood]

That's great that you got it to work but I'd still like to work out exactly why you couldn't get checkerboard out of the mac to work with the DLP 3D mode.
When you get a chance can you note the sync rate and exact resolution when Windows XP/Vista is connected and working with the monitor? I wonder if the sync rate needs to be at 120Hz and the mac is syncing to 60hz?

Also, the watermark will cause some issues with the over/under mode simply because of its placement. We decided on the lower left because I figured most of the users who wanted to use it for free would be outputting in anaglyph modes for Vimeo.
Unfortunately the watermark is a function of FxFactory's licensing system so I can't program control of its placement based on output mode.

BTW, I added a "Getting Started" video tutorial to the website that demonstrates one possible workflow.

[Pavel Houda]

On the PC side I am using "player" by 3dtv.at. I can input either SBS or O+U and let the S/W output the checkerboard. If works well, but the logo is especially annoying when I input SBS. The frame rate required by the TV is 60p, but the frame buffer pixels over the HDMI must be exactly alligned with the pixels on the TV, and always, consistently so. Any scaling, pulldown, or other cause of pixel misalignment or L/R reversal, and the 3D effect is gone. During the frame period, the TV internally switches during the 1/60 sec frame period, so the internal rate is 120Hz, but externally we just need to input full L/R checkerboard at 60p. There are couple of documents on this here: http://dlp.com/downloads/DLP%203D%20...Technology.pdf and http://dlp.com/downloads/Introducing...Whitepaper.pdf .

When I burn a DVD, I use Toast 10, but I think it generates 24Hz output, so the 3:2 pulldown by the Playstation damages the 3D effect (I think). I have no way to prove it, Roxio doesn't specify (like many others), what they are really doing, and there is no control of these parameters. I just know for sure, that I see both frames with either eye.

I also managed to output the FCE edit using MP4 file and to play it on PS3, and the 3D effect is there, but for some reason the image quality is not too good.

I appreciate all your support and the tutorial. Best regards,

Pavel

BTW, I know that you know all that I mentioned, I just wanted to explain what I did and why.

[Pavel Houda]

Snow Leopard improved OS/QT so much that it just removed the MP4 checkerboard pixel aligned output capability, stereoscopy, so I am back to SBS output, conversion to wmv/vc-1 and output via the PC and the 3DTV.at player, with the awful watermark interference to go with it.

[Tim Dashwood]

Just out of curiosity is your sequence set to interlace? If so set the field dominance to "none" and see if some of your issues go away. Your watermark should definitely become smaller and it might even solve the other issue you are having.

BTW, we're still testing in Snow Leopard and NI is recommending that anyone who uses After Effects not upgrade yet because of a known conflict with 3rd party plugins that Adobe and Apple need to work out.

There is also apparently an issue with Motion 4 and Snow Leopard 10.6.0 where it can "become unresponsive." 10.6.1 will fix this according to MacRumors.