Auto depth from stereo image, for 3D CGI integration? at DVinfo.net

Matt Faw · January 27th, 2012, 07:24 PM

Dear all,

I'm creating a short doc project that includes a good deal of 3D CGI inserted into stereo live-action shots, and even more stereo live-action (shot in front of greenscreen) into CGI environments. The latter is accomplished by bitmapping one eye at a time onto a plane in 3D Studio Max, placing that plane into the CG environment, and rendering out using the 3Dhippie stereo rig.

I'm very happy with the results so far (especially considering how brand new I am both to 3D Studio Max and to stereo). However, I can imagine a solution that would speed up my workflow considerably, and it's hard to believe that someone else has not yet come up with it. Hopefully someone on this forum has heard of such a solution that could help me out.

Specifically what I'm wondering, is if any of the 3D CGI programs (or plug-ins) will analyze the depth of a stereo image, and create a simulated 3D environment (or object), based on that analysis? I've seen a beta program that creates a depth map from a 2D image (based on heuristics about horizon, perspective, etc.), so it seems like it should be very easy for a computer to come up with an accurate depth map based on a stereo image.

Applications for this are manifold: I could take a live stereo shot, and fly a CGI object around it, including passing behind objects in the foreground, because the computer should know where those foreground objects' edges are, and what their depth is. Also, I could take one of my live-action greenscreen actors, and create a 3D person to interact in a CGI world, without having to fool around with the planes, etc (obviously, that person would not be true 3D, in the CGI sense, because it wouldn't have a backside, but it could still have some perspective shift for camera moves). Plus, it seems like auto-rotoscoping and match-moving should be a breeze, since the computer could identify edges so easily.

Suggestions, anyone?

Arnie Schlissel · January 28th, 2012, 02:46 PM

Short answer: no. But Nuke with the Occula plugins (aka Nuke X) has many workflow enhancements to speed up working in stereo 3D.

Tim Dashwood · January 29th, 2012, 07:28 PM

This is absolutely possible. I've been experimenting with depth mapping from stereo images for a long time. In fact, the first few beta versions of Stereo3D CAT had a depth map feature but most of the computers we tested could only run the pixel-comparison algorithm at about 1 frame per second, which is too slow for live preview, so I removed the feature from v1.0.

I'm putting finishing touches on the next version of our plugins so I may revisit the code we developed as a post tool.

Paul Newman · January 30th, 2012, 03:04 AM

+1 for Nukex and Ocular

Paul :-)

Matt Faw · January 30th, 2012, 08:43 PM

Thanks for the replies, everyone. Tim, I'm looking forward to seeing what you come up with next.

Also, thanks for the Nuke X and Ocula recommendations. Nuke is way out of my price range, but I know they have some free trial and student deals that I may be able to ride for a little while. Currently, I'm pulling my greenscreen keys on After Effects, and had planned to do my final composite there, but Nuke does sound like a great alternative.

January 27th, 2012, 07:24 PM	#1
Matt Faw Regular Crew Join Date: Nov 2011 Location: Los Angeles, CA Posts: 139	Auto depth from stereo image, for 3D CGI integration? Dear all, I'm creating a short doc project that includes a good deal of 3D CGI inserted into stereo live-action shots, and even more stereo live-action (shot in front of greenscreen) into CGI environments. The latter is accomplished by bitmapping one eye at a time onto a plane in 3D Studio Max, placing that plane into the CG environment, and rendering out using the 3Dhippie stereo rig. I'm very happy with the results so far (especially considering how brand new I am both to 3D Studio Max and to stereo). However, I can imagine a solution that would speed up my workflow considerably, and it's hard to believe that someone else has not yet come up with it. Hopefully someone on this forum has heard of such a solution that could help me out. Specifically what I'm wondering, is if any of the 3D CGI programs (or plug-ins) will analyze the depth of a stereo image, and create a simulated 3D environment (or object), based on that analysis? I've seen a beta program that creates a depth map from a 2D image (based on heuristics about horizon, perspective, etc.), so it seems like it should be very easy for a computer to come up with an accurate depth map based on a stereo image. Applications for this are manifold: I could take a live stereo shot, and fly a CGI object around it, including passing behind objects in the foreground, because the computer should know where those foreground objects' edges are, and what their depth is. Also, I could take one of my live-action greenscreen actors, and create a 3D person to interact in a CGI world, without having to fool around with the planes, etc (obviously, that person would not be true 3D, in the CGI sense, because it wouldn't have a backside, but it could still have some perspective shift for camera moves). Plus, it seems like auto-rotoscoping and match-moving should be a breeze, since the computer could identify edges so easily. Suggestions, anyone?

January 28th, 2012, 02:46 PM	#2
Arnie Schlissel Major Player Join Date: Jul 2006 Location: New York NY Posts: 322	Re: Auto depth from stereo image, for 3D CGI integration? Short answer: no. But Nuke with the Occula plugins (aka Nuke X) has many workflow enhancements to speed up working in stereo 3D. __________________ Post production is not an afterthought! www.arniepix.com

January 29th, 2012, 07:28 PM	#3
Tim Dashwood Wrangler Join Date: Aug 2005 Location: Toronto, ON, Canada Posts: 3,637	Re: Auto depth from stereo image, for 3D CGI integration? This is absolutely possible. I've been experimenting with depth mapping from stereo images for a long time. In fact, the first few beta versions of Stereo3D CAT had a depth map feature but most of the computers we tested could only run the pixel-comparison algorithm at about 1 frame per second, which is too slow for live preview, so I removed the feature from v1.0. I'm putting finishing touches on the next version of our plugins so I may revisit the code we developed as a post tool. __________________ Tim Dashwood

January 30th, 2012, 03:04 AM	#4
Paul Newman Major Player Join Date: Nov 2007 Location: Agua Blanca Ibiza Posts: 305	Re: Auto depth from stereo image, for 3D CGI integration? +1 for Nukex and Ocular Paul :-) __________________ Another Sunny Day in Ibiza

January 30th, 2012, 08:43 PM	#5
Matt Faw Regular Crew Join Date: Nov 2011 Location: Los Angeles, CA Posts: 139	Re: Auto depth from stereo image, for 3D CGI integration? Thanks for the replies, everyone. Tim, I'm looking forward to seeing what you come up with next. Also, thanks for the Nuke X and Ocula recommendations. Nuke is way out of my price range, but I know they have some free trial and student deals that I may be able to ride for a little while. Currently, I'm pulling my greenscreen keys on After Effects, and had planned to do my final composite there, but Nuke does sound like a great alternative.