Light Field Technology!

[Nicholas de Kock]

Have a look at light field technology it's going to change the way we take photos! Pretty amazing stuff...
Lytro's light field camera captures 'unprecedented' images, lets you choose focus later -- Engadget

[Zach Love]

Wow, finally coming to the consumer market. I've been wondering when this would hit since I first read about it in 2005.

Light Field Photography with a Hand-Held Plenoptic Camera

Can't wait to see the model that ships & even more can't wait until they get it into video cameras. I think this could even be great for green screening w/o a green screen. Just tell the software that everything more than "X" number of feet away, delete.

This has the potential of making the "DSLR revolution" a forgotten memory. Who wants shallow DOF recorded, when you can have any DOF in post?

[Nevin Styre]

I wonder if 2 different depth ranges could be in focus, while the areas in front, behind and between are out of focus.

[Dave Blackhurst]

Dang, that really IS a game changer...

[Zach Love]

As for the game changer, I think all the cameras they have for this are pretty low resolution, like 1-2mega pixels (I could be wrong). Not sure if it has to do with the lens array, or the amount of information that has to get passed through.

If it is the lens array, then making this for video seems not too far away. If it is the amount of information that keeps the resolution small, then video would be farther in the future, as they'd have to get the frame rate up.

Still the amount of R&D in cameras today & how quickly they're improving & getting cheaper, nothing is too far away to wait for.

Originally Posted by Nevin Styre

I wonder if 2 different depth ranges could be in focus, while the areas in front, behind and between are out of focus.

I think this would be possible, as it would probably just making a good algorithm in the software to handle it.

I think that it would be great to adjust an over the should shot 2-shot, so the background stays locked at a constant out of focus; subject stays locked in focus; but the interviewer's focus can be adjusted.


I can't wait to get my hands on one of these cameras & start playing around with it. Game changer I don't think is the right phrase, it seems bigger than that if it can do half the things I'm thinking about.

[David Heath]

Sorry to be a wet blanket, but you rarely get something for nothing - and that is the case here.

The principle has been around for a while - Plenoptic camera - Wikipedia, the free encyclopedia - and whilst perfectly sound, the flexibility (such as being able to refocus) comes at a price. In this case, it's fundamentally that for a given image resolution, the data etc corresponds to a far higher resolution image. From the wikipedia link:

The Adobe light-field camera is a prototype 100-megapixel camera that takes a three-dimensional photo of the scene in focus using 19 uniquely configured lenses. Each lens will take a 5.2 megapixel photo of the entire scene around the camera and each image can be focused later in any way.

Hence, for the foreseeable future, even with a high resolution base (and big file sizes), the output images are likely to be quite low-res by todays standards, and don't expect video at decent resolutions with current technology.

[Dave Blackhurst]

Read the info on their web site - it appears that some fancy mathematics have been applied to reduce the hardware significantly, and their samples don't look too bad... They specifically refer to reducing a roomful of cameras to a pocketable device, and they apparently have photographers "in the field" working with these right now...

Too early to know much, but it certainly could represent a breakthrough technology presuming resolution is within expectations. They do claim to be shipping in the relatively near future.

As for the amount of data, there have been several algorithms I've seen suggested that might drastically reduce the amount of data required to achive acceptable resolution - offhand, I can see several of the "academic" anouncements of experimental techniques I've heard of coming together to make this "work".

It will be interesting to see whether video is on their radar - at the moment it appears stills are the "focus", but it would be quite interesting to apply the principles to video... if they can apply the concepts across time instead of in a limited timeslice, it should be feasible, if not now, in the future with faster more capable DSPs...

[David Heath]

Originally Posted by Dave Blackhurst

Read the info on their web site - it appears that some fancy mathematics have been applied to reduce the hardware significantly, ........... Too early to know much, but it certainly could represent a breakthrough technology presuming resolution is within expectations.

As a lot of replies on their blog say, the site is very lacking on specs - and noticeably to do with resolution, or even whether zoom lenses would be possible. I'm not doubting that it will work - but strongly suspect that we may be looking at something geared far more towards the consumer point-and-shoot market than serious photography.

Which is exactly where the technology is likely to be most relevant. Take a 16 megapixel pixel image, and derive something like a 1 megapixel refocussable image (with very large file size), maybe?

They specifically refer to reducing a roomful of cameras to a pocketable device,

Which again comes at a cost. The "roomful of cameras" refers to being able to produce a result whereby you can gain an effect of walking round (and over, and below) a scene, as well gaining a conventional 3D image, or a 2D iamge with variable depth of field - depending how the data is processed. Such techniques are not new - for example, a museum may use them to record a statue, say, in unprecedented detail.

This company is likely talking about a product with very limited view angle etc by comparison - but which may be just what some customers want..

The clue may be in the FAQs of the company:

What began in a lab at Stanford University has transformed into a world-class company, forty-four people strong, sparkling with talent, energy and inspiration.

Wikipedia refers to the Stanford research -

A team at Stanford University used a 16 megapixel camera with a 90,000-microlens array (meaning that each microlens covers about 175 pixels, and the final resolution is 90 kilopixels) to demonstrate that pictures can be refocused after they are taken.

Now I don't doubt that this has progressed in the six years since then, but there are good technical reasons why the final product is far more likely to be something intended for relatively low-res point and shoot - not what most on this forum would call serious photography. But I wait to see more detailed information on their website.

In the meantime, you may be interested in the original Stanford University research paper from 2005 - http://graphics.stanford.edu/papers/...era-150dpi.pdf - one of the authors is the founder and CEO now listed for Lytro.

[Jon Fairhurst]

Taking this to the extreme...

Imagine a big, empty cube that you would bring into a room. It would have Light Field technology, HDR, RAW recording, and virtual camera positioning. Turn it on, hit REC, and yell, "action." The focus, exposure, color balance, and camera position/movement could all be chosen in post.

Some would say that this takes away from the art of filmmaking. I would disagree. It would decrease the skills needed to shoot, but could allow more artistic end results. It would also save a ton of time. There would be no more botched focus, exposure, colors or unintentionally jerky motions. We wouldn't have to move and level tracks, change lenses, and on and on. It would ONLY be about the performance of the actors.

This would be killer for composites as well. Camera motion tracking between layers would be perfect.

But for now, I'll focus, adjust the camera settings, and deal with dollies and jibs and imperfect movements. And I'll keep saying "cut! We need to do that again!"

[Zach Love]

David Heath, remember the Nikon D1? A "professional" camera with 2.7 megapixels, hardly usable resolution for any "professional" still photo.

But how long did it take after that to get DSLR cameras with respectable resolutions?

If I can get a 1megapixel light field camera this year for under $300, I'd go crazy & start playing around with it as much as I can. It would be a novelty at that resolution & not great for paying jobs, but still fun to play with.

And if it sells well enough, you can bet that all the major camera manufactures will be releasing better cameras as soon as they get the chance to take the money from buyers. (I'd be shocked if they weren't already doing R&D).

[David Heath]

Originally Posted by Zach Love

David Heath, remember the Nikon D1? A "professional" camera with 2.7 megapixels, hardly usable resolution for any "professional" still photo.

But how long did it take after that to get DSLR cameras with respectable resolutions?

Yes, that's a good point. The point about the D1 was that it was targeted primarily at photojournalists, and for news the resolution was "good enough", but the time saving by no processing, and being instantly able to send the results electronically was key. But what the Stanford team produced was 0.09 megapixel output resolution (292x292), and that from a starting point of 16 megapixel. Tricks can obviously be done by trading off refocussing coarseness for resolution, but I think even 1 megapixel output resolution may be highly optimistic. We'll see, but something like 640x480 (0.3 megapixel) max might be more likely - and more than good enough for "point and shoot" and display on a smartphone screen.

The 16/0.09 is a ratio of nearly 200:1. That means to get an output resolution of even 2.7 megapixel would mean a sensor of about 500 megapixel - and video would mean reading and processing it 60 times each second. Never say never, but I don't see it happening soon.
[/QUOTE]If I can get a 1megapixel light field camera this year for under $300, I'd go crazy & start playing around with it as much as I can. It would be a novelty at that resolution & not great for paying jobs, but still fun to play with.[/QUOTE]
I don't disagree with any of that, just don't expect too much, too soon.

[Dave Blackhurst]

David -

Think on this for a moment. If I have grasped the fundamentals properly... the design uses MULTIPLE lenses with multiple "views" of an object from ever so slightly different angles... meaning there is an ABUNDANCE of data sets, which could easily be interpolated into a MUCH higher resolution image with some fancy math.

I recall an article (think it was linked here?) where they were using some "fancy math" to take relatively small file sizes and interpolate the details (something which still baffles me, as I don't see how you could get something from "nothing"!? Here there IS something though!) in order to recreate very high resolution, high detail images.

So let's take those two "technologies", mush them together, take a large number of ever so slightly different data sets, combine with algorithms that would analyze all those data sets at a high rate of speed, consider all the similarities and differences and "compute" a resulting image (looks sort of like what they are hinting at). You're not limited by "film", or sensor resolution, your only limits are data throughput and the efficiency of your algorithms in handling the data, and perhaps the quality of your lens(es).

In effect you get infinite focus (check), nearly infinite exposure (they claim very good low light), and nearly infinite theoretical resolution... gives me a headache thinking about it, and the computing horsepower in that camera would be pretty impressive, BUT methinks that's how they are doing it.

From a video standpoint, it may become rather problematic simply because scaling the calculations and horsepower required to take a single image enough to handle 30-60 images per second... well, it is mind boggling... but give it time, it's easy enough to do if the foundation is there, just have to wait for computing horsepower to catch up!

The more I contemplate this technology, the more intrigued I get - there is not a huge leap from this to "holocameras" capable of complete 3D capture, and all sorts of other crazy stuff!! Even crazier, what if you could "reverse" the algorithms to make a holo projector!? Eh, I'll take the camera since it looks like it's closed to "baked" already!


FWIW, the above is my "guess" based on pretty limited review of their website and that old technical paper, I could be completely off base, so don't take it as gospel or anything beyond a WAG (Wild A** Guess) at what is coming... but it's fun to contemplate!

[Charles W. Hull]

Originally Posted by David Heath

The 16/0.09 is a ratio of nearly 200:1. That means to get an output resolution of even 2.7 megapixel would mean a sensor of about 500 megapixel - and video would mean reading and processing it 60 times each second. Never say never, but I don't see it happening soon.

Except that it is a little hard to define what would be meant as video processing, since much of the image processing with LFT is done in post. You could imagine shooting with just a huge amount of high speed storage, and then doing all the video processing in post.

[Dustin Moore]

All of these plenoptic cameras can be seen as a hack to do bokeh in software rather than
just using a wider aperture. The bonus of simulating a wide aperture in software is that you
can pick the focus plane after the fact.

It helps to understand where bokeh comes from but I think you can do a thought experiment
and get an idea for how this all works. I used the following explanation on
another website but maybe it will work better here:

Imagine you have a stereo rig with reasonably normal lenses that are almost zero
radial distortion and have the left eye and right eye files on two separate tracks in
your favorite video editor. Set the track on top to be 50% opaque so that your
output video is just both frames arithmetically averaged together.

In this new output video, most things will have two ghosts. Say you shift the
top frame left a few pixels until some of the objects loose their ghosts and
line up. You will probably find that the objects that it are possible to line up
(remove the doubled ghosts) are the same distance from the camera,
Congratulations, you have just focused your plenoptic camera at a particular depth.

You will notice though that everything in front of and behind the plane that is
in focus has two ghosts. This "blur" is basically the bokeh of your system. It
arises from the fact that the left edge of your new aperture views reality from
a different angle than the right edge of your aperture (the right eye camera).

Now this bokeh sucks. It's not a nice round diffuse ball, it is two points that tend
to make everything just look ghosted rather than actually blurred.

If you want a nice bokeh, use 20 cameras arranged on some sort of regular
two dimensional grid. It's more difficult to figure out how to translate each
view left and right so that the depth plane that you want is in focus, but it is
doable. Hard to think about but hopefully this makes sense.

If you want a whole lot of bokeh, add cameras that are farther from
the middle of the grid. If the bokeh is ugly (uneven/point like) add more cameras
between the cameras already on the grid.

You could also just make one big lens that was the size of the
frame holding the grid of cameras and get the same bokeh, but this could
be heavy and unwieldly. But the reason that the big aperture has big bokeh is
because the edges of the aperture are far apart they see different views of the
scene that don't line up right. Hence when they get averaged together, they
come out blurry and turned into bokeh.

One note:

meaning there is an ABUNDANCE of data sets, which could easily be interpolated into a MUCH higher resolution image with some fancy math.

No. These cameras have sacrificed spatial resolution in exchange for recording
angular information about how a light ray came into the camera. You can not generate
a high resolution 2 dimensional image. What you can do is generate a low resolution
3 dimensional image (or a surface mesh) more easily that you can with a stereo camera.

[David Heath]

Originally Posted by Dave Blackhurst

David -

Think on this for a moment. If I have grasped the fundamentals properly... the design uses MULTIPLE lenses with multiple "views" of an object from ever so slightly different angles... meaning there is an ABUNDANCE of data sets, which could easily be interpolated into a MUCH higher resolution image with some fancy math.

No, it doesn't work like that. It uses multiple lenses in front of the image plane to take multiple views, indeed, but they are all low resolution. The maths won't work to then reconstruct a high res image, it will work to reconstruct the effect of varying focus.

Originally Posted by Dave Blackhurst

In effect you get infinite focus (check), nearly infinite exposure (they claim very good low light), and nearly infinite theoretical resolution... gives me a headache thinking about it, and the computing horsepower in that camera would be pretty impressive, BUT methinks that's how they are doing it.

No - you get the ability to alter focus in post, but only over a finite range - here fixed by the number of sub images. Infinite exposure is not true - what you will be able to do is correlate the images to improve s/n, but only to a limit. I suspect it will be of the order (in the Stanford case) of what would be achievable if they had simply had a 270x270 sensor of the dimensions of the 16 megapixel one they used. (Hence each photosite being large.) And the resolution will be far from infinite, it will be equivalent in this design to the number of microlenses, and the number of pixels under each microlens will determine the extent of refocussing that can be done.

All this is illustrated very clearly in Figure 13 of the Stanford report, and the text beneath it.

If you doubt me, try e-mailing the company and asking directly what the max output resolution of the system for sale will be?

Originally Posted by Charles W. Hull

You could imagine shooting with just a huge amount of high speed storage, and then doing all the video processing in post.

Even reading all the data off a 500 megapixel sensor at 60x times a second is a mammoth task, and, I suspect, way, way beyond current practical technology, and that's before we even start to think about processing. Applying this to high definition video may be theoretically possible - but practically is likely to be way, way off.

[EDIT - Just seen the last paragraph of Dustins reply above - exactly so, well put.]