Synchronizing audio and video on consumer cameras

[Karl Reinig]

I apologize in advance if this is too simplistic a question. I have attempted to answer it myself through the FAQs, but they seem to be on a whole higher plane than I am thinking. So, here goes.

I have video (taken from an I-phone) that shows an object repeatedly colliding with another object. I am interested in using the sound of the collision to interpolate between the closest video frames on either side, so that I can learn more about the state of the rest of the scene at impact.

When I look at the video in MovieMaker and view the audio, I see very nice spikes. However, they are clearly a couple of frames too late. The source of the noise was only about ten feet from the recorder. At 1,100 feet per second, I could see having maybe 1/100th of a second delay. But, the delay I am seeing is more like 7/100th of a second, which makes it useless for locating and interpolating frames.

Can anyone tell me if this is just tough luck? If not, is there a known way to get the audio synchronized with the video in an automatic way (known delay, etc.)? I could throw a clapper into future scenes, but I'm trying to do it in a way where the user does not have to do anything unnatural like that. My intuition is that the synchronization must be better than I am seeing or people's lips would look out of synch. But, I have no experience in this area.

[Brian P. Reynolds]

Digital recording devices video / audio often have different transfer rates, normally video is slower than audio.
Do a track split and shift the audio /or video to match...... Quite a common problem these days

[Richard Crowley]

Attempting to use casual video from something like a cell phone is hardly condusive to any sort serious analysis.

You made no mention at all what is the source of this unsynchronized audio track??? As others have mentioned it is expected that the clock/sample rates of two unmatched devices (a cell phone and a mystery audio recorder) will not match.

If the audio and video are simply offset by some fixed amount, it is trivial to simply slide the video track into position to correlate with the audio track. This seems so simple and obvious it makes me think that your question is about something else that we don't understand???

[Brian Drysdale]

Also, beware aware that devices like phones may be using a variable frame rate, rather than a constant frame rate. This can knock out the sync when you're trying to match in post.

[Jay Massengill]

We also need to know more details like which iPhone?

Is it an iPhone 5s slo-mo 120fps or normal video?

Exactly what version of editing software are you using?

Have you matched the editing software session settings to your video format?

This may just be a case of needing better video editing software and/or making sure your editing session properties are matched to the footage you're trying to examine.

Also the faster your computer, the better job it can do of displaying video and audio in sync inside the video editing software's preview mode.

[Rick Reineke]

Many consumer NLEs can also not move audio clips to a sub-frame level, necessary for good sync. As well as different audio sample rates can be a major problem.

[Brian Drysdale]

You don't really need sub frame adjustments for picture and audio sync, although you may want that feature for matching audio tracks due to time delay differences between mics.

Sync loss becomes more noticeable if the audio leads the picture, but audio lagging video lagging by 1 frame is within standards for TV. For film and I assume this for 35mm (rather than 16mm), the standard is within 1/2 a frame, but that could because you can set the audio clap midway relative to the frame due to there being 4 perforations per frame on 35mm mag film. Sort of laws of averages thing on the frame with the slate clap, but tests show that you can lag by a couple frames and it'll not be noticed.

[Karl Reinig]

I would like to thank everyone for their input. From Richard's response, I did not make it clear that both the audio and the video came from the same I-phone. I also agree that great analysis will not come from consumer phones. What I'm attempting to do is use image processing to reconstruct the state at the time of the collision and thus give a wide variety of an people an inexpensive method of seeing what their video cannot show them directly. My hangup at the moment is that I was hoping to directly use the audio to know what frames (on either side) to process. And if it was really accurate, just where to interpolate between those frames.

The delay of the audio seems consistent (at least across the one example video I took). I was hopeful that certain standards are followed in video and that I could therefore always move the audio a known amount. Since the thing that is colliding rarely actually does so in a frame, I don't have a simple way to automatically find the offset.

From Jay's response, I looked throughout the MovieMaker settings and could find no relevant settings. That was disappointing as that could have really explained what was going on. I also found a website that would extract the audio for me (audio-extractor.net). I then fed that audio into a program called WavePad that allows me to zoom in on the audio. The numbers closely matched those in MovieMaker, giving the same offset. This makes me feel a little more that the delay is independent of the editor. Although it could be that the editor is handling the audio correctly but not the video.

I expect to write a program to read in the video and audio, use the audio to detect collisions, and then do the image processing all automatically. So, ultimately, the editor will not be the issue. However, if the consumer cameras don't stamp both the video and audio with accurate time code, then the problem will be much more difficult.

Brian's comments about leading verses lagging audio gives me the unsettling feeling that I-Phone folks may be fine with the lag I am seeing, as it does not have as bad an effect as having the audio lead. If for some reason their audio timing was beyond their control a bit, I could see where they might always have it lag, just to avoid the possibility of lead. But, that seems like it is reaching, especially when the farther you get from the source, the more it would lag naturally.

So, at this point, I think I'm stuck with trying to figure out more about the particular I-phone (if all of the different devices have a different lag, that could be the end of this project). It was disappointing to see the lag in the first place and I was really hopeful that there was some sort of video/audio standard/phenomena to explain it. Again, thank you all for your responses. And if anything in this response helps sparks an idea, I'm all ears (which, fortunately, are already remarkably well synched with my eyes).

[Brian Drysdale]

The variable frame rate that iPhones can give, seems to be a common issue in syncing audio with them. A number of NLEs need a constant frame rate, so the original video has to be converted to give this.

[Karl Reinig]

I can see where a variable frame rate associated with I-Phones would be a large problem when synching with audio that was taken from a different device. But, are you saying that it is a problem even when the audio is coming from the same device? I would have thought that both the audio and video would have received time stamps from a clock on the I-Phone that was common to both.

I think that at this point my task is to figure out whether the I-Phone is creating a poorly synchronized audio and video or whether what I'm seeing is a result of the particular NLE (MovieMaker) that I'm using. It is hard to picture how any NLE can do anything other than look at the time associated with a frame of video and a time associated with the Audio and put them together accordingly, which would make this a device problem.

On a related note, I can picture a frame having a time associated with it. But, it is not as clear to me how audio would associate its time. I would think that every sample of audio (or at least every nth sample) would get a time associated with it. But if that is not the case, then I suppose that there is no real difference between audio taken from the same I-Phone or that captured from a separate source.

[Brian Drysdale]

Not all NLE's can handle variable frame rates, they're designed for constant frame rates. I gather FCP X can, but since Apple's phones use variable frame rates, it would be egg on their face if it couldn't.

[Karl Reinig]

So, with the last suggestion, I headed to the Apple store. We loaded the video into Final Cut Pro X and saw the same delay. My next two steps will be to contact Apple about what I would describe as a problem with my phone (likely shared by many others) and to try this with my Son's Iphone, which, of course, is more modern than mine.

If anyone is interested in what I find, just post your interest and I will post what I find. I realize that this is considerably off the beaten path of what I see going on this forum and again, very much appreciate that you helped any way.

[Bruce Watson]

Originally Posted by Karl Reinig

I can see where a variable frame rate associated with I-Phones would be a large problem when synching with audio that was taken from a different device. But, are you saying that it is a problem even when the audio is coming from the same device?

Yes. Exactly that. I tried video on an iPhone exactly once, and after spending hours trying to sync the video from the iPhone with the audio from that same iPhone, I vowed never again. What idiot at Apple thought that a variable frame rate was acceptable? Major cluelessness.

[Jay Massengill]

Last night I loaded the paid MoviePro app on my iPhone 5s (and my iPhone 4s and older iPad 2) but haven't had time to test the results extensively.
This app allows you to choose the frame rate across a very wide range, as well as set resolution, manual audio level, lock the exposure, focus and WB independently etc.
I don't know if it truly locks the frame rate at the value you select, or if that's just the starting point for the iPhone to be flexible around.
You may want to try this app, especially with the 5s, and see if you get better results.