XDCAM Transfer 2.7.1 Bug/Glitch when Importing Clips Larger than 8GB

[Tony Tang]

I'm importing a clip into Final Cut Pro whose size is greater than 8GB once it is stitched together by XDCAM Transfer. Once the clip is imported into QuickTime format for FCP, there is a fairly serious 1 second audio and video glitch within the first 5 minutes of the imported .mov file. I can reproduce this 100% of the time and was wondering if anyone else had noticed the same problem?

My setup:
- PMW-EX1
- XDCAM Transfer 2.7.1
- SanDisk 16GB SxS

Steps to reproduce:
1. Configure the EX1 to shoot in 1080/60i SP mode. I suppose you could try 1080/60i HQ mode, but I've been shooting with 1080/60i SP.
2. Record about 45 minutes of continuous video on the EX1 in the above mode. Do not stop the recording during the 45 minutes. This should produce a set of files greater than 8GB when strung together.
3. Insert the 16GB SxS card into a MacBook Pro and launch XDCAM Transfer 2.7.1.
4. Select the 45-minute clip in XDCAM Transfer.
5. Click Import.
6. Open the imported 45-minute .mov clip in FCP or QuickTime.

Actual results:
Usually within the first five minutes of video, you will see a 1-second glitch in the video that looks like a bad MPEG stream... like when you have a scratched DVD and the video gets pixelated. This is usually but not always accompanied with a high-pitched sine wave or screeching tone that lasts for about 1 second.

Expected results:
No glitch in audio or video.

Additional notes:
- If you try to re-import the 45-minute clip, then the glitch will occur in the exact same spot in the video.

- If you specify an in and out point 15 seconds before and after the glitch and import that clip, then there is no glitch. This result probably happens because the imported clip is smaller than 8 GB.

- If you play the video within XDCAM Transfer around the point where the glitch occurred, then you will see and hear NO glitch. This suggests that the original file is okay and that the glitch is introduced during the import process by XDCAM Transfer.

- I have noticed this behavior whenever the imported file size is larger than 8GB.

- When you try to render or export your FCP timeline with the bad glitch, then it will fail unless you cut out the portion of video with the glitch.

Workaround:
When importing files, do not exceed 8GB in size. Instead, import subclips one at a time to avoid exceeding the 8GB size limit.

Since I can reproduce this problem with absolute consistency, I will phone it in to Sony. Hopefully it is something that can be addressed in a software update to XDCAM Transfer.

Note: I attached a screenshot of what the glitch can look like. This is supposed to be a soccer game, but it is rather unrecognizable. This is the glitch at its worst. Sometimes it's not this pronounced.

[Dave Morrison]

Good report. Looks like I'll stick with 2.5.1!!

[David Baxter]

For what it's worth, I've also just run into this problem with 2 clips this week. By the time I had noticed the problem, I had already wiped the cards, so I wasn't able to go back and do any troubleshooting. I had also wiped my copies of the cards (made with the XDCAM EX Clip Browser). Needless to say, I'll be keeping those copies around for longer now.

I've shot more than 350GBs of footage with the EX1 over the last 4 months, and this week was the first I've seen this problem.

Tony, you mention that you've narrowed the problem down to clips that are over 8GB. Oddly enough, the clips that I've had issues with are both under 8GB (one is 4.26GB, and the other is 4.99GB).

I haven't had time to do any detailed troubleshooting, but I'll post my settings and workflow here. Hopefully we can find a common thread with these glitches.

- Shooting 1080p30 HQ on the EX1
- Insert card (I have 3 8GB Sony SxS cards) into my MacBook Pro
- Copy clips from SxS card to a directory on an external hard drive, then delete clips from the card -- using the XDCAM Clip Browser (repeat for each card I have to dump)
- Once all cards are dumped, I open FCP (6.0.4), aim it at the directory on my external drive containing all the clips, and use the Log and Transfer tool to import the files into Final Cut Pro
- After importing, I generally go back and delete the source clips (the mp4's in the BPAV directory), as it has seemed foolish up until now to keep both Final Cut's .mov files and the native MP4s.


I have a bunch of shoots lined up for this week. I'm hoping I don't experience the same issues ever again, but I'll keep all the source files around to troubleshoot with if I do.

[Tony Tang]

It's interesting that you're seeing the problems with clips smaller than 8GB, David. I checked back through all my projects that failed to encode, and I did find a couple that contained files smaller than 8GB. I have not, however, been able to locate the glitch in those video files yet... I need to spend time to locate them.

Starting this month (September), I began using a 16GB card instead of my 8GB cards. Prior to using the 16GB card, I had no problems. Once I started using the 16GB cards, I began noticing issues.

I also shoot two main sports: volleyball and soccer. Volleyball always produces files smaller than 8GB because the games don't last as long as a 40-45 minute soccer half. Volleyball has so far (knock on wood) never given me any glitches or QuickTime errors when I go to encode the video in FCP. Soccer has almost always given me glitches and QuickTime errors when I try to encode the video out of FCP or in Compressor.

I've shot over 600GB of footage this month alone. So I have a pretty good sample size. But almost all of my soccer projects are backlogged because they cannot encode successfully due to a glitch in the video file.

---
HOW TO LOCATE THE GLITCH

This is a hard problem to troubleshoot because it can take a long time to find that 1-second glitch in a 45 minute clip. I basically have to watch the clip until I see or hear it. There is another way to find the glitch:

1. In FCP create a new sequence.
2. Drop your suspect video clip into the sequence timeline.
3. Set your in/out point so that only the first half of your video clip is selected.
4. Go to File > Export > QuickTime Movie, and select the option to Make Movie Self-Contained (this will ensure that FCP reads the entire source clip, which will cause it to error out when it reaches the glitch).
5. If your export succeeds, then you know that your glitch is not in the first half of the clip and you should set your in/out points to the last 1/4 (quarter) of your clip and repeat Step #4 to continue trying to find the glitch. Note that you are using a bisection strategy to eliminate half of the possible places that your glitch could be.
6. If your export fails, then you know that your glitch is in the first half of the clip. You should then set your in/out points to select the first or second 1/4 (quarter) of your clip and repeat Step #4.

Repeat the above steps until you've narrowed down the glitch to a portion of your clip (say 5 minutes that you can easily manually watch and find the glitch by eye.

Here is an example to help explain the above process... Suppose you divide your clip into two halves: parts A and B

1. Try to export half of your clip, part A (remember to select Make Movie Self-Contained when exporting)
2. Suppose your export succeeds, then you know part A is glitch-free and the glitch must be somewhere in part B.
3. Next divide part B into two halves B1 and B2, and try to export part B1.
4. Suppose your export fails, then you know the glitch is somewhere in part B1. If your export succeeds, then you know the glitch is somewhere in part B2. You can keep dividing up your sub-segments and exporting portions of it to pinpoint the glitch.
---

Hopefully I've correctly identified XDCAM Transfer as the cause of the problem. But it seems that my theory that it only happens when clips are larger than 8GB may not be true based on David's experience.

[Alister Chapman]

As a thought. Have you got file splitting enabled in FCP (limit Captue/Export Files in System Prefs)? Could it be that FCP is splitting the files into smaller chunks and it is that that's causing the issue?

[Tony Tang]

Alister, good thought, but no. I do not have that setting configured in FCP.

Plus, I run XDCAM Transfer outside of FCP. It's a standalone app after all. When I watch the .mov file it creates, I can see the glitch in quicklook or QuickTime (totally outside of FCP).

Thanks for the idea though.

[John Wiedenheft]

Has anyone found a solution to this besides the IN/OUT export clip around the affected segment?

Here's my situation: I recorded a multi-cam production, recording three ISO shots to three locations, two FCP6 systems, recording in ProRes 422HQ and a PMW-EX30 deck recording to 1080/60i. When I went to make a multiclip of the three angles, Final Cut said I couldn't because of different codecs. I went to Compressor and set about encoding the XDCAM EX footage to ProRes 422 HQ, but multiple times came up with errors.

I thought my problem might be the problem discussed here, as I've had the problem a few times before. I found the few affected frames, so I know the problem is the one discussed here and the same one experience before.

Is there another way around this so I can make my multiclip?

[Andy Schocken]

I've had quite a few glitches on playback, but they seem to be somewhat different than the others here. There's usually audio clipping for about a second, and some level of mosaic breakup in the picture. Happens on long clips and short clips, and repeats in the same place when played back. Sometimes there's a tiny, barely noticeable glitch, and we only notice because FCP won't export a quicktime file until we locate the problem clip and replace it. Shooting SxS, using clip browser with crc, then xdcam transfer (independent of fcp) to convert the BPAVs. All apps have been up to date.

The strange thing is that the clips play fine on my system (I'm shooting and converting), but when I copy the footage to my editor's system, the glitches show up. You'd think there was a problem in his software or drives, but if we then copy the clips back to my system, I get the glitch on playback. We can't figure out what's going on- it seems to show up when we're transferring between drives. I've always backed up the BPAVs, so whenever we have the problem, we reconvert the file using xdcam transfer, and the new files have always played back flawlessly. Any ideas?

[Matt Davis]

This really is a far reach, but when I first got my EX1, I had no end of troubles reliably getting footage off SxS cards - some strange error to do with QuickTime Atoms IIRC.

The problem came down to some sort of issue with codecs, specifically the DivX and the Flash codec from the pack with an icon that looks like a swiss army knife. Got rid of these two and all was happiness in the garden.

If it happens on one system (e.g. editor's system) and not on shooter's system, I'd start looking at what Codecs are *really* needed on the editor's system and chopping out all unnecessary ones.

HTH...