Canon Vixia HG10 2GB File Limit Headache

[Aaron Courtney]

I am starting this thread for a couple of reasons. First, it's nice to have a central repository where everyone can dump issues and experiences related to the topic. Second, I sincerely hope that everyone who owns an HG10 (and this problem will probably be present in the HF10 & HF100 too since Canon has included the same software) puts serious pressure on Canon to fix this problem immediately. As far as I'm concerned, as it stands today, this camera is useless for recording clips longer than ~17 minutes in HXP mode. I'm guessing that most people who have purchased this camera will not have the array of outboard audio equipment at their disposal that I have assembled over the past 15 years, so this should be a serious issue for every owner of this camera.

I purchased the HG10 because (1) it is basically the AVCHD equivalent to the venerable HV20, and (2) I am fully onboard with the AVCHD workflow. Honestly, I knew about this problem before purchasing this cam but figured it was isolated to the included editing software, which would not have affected me.

Here's the issue in a nutshell. The file system on the HDD (I am guessing) used in the HG10 imposes a 2GB file size limit. Apparently, this 2GB limit is not part of the AVCHD spec so it is NOT a problem with the format. This brickwall limit is easy to prove. Set the recording mode to HXP, place the cam on a table, press record, and walk away. After 20 minutes, press the record/stop button, plug in the power adapter and USB cable from your PC. The cam's HDD will be mapped to a logical drive (G:\ on my PC) and you can simply browse the contents of the cam's HDD, which is essentially the same structure as a blu ray disc. The video clips (stored as .mts files) are all located in the "Streams" subdirectory. They are incrementally numbered (e.g., 00000001.mts, 00000002.mts, etc.).

If you started with zero footage on the cam and shot 20 minutes of HXP footage, you will find exactly two .mts files in the Streams subdirectory. If you watch the 20-minute clip prior to connecting the cam to your PC, you will see and hear 20 minutes of seamless video/audio. If you drag and drop the two .mts files and open them in your NLE, you will notice 418 samples of audio (originally sampled in cam @ 48K) missing just prior to the splice point and exactly two dropped frames just after the splice point. Doesn't matter if you shoot 24PFor 60i.

If you transfer the 20-minute footage using the included Guide Menu app, you will have two <datetimestamp>.M2TS files. If you open those files in your NLE, you will see the exact same result.

I also converted the .mts files into a Cineform intermediate .avi file using NEO HDV to see what would happen. Interestingly, after converting two .mts files shot in 24PF (resulting in two Cineform .avi files) and opening those two files in Vegas Pro 8.0b (build 217), the two dropped frames were gone from the second file but the audio was exactly as it was in the first raw .mts and Guide Menu converted .m2ts files.

[Aaron Courtney]

Here's my original complaint sent via the web...

"After thorough testing, it appears that the software Canon has provided to transfer video from the HG10 to a PC (running WinXP) irreparably damages both the video and the audio streams when transferring videos shot in HXP mode exceeding ~15-17 minutes in length. The exact damage is in the form of audio and video dropouts at the splice point(s).

The only caveat regarding this issue listed in the documentation accompanying the HG10 is the following, "Longer scenes will be divided into files smaller than 2 GB. They will be saved as separate scenes when transferred to a computer. After these separate scenes are transferred back to the camcorder, you may notice audio drop-outs during playback at the points where the original scene was divided."

<EDIT: the above quote is referring to the included Corel DVD MovieFactory SE editing software>

The above statement is simply incorrect. The audio drop-outs do NOT occur as a result of transferring the video from the PC to the camcorder. Just the opposite. They are caused by transferring the video from the CAMCORDER TO THE PC.

I have no intention to transfer edited videos from the PC back to the camcorder. So the above quote from Canon's documentation should not apply to me. However, the problem is clearly apparent as I discovered while evaluating both the audio and video streams in my NLE. What is extremely frustrating is the fact that playing back the spliced video files while they remain on the camera yields absolutely flawless audio and video at the splice points.

Therefore, I am reporting this as a non-documented bug and request that it be corrected. If it cannot be corrected, then I will be forced to return the camera to Canon as defective and expect a full refund of my purchase price."


I really don't intend to return the camera at this point. So I merely wanted to convey the extent of how serious a problem I consider this to be. And I wanted to catch the attention of competent help right away. Unfortunately, that did not happen. To Canon's credit, though, I did get an email response in less than one hour <-- seriously impressed me considering it was ~11pm.

First response:

Subject: Re: Response from Canon - Technical Support
(KMM8279231V63919L0KM)
Dear Mr. Courtney:

Thank you for your inquiry. We value you as a Canon customer and appreciate the opportunity to assist you. I sincerely apologize for any inconvenience you may be experiencing due to the video and audio dropouts that you are receiving while trying to print with the HG10 camcorder.

Canon camcorders generally record at a framerate of 29.97 fps. Verify that the video editing program you are using is set to this, and not 30 fps. This is a difficult issue to address, as several different video capturing programs have their own specific settings that could effect recorded video. If the video can be viewed properly on a TV using the STV-250N Stereo/Video cable, without the drop outs, then it would be necessary to contact the manufacturers of the video editing program that is being used. If the drop outs do appear on the TV then the camcorder heads may need to be cleaned.

We hope this information is helpful to you. Please let us know if we can be of any further assistance with the HG10 camcorder.

Thank you for choosing Canon.

Sincerely,
Stephen
Technical Support Representative


*extremely frustrating*

[Aaron Courtney]

Second email...

'If the drop outs do appear on the TV then the camcorder heads may need to be cleaned.'

This is a HDD based capture camcorder, not a DV20. THERE ARE NO HEADS TO BE CLEANED.

'If the video can be viewed properly on a TV using the STV-250N Stereo/Video cable, without the drop outs, then it would be necessary to contact the manufacturers of the video editing program that is being used.'

Wrong. I've already stated exactly how and why the dropouts occur. Playback of recorded footage in the camcorder connected to ANY display yields perfect footage. It's the process of transferring the footage outside of the camera that destroys data at the splice point(s). When I open spliced files from a Sony CX-7 after transferring using the accompanying PMB software, data at any splice point is presented perfectly and contiguously. <thanks here to Dave Blackhurst for his verification> And yes, my video editing software is set to the correct frame rate for the footage to be edited.

I'm seriously impressed with the response time to my inquiry. However, I clearly know more about this camcorder than you. I suggest you either escalate this issue to a higher support level so you don't continue to insult my intelligence, or shoot 20 minutes worth of HXP footage on an HG10, walk over to a PC, transfer the files using the accompanying Media Guide software, open the two files in whatever editing software you want, look at the video frames on the timeline and the resulting audio wave, and then come back and tell me what you've discovered..."

Some people may think some of my reply was a bit over the top, but I've spent the last 15 years of my life in IT consulting dealing with uninformed initial responses such as these. After a while, it tends to get to ya, LOL.

[Aaron Courtney]

Second response:

Subject: Re: RE: Response from Canon - Technical Support (KMM8283462V77146L0KM)

Dear Mr. Courtney:

Thank you for writing to us. I apologize for the confusion regarding your inquiry. We do our best to answer all customer inquiries as accurately and efficiently as possible. I am sorry that incorrect information was sent in response to your message. We do regret any inconvenience or difficulty that resulted from our miscommunication.

While Canon does provide a copy of Corel DVD MovieFactory SE, we do not offer support for this application. For support concerning the Corel Application Disc, please contact Corel directly, at:

1-510-979-7118 or 1-800-772-6735, Monday - Friday, 09:30 a.m. - 05:30 p.m. (PST)

http://www.ulead.com.tw/Ulead/Survey...?pg=1&fs=us#cs

Please note, Canon does not produce video editing software, therefore, we can offer no support on specific editing programs. Because there are so many editing programs available, it is not possible for us to offer support or to catalog those programs. We recommend that you contact the software manufacturers, or perhaps visit their websites, to find the information you seek. We apologize for any inconvenience this may cause.

Have you tried using the Canon Backup utility to drop the files from the camcorder directly to your hard drive, then manually rename the ".M2T"
files that the Backup Utility copied to ".M2TS" for opening in your NLE software?

We would suggest trying that workflow to see if cutting out the Corel Software usage altogether will work for you. Nothing in the Canon Backup Utility, says clips have a 2 GB limit like the Corel software, so maybe the timeline and audio won't be affected.

We hope this information is helpful to you. Please let us know if we can be of any further assistance with your HG10.

Thank you for choosing Canon.

Sincerely,

Chris
Technical Support Representative


*finally, some degree of competence* But still, they're just not getting it. My reply today:


"Chris, finally, great response. Thanks for the suggestion, but it still doesn't fix the problem. First, I need to clarify that I am not using Corel/Ulead DVD MovieFactory SE. It's garbage. I am using Sony Vegas Pro 8.0b (build 217). I have no problems whatsoever opening either the raw (dragged & dropped) .mts files or the converted (using Guide Menu) .m2ts files.

Again, here's the problem. Apparently you guys are not using these AVC cameras otherwise you should know this. The HG10 (or perhaps the file system used in the camera) imposes a 2GB file limit on video clips. This is very simple to prove. And really, I want someone on your support team to do this - shoot 20 minutes of continuous HXP footage, push stop, and then connect the camera to a PC using your USB cable. When you browse the removable media drive (HG10) in Windows, you will see that the single 20 minute clip is split into two .mts files ON THE CAMERA.

If you play the 20-minute clip ON THE CAMERA, it plays back perfectly. So clearly, there is something inside the camera that enables correct splicing of the two files to provide a seamless transition between the two files. As soon as you remove those .mts files from the camera, either by simply dragging and dropping or via the included Guide Menu software, ~400 samples (sample rate = 48K) of audio are dropped from the first file and exactly two video frames are dropped from the second file at the splice point. Curiously, the audio in the second file is perfect, while the video stream in the first file is also perfect. Makes no sense to me.

What appears to be happening is that some form of metadata present on the HG10 which enables proper splicing of the two files is not being transferred to the PC when the raw .mts files are transferred. This KILLS the chance of using this camera for any type of shooting requiring >17 minutes of HXP capture (and seriously, no one is going to use the low bitrate settings anyway - let's be real here). I can handle a dropped video frame or two, but dropped audio is an entirely different story. Imagine shooting your kid's Christmas play and your audio drops out at a key moment in his/her dialogue. There are going to be a lot of pissed off parents as soon as they transfer that video from the HG10 to the PC for high def disc burning.

So, shoot 20 minutes worth of continuous HXP footage on an HG10, transfer the files to a PC using whatever method you'd like, then open those files in whatever NLE you prefer and then come back and tell me that the audio isn't butchered at the splice point. As far as I'm concerned, the transfer ustility Canon provides with the HG10 irrepairably damages the video footage upon transfer. Seriously, what good is a HDD video camera when you can't reliably extract the footage from it?"


This is where it stands as of right now. I'll post Canon's reply. Anyone who cares to, please jump into this discussion (so I'm not talking to myself, LOL!). To be fair to Canon, I haven't actually tried the backup utility route yet. I will do this when I get home tonight and really hope it does work. If it does work, then Canon needs to clearly note this workflow workaround. If it doesn't, perhaps there will be a good chance that someone will fix the issue with this provided software since it is a Canon-specific app.

[Dave Blackhurst]

Hi Aaron -

A couple observations... the Sony PMB software creates a single long file from the component .mts files, and it is seamless, just like in the camera.

There must be something in the AVCHD spec that is dictating the arbitrary 2G file sizes and stitching... and clearly there are some implementation issues.

In another thread I started I observed that the first .mts file from a Sony CX7 is absolutely unreadable unless processed through the PMB software - a fact confirmed by Sony... it appears the first file with the Canon IS readable... but then the "stitching" is flawed.

Don't know how good you are at taking the file code apart (it's just a bit over my head), but there may be clues in there.

I've accepted that I have to use the PMB software with the CX7, it's not an inconvenience, but what you've discovered scratches Canon off my list until they figure it out.

I think that CANON has it right if playback on the camera is correct - IOW the camera correctly "stitches" clips together. BUT, the way the files are tagged prevents just sticking them together with sucess, and apparently the Corel app is not putting the pieces back together properly either. Have you tried using hte Corel app JUST for importing, and seeing if it will create a single file from the bits? It looks like you did, but figured I'd ask.

FWIW, tech support is often pretty clueless at the first level - they deal with people using the DVD tray as a cupholder... but even at the higher levels, sometimes they are stumped by an engineering decision - NO one at Sony can figure out why the first .mts of a long clip can't be played without being processed through the PMB software, but it can't, no matter what you throw at it - camera playback is fine, and once run through PMB, everything is hunky dory!

Sounds to me like Canon has a similar issue, but their "outsourced" software is at fault, from your description.

Hope you figure out the "fix", I know I was gald once I solved the questions with the CX!

[Aaron Courtney]

Thanks Dave for providing me some anecdotal ammunition to throw at Canon! There was just no way I was not going to use it in this case, LOL!

There are three primary Corel/Ulead app's included with the cam - (1)Guide Menu - the "approved" cam ->PC & PC -> cam transfer utility, (2) the InterVideo Win AVC player, and (3) Corel Movie Factory DVD SE.

As I said, I both dragged & dropped the .mts files and used the approved Guide Menu app to get the clips onto the PC. The result was the same, as indicated. I also tried the cheesy DVD authoring software which actually provided a dedicated button to link clips together into a single file. Playback of the resulting file yielded the same result as editing the .mts and Guide Menu software converted .m2ts files in Vegas Pro.

I then converted the Corel Movie Factory "stitched" clip into one of the default AVCHD formats provided in the editor. Since that was going to take 3+ hours to render, I decided to go to bed. I'll look at the resulting file tonight but don't expect anything different than what I've already experienced.

Right now, it's the Canon backup utility or bust.

Originally Posted by Dave Blackhurst

There must be something in the AVCHD spec that is dictating the arbitrary 2G file sizes and stitching...

What we need is for someone with a flash mem AVCHD cam to state whether or not their cam splits long continuous footage into files <2GB. That should clear up this question...

[Dave Blackhurst]

CX7 is a flash memory camera.

and BTW, I tried recording in different bitrates - they still are split at the 2g point (s), just takes longer at the lower resolutions.

[Aaron Courtney]

Thanks Dave for that clarification. If it truly is part of the AVCHD "camcorder" spec - even if unwritten - then this could prove to be quite a problem for mass adoptation of AVCHD unless every manufacturer really dials this in and gets it right out of the gate.

Canon's backup utility simply copies the entire contents of the HG10's HDD into a specified directory on the PC. The .mts footage files are identical whether they are "backed up" via the software or dragged and dropped individually by the user.

I don't think this is a Vegas issue because I converted via Cineform NEO HDV a 2GB clip into an .avi, opened that in Vegas, and the audio was chopped identically to the .mts (and .m2ts) AVCHD file. To make sure, my next step is to download and install a trial version of a different NLE and see what happens...

[Aaron Courtney]

By the language, it seems that I'm getting closer to talking to the right people here. At least now they're admitting to the 2GB limit being imposed by the HG10...

Dear Aaron J. Courtney:

Thank you for writing, I am sorry to hear of any difficulties with the video footage from your HG10 camcorder and our email responses.

Regrettably, the HG10 does not have any settings to change the 1.9GB size that your videos and MTS files download.

I have forwarded your comments to Canon USA through our Customer Feedback process. This process allows us to capture important feedback from our valued customers. As we constantly strive to improve our products and services, your comments are vital to our continued success.

I will need to research this question in more detail, and will write back within 48 hours.

I am sorry for any inconvenience this may have caused you. Please let me know if we can be of any further assistance with your Canon HG10.

Thank you for choosing Canon.

Sincerely,

Deva
Technical Support Representative

today's follow-up from Canon

Dear Aaron J. Courtney:

Thank you for your inquiry. I am sorry for the delay in my response
regarding your HG10 camcorder.

I completely understand your frustration regarding the dropped audio and video once downloaded!

I have confirmed with one of our Senior Level Technician that it appears that the camcorder is working correctly.

This would be an issue with the software, we would recommend contacting Sony for support regarding the assembling of the 2GB clips.

I would also recommend trying this out with the Corel software that was shipped with your camcorder.

If this doesn't answer all your questions, please contact me again with the more details.

I am sincerely sorry for the inconvenience this may have caused you.
Please let me know if we can be of any further assistance with your Canon HG10.

Thank you for choosing Canon.

Sincerely,

Deva
Technical Support Representative

[Aaron Courtney]

Ok, you guys are still not getting it. Yes, the camcorder is functioning perfectly. As it stands right now, however, NO video footage exceeding 2GB worth of file size captured by the camcorder can be removed from the camcorder without ruining the audio stream in the process.

I would like you to ask that same SR. level tech how he/she would playback 20 minutes of continuous video shot @ HXP that has been transferred to a PC.

The provided Corel software cannot reassemble the .mts files from the HG10's HDD into one seamless video and audio stream if that combined file size >2GB. I'm not talking about stringing multiple clips together here. I'm talking about reassembling a 20-minute clip after it has been split into multiple .mts files <-- and how stupid is this anyway! Seriously, there is no reason on Earth to limit video files to this stupid 2GB limit anyway. That was a file size limit imposed under FAT16, not NTFS. If I shoot for 8 hours continuous @ HXP on this camcorder, I should get ONE 40GB CLIP, NOT TWENTY 2GB FILES that have to reassembled into the original clip - which cannot be done seamlessly outside of the camera with the software Canon has provided with this camera (or any software in the world for that matter).

Neither the included editing software nor the player can playback HG10 footage without dropping the audio (and two video frames) at the splice point.

My question is INCREDIBLY simple:
I would like to know how to seamlessly playback spliced .mts files originating from one, lengthy (>17 minutes @ HXP) captured video on ANYTHING OTHER THAN THE HG10.

[Aaron Courtney]

So far, I have come up with two PERMANENT solutions to this problem...

(1) Use outboard audio and shoot 24PF.
(2) Capture footage via HDMI out on HG10.

The outboard audio is pretty much self explanatory. Using pro gear will always yield better results. But it's added expense and inconvenient too. After converting 24PF footage (pulldown removal) into Cineform's intermediate .avi, I found no dropped frames. Now that could have been purely by chance (cadence).

The HDMI capture would also work because the cam correctly reassembles the .mts files on the fly and will push that out over the HDMI port. BUT, then you're back to a realtime (ala HDV) workflow process again. YUCK! Not why I bought into the whole AVCHD deal in the first place...

[Dave Blackhurst]

HMMMM -
I think you need to eliminate one possibility - and it's probably hard to do on the camera screen... if you playback the original long clip (and I presume it is shown as a single clip in the cam, as on the CX?), is there any dropout at the splice point - probably nearly impossible to spot without having the whole deal on the timeline, where the small dropout is hard enough to spot...

This has me wondering about how clean the splice is on the CX - going to have to set up a test with a moving object and something for audio, but I'm pretty sure the PMB software assembles gap free. I've got my tests saved, maybe I can review those and see if anything is there - the one "live" clip I shot seemed perfect, but the gap you're describing is sooooo small, I'll want to double check it!

IF you can play back the clip glitch free either in camera or played back to a big TV and audio system, then it's not the camera per se, but with the software, but I'm having trouble seeing that as a possibility - I *think* you will find the glitch is there on playback direct from in camera... if it is, maybe a firmware update chould fix it? If it isn't, then they need to jump up and down on Corel for a fix.

Just my thoughts, will have to set up a test out of curiousity for the CX, but I'm feeling better about my Sony all the time <wink>!

[Aaron Courtney]

Hi Dave, the piped in audio via the Mic in jack makes it really easy to locate exact points in time on both the cam and in any NLE. That's why I chose to dub in a live show I recorded. You simply locate the exact splice spot in your NLE, take note of the song and the location in that song, and then dig through the clip on the cam until you hit the exact same spot. I even had both PC and cam playback streams semi-sync'd within a couple 1/10's of a second.

The in-cam playback is perfect, both audio and video. I reviewed the exact splice spot at least 10 times and I can definitively say it is flawless. As I said earlier EVERYONE will hear the audio drop when playing back audio of a "continuous" nature - music, rhythmic, etc. You can't miss it.

The clip in the preview/playback window in the cam does appear as a single clip (looking at a video frame, not the actual file structure). However, if you browse the HDD contents of the HG10, that single clip is comprised of two separate .mts files (assuming <~34 minutes @ HXP).

[Dave Blackhurst]

OK, just wanted to confirm that. That's how the CX does it as well - shows a single clip, but when you look at the file structure, it's mulitple files, 2G limit...

SO, I think you've eliminated one issue... sort of... the data is there SOMEWHERE, or you'd have a dropout on cam playback. THE BAD news is this means there's a software glitch that seems to be causing the problem when reading back the .mts files... and it seems to be on everything you've tried.

Perhaps this is why Sony has it's own software (PMB) - so as to avoid the problem on import. Obviously the camera internal firmware gets it right somehow, so that's a "start", but theres' still some issues to unravel.

[Leopold Hamulczyk]

Juse wondering. Have you tried copying the files to your PC with HGBackup, formatting the camera and then copying back to the camera with HGBackup? Does the camera still play seamlessly?

I will have to try with Procoder and with Elecard AVCHD Converter Studio to see what they do. I will be recording meetings in April which will run over the 2GB limit several times over, so the break over the splice point will be annoying (but I can probably live with it).