Canon Vixia HG10 2GB File Limit Headache

[Andreas Michaelides]

Hi,

I might try this myself sometime, but since you already have the files.

Have you tried from the command prompt:

copy /b 000001.mts+000002.mts joined.mts

(change the filenames as appropriate)

Maybe the files are a straight binary join. When you treat the second one individually it may just start at the next keyframe or something, thus losing samples.

Regards,
Andreas

[Aaron Courtney]

Originally Posted by Leopold Hamulczyk

Have you tried copying the files to your PC with HGBackup, formatting the camera and then copying back to the camera with HGBackup? Does the camera still play seamlessly?

I have not done this, but it should not affect in-cam playback because the backup utility effectively transfers the entire filesystem contents from the HG10 onto your PC. So moving data back and forth in this manner should not affect the viability of that data. I will verify this, if I remember, when I need to dump the contents of the HDD due to lack of capacity. But....

It appears Andreas has provided the solution to this problem. I performed a cursory review in Vegas after using the good 'ole DOS syntax and I think this problem is solved. Great job Andreas! I am running an extended test using the entire ~90-minute concert to make absolutely sure this issue can be put to rest...

[Leopold Hamulczyk]

I hope it does solve the problem.

Incidentally, what is the difference between the .mts files straight from the camera, and the m2ts files from Guidemenu? Is it just a case of renaming? I notice that the file sizes are identical.

[Aaron Courtney]

Andreas' fix is golden. Once again, I DI'd 80 minutes of a live show I recorded into the HG10, placed the cam on a table, pressed record, and walked away. Five .mts files were created as the cam split the 80-minute clip according to its spec. I dragged & dropped the five files from the cam to my PC, ran the binary copy command, and waited. Took about 20 minutes total to recombine the five files into one .mts file. I then opened that file in Vegas. Next, I successively opened each of the component .mts files on the same timeline so I could easily see the "snap to" splice points. Muted the Vegas' spliced file's audio track and pressed play. The result was perfectly continuous audio at every splice point. LOL, I still can't believe how simple of a fix this turned out to be!

I was beginning to think this actually was a Vegas or CoreAVC PRO problem, so I set up a second PC with the various Elecard AVC offerings. Didn't matter - audio and vid drops at the joint.

It seems that Sony clearly has the more complete solution here. I'd guess that their cameras tag additional information onto to the beginning .mts file about how to correctly reassemble the clips via their software. That is much nicer than the Canon "fix" because you don't have to hunt around for all the separate clips and take care of the reassembly on your own <--which could lead to mistakes and additional time to redo it correctly.

One thing is for sure - it's really nice to have an NLE that can natively open these .mts files. It greatly reduced the time overhead in dealing with this manual assembly process.

[Dave Blackhurst]

All things are "simple" once you figure them out, but htis was a big BUG, any way you slice it, glad Andreas had a fix!! I think he's on to the cause of it too, probably a cadence thing of some sort.

It's absolutely silly that Canon couldn't have figured out the fix, but I guess that's what make DVinfo THE place - we can combine our expertise and "fix it in the field"!

MODERATOR - highly suggest this be made a STICKY, with the fix noted in the title... so anyone can find it fast! Since I might try the new Canon AVCHD cams, I'll know to look here for the fix!

[David Simser]

Kudos to Aaron for characterizing the problem and to Andreas for theorizing the solution!

In addition to making this thread sticky, as a new owner of an HG10 myself, I would love to see Canon/Corel fix this issue as a patch (either camera firmware or as an update to the Guide Menu software provided by Corel).

I would kindly ask Aaron to followup his open ticket with Canon and provide them with the solution from this thread. It seems to my that it would not be all that difficult for the Guide Menu software to re-assemble the mts files while copying them from the camera to the PC. That way, we would end up with one m2ts clip for each start/stop during recording on the camera -- seems natural.

Dave.

P.S. It would also not hurt if other members of this forum also "discover" this problem and raise it with Canon -- may increase the priority of providing an official fix.

[Aaron Courtney]

they're still not getting it, LOL...

Dear Aaron J. Courtney:

Thank you for your reply. My name is Raymond and I am the Online Support Coordinator here at Canon ITS. Your message was forwarded to me for a reply.

In testing the HG10 camcorder mentioned in your email inquires it appears your issue may be computer resource related. Many things will factor into how your system interacts with any software and devices connected such as drive fragmentation, the speed of the hard drives, ATA versus SATA, processor and bus speed, and most importantly, RAM.

In testing on a 1.6GHZ Windows XP 1 GB RAM computer, we were able to reproduce the same problem using the supplied Corel applications.

On a 3.6GHZ Hyper threaded Windows XP 2GB RAM, no problems when using supplied Corel applications.

Also, for comparison, when using iMovie08 on a 2GHZ duel core iMac with 1GB RAM, there were no problems.

If you have not already defragmented your primary hard drive, you may want to consider running disk utilities to free any virtual memory that is being accessed.

Also if you are working with the files directly on the camcorder you may want to copy the AVCHD folder to a hard drive and work with it from there. This will eliminate any bottleneck that may be caused from the data transfer through the USB connection.

The issue you are encountering is not related to the data files being broken into sectors; rather, it appears to be a result of how your computer is handling those large files.

Please let us know if we can be of any further assistance with your HG10.

Thank you for choosing Canon.

Sincerely,

Raymond
Online Support Coordinator

[Travis Michael]

As a new member of DVi and a new owner of the Canon HG10, I have been following this thread closely. With only a few days with the HG10, I have yet to run into this problem since I haven't recorded long enough to have split MTS files. But I know at some point I will, so finding this information will certainly prove to be invaluable. Thanks to all the hard work of you guys!

Working with AVCHD has certainly been a learning experience for me. I know a little bit about video formats and codecs, but finding the right software for this new video camera has been arduous at best.

First, I had to find out how to actually view the raw MTS files on my PC because I wanted to be able to quickly copy the files from the HG10 to my PC via Explorer and then open then up for immediate viewing. Purchasing the COREAVC product was the route I took, which required re-installing Haali Media Splitter because it was still trying to use FFDShow instead of CoreAVC at first. I really didn't want to install any of the included software that came with the HG10.

Now then, I use my Xbox 360 as a media center via a nifty program called TVersity. However, it won't work with the raw MTS files (not much does, it seems). So my current workflow is to use Windows Media Encoder to convert my MTS files to WMV-HD and they play flawlessly on my Xbox 360 and look great! So I can keep all of my actual video files on the PC and just stream them to the 360 when I want to view them on my HDTV.

So far I really like the HG10. My only main gripe with it is the lack of manual Gain adjustment. I don't like the high level of video noise when the camera boosts the gain in order to increase brightness. I did find that little trick that locks the exposure with no gain, but that's still not as good as having a manual gain adjustment. Other than this, however, I really enjoy using the HG10.

[Aaron Courtney]

Thanks Raymond for continuing this discussion with me. I appreciate it because I firmly believe this is going to soon become a much greater problem that Canon will be forced to deal with. Again, I don't believe we're talking about the same problem here.

I am not referring to stuttering issues due to insufficient PC resources in reference to the hardware demands the AVC codec imposes. Actually, my editing platform is more powerful than all three computers you list. Specifically, I have an Intel Core 2 Duo running @ 2.2GHz, 2GB of RAM, a 250GB SATA HDD for the O/S and apps and a separate hardware accelerated (AMCC 3Ware 9650SE Controller) RAID 0 array using three 250GB Seagate 7200.10 SATA-2 3.0Gb/s NCQ HDD's. This machine was built specifically for AVCHD editing and HDM authoring.

The problem with Canon's AVCHD implementation is quite simple. Canon does not provide a utility to reassemble multiple .mts files resulting from an extended video shoot (e.g., recording >~17 minutes in HXP mode). To be absolutely clear, here's what I did, which can be duplicated by ANYONE with a Canon AVCHD cam. I mic'd in an audio feed of a 90-minute rock concert that I had my audio mastering software play back from the beginning. The audio track gives you continuous, rhythmic sound which makes it incredibly easy to hear audio dropouts.

I placed my HG10 on a table, pressed play in the audio package to start the concert, simultaneously pressed record on the HG10, and walked away. Eighty minutes later, I returned and stopped the HG10 from further recording. I then connected the cam to my PC and browsed the HG10's HDD where I noticed that five new .mts files were created from this single 80-minute recording. Of course, a single new "clip" was presented for review in the cam's LCD window.

Using the provided Guide Menu software, I then transferred all five .mts files to my PC. And herein lies the real problem with Canon's AVCHD implementation. Instead of recombining the five .mts files back into a single continguous .m2ts file, the software simply copies the five .mts files into your chosen directory and renames each of the five files as <datetimestamp>.m2ts. When you playback these five files successively in ANY AVC player, NLE, converting utility, etc., you will experience a loss of audio and exactly two dropped video frames at each and every splice point where the previous .mts (or .m2ts - doesn't matter) file stops and the next successive file (in correct order) begins. Personally, I believe that the HG10's filesystem halts the audio recording in the first clip before it halts the video. This is why there are two dropped video frames in the second clip. Actually, I don't believe any video is actually lost, simply the audio and video are not sync'd correctly at the splice point. I think the audio preceeds the video in the successive .mts files and the AVCHD players and NLE's add two blank video frames to match the beginning of the audio.

What this means is that you cannot playback or edit any clip that exceeds 2GB of disk space without experiencing audio and video dropouts. The ONLY way anyone can playback a video of this length is IN THE CAMERA because the cam alone possesses the logic to splice back together the .mts files in the streams subdirectory.

Canon's backup software is of no help either because the software (correctly IMO) copies the entire filesystem of the HG10 to your PC.

So, I'm sorry, you did not successfully playback spliced footage from an HG10 on either the PC or the Mac without experiencing dropped audio and video around the 17-minute mark, assuming the footage was shot in HXP mode, because Canon does not provide a utility to reassemble these multiple .mts files back into the original seamless clip.

Fortunately for Canon, some bright individuals at dvinfo.net put their heads together where a work-around to the problem came from a user in New Zealand. As it stands right now, the only way to seamlessly splice these .mts files back into the orginal clip and avoid dropped audio and video is to use the copy /b command at a Windows command prompt.

I used this command on the five .mts files created from the 80-minute recording, opened the resulting 9GB .mts file in Vegas Pro and both the audio and video played back perfectly, without any drops at any of the original splice points. To make absolutely sure, I then opened in another video and audio track (on the same timeline) each of the five .mts files successively which identified exactly where each splice point originally existed. I carefully reviewed all four splice points along the timeline in the copy /b command spliced 9GB .mts file and confirmed that I could not find ANY dropped audio samples or video frames.

Although this is a workaround to this serious flaw, it is definitely not an ideal solution. Sony clearly has the best solution to this 2GB limit AVCHD problem because their software, upon transferring the clips to the PC, reassembles on the fly the multiple .mts files back into one contiguous file (representative of the point in time where the user pressed the button to start and stop recording) that seamlessly plays back in any decoding playback or NLE AVCHD compatible software.

Therefore, I, on behalf of all dvinfo.net HG10 owners, am requesting that additional software or firmware be provided by Canon that will correctly (i.e., seamlessly as the original recording was seamless) reassemble the multiple .mts files back into the single clip as it appears in the cam's playback menu window. If you care to, you can view the original thread documenting this issue from start to current work-around here http://dvinfo.net/conf/showthread.php?t=113083

Please respond with any further questions. And I would seriously appreciate your forwarding of this entire message to the team responsible for the HG10's (or Canon's AVCHD product) development.

Thanks.

[David Simser]

Bravo, Aaron! One suggestion for dealing with tech support: you might try producing a pair of "before/after" videos, one with and without the dropouts and posting the result online for them.

The following site has a channel devoted to videos created on the HG10:

http://www.vimeo.com/hg10

Dave.

[Aaron Courtney]

Looks like the information in this thread (and the thread itself, LOL) will make it to the right people...

Dear Aaron J. Courtney:

Thank you for the very detailed feedback on this issue. I apologize for the inconvenience this has caused you.

I have forwarded your inquiry to our product engineers for evaluation and testing. Once they have done this, we hope to have additional information for you.

Please feel free to contact us again if you have any additional questions or concerns.

Sincerely,

Ron
Sr. Contact Center Supervisor On-line Support Canon ITS


Boy, they sure are polite, LOL! But give me a fix over politeness any day. I'll follow-up to this thread when I receive more info...

[Riad M. Hanna]

Hi everyone,

If you playback the stitched-up file in your preferred media player, does it report the correct/new video time length (time stamp)? Can you seek through the file properly?

The reason I'm asking this is because some times I require joining few short clips together without the need of editing or re-encoding. In order to get the joined up file 100% compatible in all media players, I need to do the following:

1. Join the clips using either copy /b command or TSSplitter
2. Run the joined up file through tsremux and output as TS format (not M2TS)
3. Run the file again through tsMuxeR and output as TS format (not M2TS)

My HG10 is nearly full, and cannot record a long clip and test it at the moment.

Thank you all,
Riad

[Brian W. Smith]

Originally Posted by Aaron Courtney

Looks like the information in this thread (and the thread itself, LOL) will make it to the right people...

Dear Aaron J. Courtney:

Thank you for the very detailed feedback on this issue. I apologize for the inconvenience this has caused you.

I have forwarded your inquiry to our product engineers for evaluation and testing. Once they have done this, we hope to have additional information for you.

Please feel free to contact us again if you have any additional questions or concerns.

Sincerely,

Ron
Sr. Contact Center Supervisor On-line Support Canon ITS


Boy, they sure are polite, LOL! But give me a fix over politeness any day. I'll follow-up to this thread when I receive more info...

You can see it on this test clip I made a while back. I recorded an hour of my cell phone stop watch and then made a video clip. You can see the drop outs. (I spliced out the in between parts..the rest is losing frames between those times I list below).


Test here:

http://www.stage6.com/user/bas_vp/vi...le-Splice-Test

in HXP mode

file1: finish 18:03.2
file2: start 18:03.2 finish 36:01.7
file3: start 36:01.8 finish 54:00.7
file4: start 54:00.9

[Ken Ross]

The unfortunate thing here is that Canon will probably not get too many calls on this for the simple reason that most people won't be shooting continuously for that long. The problem will therefore elude them. I'm not even sure that the casual shooter will feel compelled to call if he does see 1 glitch in his recording.

The more people that report this issue the more likely it is to get fixed. This is a significant issue and deserves immediate attention by Canon. It is not reasonable to assume people will be using DOS prompts to workaround this.

[Aaron Courtney]

^^^Agree 100% and that's why I wrote what I did in my final follow up message to Canon. Hopefully, the people with the authority to actually do something about this issue will choose to properly remedy it.

Originally Posted by Riad M. Hanna

If you playback the stitched-up file in your preferred media player, does it report the correct/new video time length (time stamp)? Can you seek through the file properly?

I did a quick check using the included WinDVD junk media player with the stitched .m2ts file and it plays contiguously without dropped a/v at the original splice points. The player also displays the correct aggregated time. I would assume you could seek through the file also, except this lame software doesn't allow you to do that with any .m2ts file.