Is there an app for managing AVCHD files?

[Nick Mirro]

Hi. I'm getting huge headaches trying to manage avchd files on a Canon HG10 and the PC hard drive.

Are there any 3rd party apps that can read from the camera drive and index or sort its contents, perform selective downloads, manage video thumbnails and generally organize the avchd files stored on the desktop hard disk?

Canon's apps do not allow sorting of images. Transferring some clips from the camcorder is very difficult. All or nothing downloads are not at all useful.

Currently running:

Corel VideoStudio 12
Corel GuideMenu
Canon HGBackup.exe

[Kevin Bjorke]

Originally Posted by Nick Mirro

Hi. I'm getting huge headaches trying to manage avchd files on a Canon HG10 and the PC hard drive.

Are there any 3rd party apps that can read from the camera drive and index or sort its contents, perform selective downloads, manage video thumbnails and generally organize the avchd files stored on the desktop hard disk?

Canon's apps do not allow sorting of images. Transferring some clips from the camcorder is very difficult. All or nothing downloads are not at all useful.

Currently running:

Corel VideoStudio 12
Corel GuideMenu
Canon HGBackup.exe

I have a little homebrew python script I use for managing clips from my old cameras, stills, and audio recordings. I'm going to be trying to update it to work with AVCHD (just picked up an HF S11) & would be happy to share it here once I get it functional. Because it's python, it works under both Windows and linux (my travel kit is a portable HD and a linux Eee PC).

I *do* like all-or-nothing transfers, though my script as-is never bothers transferring clips if they've already been sent.

Does anyone know offhand if I can run neoscene from the command line, so I could incorporate transcoding in a single step? Hmm.

[Kevin Bjorke]

Assumptions:
1. you have python on your system
2. you know how to use it to do things like change drive names

Save the code below as "kbImport2.py"

# /usr/bin/python

"""
# My quick "one size fits all" import and archive script.
#	THIS VERSION: 27 SEPTEMBER 2009
#	NO WARRANTIES EXPRESSED OR IMPLIED. THIS WORKS FOR ME. I AM JUST SHARING INFO.
#	I CHANGE THIS AROUND ALL THE TIME AS MY PERSONAL HARDWARE CHANGES.
#	OKAY TO REDISTRIBUTE AS LONG AS THIS NOTICE IS RETAINED IN FULL.
#
# Usage:
#	Plug in a card reader, camera, or audio recorder, and an external disk.
#	The External disk should have a directory called "Pix" to archive images.
#	The External disk should have a directory called "Vid" to archive video.
# 	The External disk should have a directory called "Audio" to archive sounds.
#	(Under windows, if there is no external they will be on "D:" and called "LocalPix" etc)
#
#	(windows) python kbImport2.py [JobName]
#	(linux) sudo python kbImport.py [JobName]
#
# Individual archive folders with names based on the FILE date will 
#    be created within those archive directories. The optional [JobName] maybe be
#	appended to the date, e.g. for JobName "NightSkate":
#	    R:\Vid\2009\2009-09-Sep\2009_09_27_NightSkate\AVCHD\BDMV\STREAM\02332.MTS
#
# Types recognized include Canon and Panasonic picture formats, AVCHD and QT and AVI files,
#	MP3 and WAV audio
#
# AVCHD support added -- it gets a bit complex for my Canon flash camcorder, as when it
#	mounts it mounts as MULTIPLE drives -- the internal flash, the internal HD, and
#	possibly an extra SDHC card.... this will tend to just get the G: drive until I
#	can figure out a better way to sort-through these. I also try to handle the Canon
#	thumbnail setup.
#
# Doing some experiments - Canon sets the creation time (which I had been using) on AVCHD files
#	to 1979, while the modification time is correct! So now using modification time. Will
#	tweak this for still cameras and audio as needed.
#
# Kevin Bjorke
# http://www.photorant.com/
"""

import sys
import os
import shutil
import time
import re

##############################################################
##### global variables and settings ##########################
##############################################################

# if gTest is True, create directories, but don't actually copy files (for testing).....
gTest = False
#if gForce is True, just copy always. Otherwise, don't overwrite existing archived files.
gForce = False

# my windows archive drives 
gBigDisk = 'R:'
gLocalDisk = 'D:'

# these will be loaded by the program
gPixDestDir = None
gVidDestDir = None
gAudioDestDir = None
audioDisk = None
gDCIMSrcDir = None
gOriginalMedia = None

# currently I want "" for my Edirol
audioPrefix = ""

# where AVCHD wants various types of files
global gAVCHDTargets
gAVCHDTargets = {}
gAVCHDTargets["MTS"] = os.path.join("AVCHD","BDMV","STREAM")
gAVCHDTargets["CPI"] = os.path.join("AVCHD","BDMV","CLIPINF")
gAVCHDTargets["MPL"] = os.path.join("AVCHD","BDMV","PLAYLIST")
gAVCHDTargets["BDM"] = os.path.join("AVCHD","BDMV")
gAVCHDTargets["TDT"] = os.path.join("AVCHD","ACVHDTN")
gAVCHDTargets["TID"] = os.path.join("AVCHD","ACVHDTN")
gAVCHDTargets["JPG"] = os.path.join("AVCHD","CANONTHM")

#########################################################################################
## FUNCTIONS START HERE #################################################################
#########################################################################################

#####################################################
## Find or Create Archive Destination Directories ###
#####################################################

def year_subdir(SrcFileStat,ArchDir):
  "Based on the source file's timestamp, seek (or create) an archive directory"
  # subdir = time.strftime("%Y",time.localtime(SrcFileStat.st_ctime))
  subdir = time.strftime("%Y",time.localtime(SrcFileStat.st_mtime))
  result = os.path.join(ArchDir,subdir)
  if not os.path.exists(result):
    print "** Creating dir %s **" % (result)
    os.mkdir(result)
  return result

def month_subdir(SrcFileStat,ArchDir):
  "Based on the source file's timestamp, seek (or create) an archive directory"
  # subdir = time.strftime("%Y-%m-%b",time.localtime(SrcFileStat.st_ctime))
  subdir = time.strftime("%Y-%m-%b",time.localtime(SrcFileStat.st_mtime))
  result = os.path.join(ArchDir,subdir)
  if not os.path.exists(result):
    print "** Creating dir %s **" % (result)
    os.mkdir(result)
  return result

def dest_dir_name(SrcFile,ArchDir):
  "seek or create an archive directory based on the src file's origination date"
  s = os.stat(SrcFile)
  rootDir = year_subdir(s,ArchDir)
  rootDir = month_subdir(s,rootDir)
  timeFormat = "%Y_%m_%d"
  # subdir = time.strftime(timeFormat,time.localtime(s.st_ctime))
  subdir = time.strftime(timeFormat,time.localtime(s.st_mtime))
  if len(sys.argv) > 1:
    subdir = "%s_%s" % (subdir,sys.argv[1])
  finaldir = os.path.join(rootDir,subdir)
  # should make sure it exists!
  if not os.path.exists(finaldir):
    print "** Creating dir %s **" % (finaldir)
    os.mkdir(finaldir)
  if not os.path.isdir(finaldir):
        print "path error: %s is not a directory!" % (finaldir)
        sys.exit(-4)
  return finaldir

############

def safe_mkdir(Dir):
  "check for existence, create as needed"
  if not os.path.exists(Dir):
    print "** Creating dir %s **" % (Dir)
    os.mkdir(Dir)
  if not os.path.isdir(Dir):
    print "path error: %s is not a directory!" % (finaldir)
    sys.exit(-4)
    # return None
  return Dir

def dest_avchd_dir_name(SrcFile,ArchDir):
  """
  AVCHD has a complex format, let's keep it intact so clips can be archived to blu-ray etc.
  We will say that the dated directory is equivalent to the "PRIVATE" directory in the spec.
  We don't handle the DCIM and MISC dirs.
  """
  privateDir = dest_dir_name(SrcFile,ArchDir)
  avchdDir = safe_mkdir(os.path.join(privateDir,"AVCHD"))
  avchdtnDir = safe_mkdir(os.path.join(avchdDir,"AVCHDTN"))
  bdmvDir = safe_mkdir(os.path.join(avchdDir,"BDMV"))
  streamDir = safe_mkdir(os.path.join(bdmvDir,"STREAM"))
  clipinfDir = safe_mkdir(os.path.join(bdmvDir,"CLIPINF"))
  playlistDir = safe_mkdir(os.path.join(bdmvDir,"PLAYLIST"))
  backupDir = safe_mkdir(os.path.join(bdmvDir,"BACKUP"))
  canonthmDir = safe_mkdir(os.path.join(avchdDir,"CANONTHM"))
  # should make sure it exists!
  return privateDir

#############################################################
## Recurse Throufgh Source Directories, and Archive #########
#############################################################

def archive_pix(FromDir,PixArchDir,VidArchDir):
  "Archive images and video"
  global gAVCHDTargets
  # first make sure all inputs are valid
  if not os.path.exists(PixArchDir):
    print "Hey, image archive '%s' is vapor!" % (PixArchDir)
    return None
  if not os.path.isdir(PixArchDir):
    print "Hey, image destination '%s' is not a directory!" % (PixArchDir)
    return None
  if VidArchDir is not None and not os.path.exists(VidArchDir):
    print "Caution: Video archive '%s' is vapor, Ignoring it." % (VidArchDir)
    VidArchDir = None
  if not os.path.exists(FromDir):
    print "Hey, image source '%s' is vapor!" % (FromDir)
    return None
  if not os.path.isdir(FromDir):
    print "Hey, image source '%s' is not a directory!" % (FromDir)
    return None
  # now we can proceed
  isAVCHDsrc = False
  m = re.search('AVCHD',FromDir)
  if m:
    isAVCHDsrc = True
  files = os.listdir(FromDir)
  files.sort()
  for kid in files:
    fullKidPath = os.path.join(FromDir,kid)
    if os.path.isdir(fullKidPath):
      archive_pix(fullKidPath,PixArchDir,VidArchDir)
    else:
      # if .MOV or .M4V or .MP4 it's a vid
      # if JPG, check to see if there's a matching vid
      isSimpleVideo = False
      isAVCHD = False
      avchdType = "JPG"
      kUp = kid.upper()
      m = re.search('\.(MTS|CPI|TDT|TID|MPL|BDM)',kUp)
      if (m):
	isAVCHD = True
	avchdType = m.group(1)
      m = re.search('\.(M4V|MP4|MOV)',kUp)
      if (m):
	isSimpleVideo = True
      m = re.search('(.*)\.JPG',kUp)
      if m:
	# keep an eye open for special thumbnail JPGs....
	if isAVCHDsrc:
	  isAVCHD = True
	  avchdType = "JPG"
	else:
	  root = m.groups(0)[0]
	  for suf in ['M4V', 'MOV', 'MP4']:
	    vidName = "%s.%s" % (root,suf)
	    if files.__contains__(vidName):
	      # print "List contains both %s and %s" % (kid,vidName)
	      isSimpleVideo = True # send the thumbnail to the video directory too
      if isAVCHD:
	avchdPath = dest_avchd_dir_name(fullKidPath,VidArchDir)
	destinationPath = os.path.join(avchdPath,gAVCHDTargets[avchdType])
      elif isSimpleVideo:
	destinationPath = dest_dir_name(fullKidPath,VidArchDir)
      else:
	destinationPath = dest_dir_name(fullKidPath,PixArchDir)
      kidTargetPath = os.path.join(destinationPath,kid)
      protected = gTest
      if os.path.exists(kidTargetPath):
	if gForce:
	  print "overwriting %s" % (kidTargetPath) 
	else:
	  protected = True
	  m = re.search('(.*).AVCHD',destinationPath)
	  if m:
	    destinationPath = m.group(1)
	    destinationPath = os.path.join(destinationPath,"...",kid)
	  print "%s already exists" % (destinationPath) 
      else:
	print "%s -> %s" % (kid,destinationPath) 
      if not protected:
	shutil.copy2(fullKidPath,destinationPath)

#############################################################

def archive_audio_tracks(FromDir,ArchDir):
  "Archive audio tracks"
  # first validate our inputs
  if audioPrefix != "":
    print "NEED     Filenames %sXXXX.MP3 etc" % (audioPrefix)
  if not os.path.exists(ArchDir):
    print "Hey, destination archive '%s' is vapor!" % (ArchDir)
    return None
  if not os.path.isdir(ArchDir):
    print "Hey, audio destination '%s' is not a directory!" % (ArchDir)
    return None
  if not os.path.exists(FromDir):
    print "Hey, track source '%s' is vapor!" % (FromDir)
    return None
  if not os.path.isdir(FromDir):
    print "Hey, track source '%s' is not a directory!" % (FromDir)
    return None
  # okay to proceed
  for kid in os.listdir(FromDir):
    fullpath = os.path.join(FromDir,kid)
    if os.path.isdir(fullpath):
      archive_audio_tracks(fullpath,ArchDir)
    else:
      fp2 = fullpath.upper()
      if fp2.endswith("MP3") or fp2.endswith("WAV"):
        # print "%s..." % (kid)
        trackDir = dest_dir_name(fullpath,ArchDir)
        print "%s -> %s" % (kid,trackDir) 
        # INSERT CODE FOR RENAMING HERE
        if not gTest:
                shutil.copy2(fullpath,trackDir)
      else:
        print "Skipping %s" % (fullpath)

# #

def archive_dir_name(ArchDir,BaseName):
  "pick the name of a good dated archive dir"
  if not os.path.exists(ArchDir):
    print "Hey, master '%s' is vapor!" % (ArchDir)
    return None
  if not os.path.isdir(ArchDir):
    print "Hey, '%s' is not a directory!" % (ArchDir)
    return None
  arch = os.path.join(ArchDir,BaseName)
  if not os.path.exists(arch):
    return arch
  counter = 0
  while os.path.exists(arch):
    bn = "%s_%d" % (BaseName,counter)
    counter = counter + 1
    if counter > 20:
      return None
    arch = os.path.join(ArchDir,bn)
  return arch

# #

def seek_named_dir(LookHere,DesiredName):
  "Look for a DCIM directory, which should have pix"
  if not os.path.exists(LookHere):
    return None
  for subdir in os.listdir(LookHere):
    fullpath = os.path.join(LookHere,subdir)
    if subdir == DesiredName:
      return fullpath
  for subdir in os.listdir(LookHere):
    fullpath = os.path.join(LookHere,subdir)
    if os.path.isdir(fullpath):
      sr = seek_named_dir(fullpath,DesiredName)
      if sr is not None:
        return sr
  return None

#########################################################################################
## MAIN EXECUTION BEGINS HERE ###########################################################
#########################################################################################

# SEEK SOURCE AND DEST DIRS ##

if os.name == "nt":
  print "Under windows I see"
  gPixDestDir = os.path.join(gBigDisk,"Pix")
  gVidDestDir = os.path.join(gBigDisk,"Vid")
  gAudioDestDir = os.path.join(gBigDisk,"Audio")
  if not os.path.exists(gPixDestDir):
    print "Drobo archive disk %s unavailable, using drive %s" % (gBigDisk,gLocalDisk)
    gPixDestDir = os.path.join(gLocalDisk,"LocalPix")
    gVidDestDir = os.path.join(gLocalDisk,"LocalVid")
    gAudioDestDir = os.path.join(gLocalDisk,"LocalAudio")
  if not os.path.exists(gPixDestDir):
    print "Something is broken? No dir %s" % (gPixDestDir)
    exit()
  for srcDisk in [ "G:", "F:"] :
    if os.path.exists(srcDisk):
      gOriginalMedia = srcDisk
      gDCIMSrcDir = seek_named_dir(srcDisk,"DCIM")
      if gDCIMSrcDir is None:
	gDCIMSrcDir = seek_named_dir(srcDisk,"AVCHD")
      if gDCIMSrcDir is None:
	gDCIMSrcDir = seek_named_dir(srcDisk,"PRIVATE")
      if gDCIMSrcDir is not None:
	break
elif os.name == "posix":
  print "Looks like Eee to me"
  disktop = "/media"
  posixDiskList = os.listdir(disktop)
  if len(posixDiskList) < 2:
    print "Sorry, wrong number of disks: %d" % (len(posixDiskList))
    for d in posixDiskList:
      print '"%s"' % (d)
    sys.exit(-1)
  for d in posixDiskList:
    disk = os.path.join(disktop,d)
    if gPixDestDir is None:
      gPixDestDir = seek_named_dir(disk,"Pix")
      if gPixDestDir is not None:
        gVidDestDir = seek_named_dir(disk,"Vid")
        gAudioDestDir = seek_named_dir(disk,"Audio")
      else:
	gOriginalMedia = disk
	if gDCIMSrcDir is None:
	  gDCIMSrcDir = seek_named_dir(disk,"DCIM")
else:
  print "Sorry no code for OS '%s' yet!" % (os.name)
  exit()

#
# If there's a DCIM directory, we're doing images and video clips.
#        If not, we must have an audio SD card.
#
if (gDCIMSrcDir is None):
  # AUDIO #########################################
  if gOriginalMedia is None:
    print "WARNING: No original media found"
    sys.exit(-3)
  print "Archiving Audio from '%s'\n\tto '%s'" % (gOriginalMedia,gAudioDestDir)
  archive_audio_tracks(gOriginalMedia,gAudioDestDir)
else:
  # PIX and VIDEO #########################################
  print "Archiving Images from '%s'\n\tto '%s'" % (gDCIMSrcDir,gPixDestDir)
  archive_pix(gDCIMSrcDir,gPixDestDir,gVidDestDir)

[Olivier Jezequel]

Originally Posted by Kevin Bjorke

Does anyone know offhand if I can run neoscene from the command line, so I could incorporate transcoding in a single step? Hmm.

If you can run neoscene (whatever it is :) from a shell, you can try that piece of python :

import os
os.system('neoscene')

This will work only if your shell recognize "neoscene" as a valid command in a shell

What exactly you script do ? i am about to buy a hfs11 and my mac is python friendly :)

Oups, i just noticed that your question was how to make work the command in a shell, and you already had the import os under control
it has maybe something to do with your env.csh but i know for exemple that shortcuts that you will create in it will not work from your python, it doesn't seems to read your .csh
I will investigate


cheers
Olivier

[Graham Hickling]

Maybe this? ....never tried it with avchd, but its free: TMPGEnc - Products: TMPGEnc KARMA.. Product Information

(Trying to leave selected stuff on my camera would give me a headache too, so I never do it.)

[Olivier Jezequel]

Originally Posted by Kevin Bjorke

Does anyone know offhand if I can run neoscene from the command line, so I could incorporate transcoding in a single step? Hmm.

if i understood well, you need to put a copy (maybe an alias??) of your executable into $HOME/tools/scripts
then its name should be recognized into a shell, so by python. But there is surely an other way in python to access your exe, i don't know enough

thanks for sharing your tool

[Kevin Bjorke]

Thanks Olivier -- it wasn't a question of "how do I run a command line process" but more a specific query about the Cineform Neo Scene program itself, which is usually driven by an interactive dialog.

It would be great if I could run it as a command-line program, too, so that I could just include the transcoding step in the importer.

The specific of using python: no problem. The specifics of using NeoScene: maybe (there's certainly nothing in the manual about this usage).

Thanks again,
kb

[Nick Mirro]

Originally Posted by Graham Hickling

Maybe this? ....never tried it with avchd, but its free: TMPGEnc - Products: TMPGEnc KARMA.. Product Information

(Trying to leave selected stuff on my camera would give me a headache too, so I never do it.)

Thanks for the tip. The app looks promising. Also you got me to softpedia, which also has MediaInfo and other possibilities. Appreciate it.