Ki Pro Mini and heat - is there an issue?

[Charles Papert]

For my Comedy Central series, we settled on using the Ki Pro Mini's as our recorders with the F3's. The idea was to record the 4:2:2 log output of the camera, then run that video into Blackmagic HDLinks to delog and dial in a basic LUT for on-set viewing and editorial purposes, and record that into a second set of Ki Pro Mini's. To keep things organized, we decided to keep all four KPM's together on the engineering cart. We would have to run SDI from the camera anyway so the cameras would always be "hosed" even if the KPM's were camera-mounted (except for a few elaborate Steadicam shots where we would camera-mount and use an HD transmitter).

On the first day, a hot day exterior, two of the KPM's started kicking out of record during takes. They were erratic; sometimes they would record for a few seconds and sometimes for longer, but eventually they would just stop recording on their own. In a fever of "trying to get through it", we swapped cables, cards, assignments of which decks were recording which signal, etc etc. The units were screaming hot to the touch, and we did what we could to improve airflow (we didn't have the standing bases so we propped them up on wedges to allow air to move around each of them). They were inside the DIT tent so out of direct sun, but the air temperature was quite high.

Eventually, they appeared to be working, in that they started and stopped on cue, with the green light indicating that they were operating as expected. The frazzled data wrangler was spot checking certain takes.

At the end of the day, it appeared that a large swath of footage was missing. Nearly all of the clips that were recorded during the "stop/start" trouble did not exist, but even later, when the decks were giving every indication of recording successfully, only the last take on each card survived. It should be noted that this was after we had swapped camera assignments, removing the possibility of bad cables or issues with the cameras themselves (i.e. the KiPro's had simply "gone bad", rejecting acceptable input signals).

One of our crew members was experienced in data recovery and he put the cards through a lengthy process to see if anything was salvageable; he did managed to extract some fragmented clips which he reassembled but it was just a fraction of what was lost for good. All in all, we lost somewhere around 35 individual shots, representing 1/3 to 1/2 of what we shot that day (between two cameras).

This, by the way, was the first day of shooting on a network series. You can imagine that the ramifications from all of this was pretty intense. It resulted in personnel and equipment changes and a lot of mea culpa. On day two we started recording to XDCAM disc recorders, which were unfortunately a lot more expensive to rent but we have not lost a frame since.

Ironically, I selected the Ki Pro Mini's after doing a lot of research and being persuaded out of other choices due to reliability issues. The Gemini, incidentally, was not yet available when we started the show. I've since heard from many people who have not experienced issues with the Ki Pro Mini so this is obviously not a widespread issue, but with so many variables in place and such a disastrous experience, I haven't had much interest in doing an elaborate post-mortem to figure out what might have happened (I'm kind of busy shooting a TV show!).

[Nick Hiltgen]

Charles is there a reason you guys went with the mini's instead of the regular ki pro's? It seems that if they're going to live on a cart you would want the more time tested device?

[Brian Lai]

Maybe 4 minis stacked together look way cooler on a cart than 4 flat decks? There's also the steadicam issue and when KPM's are onboard, operator can be tether free when powered by an AB battery which also powers and balances the camera. But, the bugs are annoying and not the worry free Nano Flash experience.

[Charles Papert]

As hinted above, yes, I wanted to protect the possibility of going untethered. In the first two weeks I had a Steadicam shot that stepped on and off a crane, and two dollys doing endless rotations on circular track during a 5 page scene. Ultimately we worked both out without the KPM's.

[William Hohauser]

I was running my Ki Pro Mini in my control room today and it did some strange things before crashing. While I had used it for shooting 10 minute or less takes in all my productions up to now, today we were filming long 15 minute takes and the control room was warm, around 80 degrees. Usually it's not like this but the building heat was on full blast despite the warm temperatures outside. The Ki Pro Mini was working well that past two days through the heat but the average take was about 7 minutes or less.

Anyway about an hour into production I noticed that the Ki Pro Mini had switched itself over to custom take which erased the previous recordings. As I was the only person operating the unit I can say that nobody was playing with it and there was no way this would have happened by accident. I switched it back to regular clip capture and resumed recording. About 10 minutes into the take the unit stated flashing all the button lights and then said "Dropped Frames" on the LCD and it stopped recording. I switched over to the second card and resumed recording for the next take. When the second card filled up I removed it to dump the footage, I tried to return to the first card. The Ki Pro Mini tried to mount the card (a recommended Delkin Devices 32gb CombatFlash) but the Ki crashed and I had to disconnect the power supply since it refused to turn off. That $300 card is now useless.

Fortunately I was running a second recording system within the Tricaster switcher and the footage wasn't lost. Is there a heat issue? I noticed the the Ki Pro Mini was very warm when the crash occurred so I turned a small room fan on the unit and continued successfully with my only card for the rest of the day. On a side note, a new 64gb SanDisk Extreme Pro card crashed two days ago as well and I am returning it

[Jon Thorn]

William H. - First, what you are describing sounds very unusual. I would suggest contacting AJA Technical Support for assistance. You might want to provide them with the following information:
- Was the unit used in the AJA Mini Stand or did you simply lay it on it's side in the machine room? (The sides of the Ki Pro Mini are designed to remove heat from the unit, so if you lay it on it's side, you could potentially be trapping heat that would normally be dissipated by the finned side surfaces).
- What fan setting were you using? The Ki Pro Mini ships with the fan setting in the Quiet Record mode for optimal quiet operation when used beside a camera, but for situations where that is less important and cooling the unit is more desirable, Normal and Quiet Auto might prove to be better choices; Normal spins the fan at a higher, constant rate compared to Quiet Record and Quiet Auto runs the fan at the higher speed, but then ramps the fan speed down during recordings.
- It's doubtful that the Delkin card you have is useless; you may just need to use a CF card reader attached to a Mac OS X computer to recover the card (unexpected card removals or unusual shutdowns can result in the directory of the media being compromised, but typically connecting to a Mac OS X computer will repair such directory errors; if there if footage on the card, back it up to your computer and then reformat the card in the Ki Pro Mini).
- Regarding your SanDisk Extreme Pro card, I would suggest the same process and I would also suggest only buying CF cards from reputable dealers; counterfeit cards do exist for the more popular brands. If you have a counterfeit card, it will almost certainly behave erratically or not want to operate at all.
- Dropped Frames prompts mean that the media is underperforming in most cases or, in some very rare cases, the signal is SDI signal is problematic in some way; if the media has ever been removed without using the SLOT button, it's a good idea to reformat it even if you recover it with a Mac OS X computer (fragmented directories will eventually lead to dropped frames on media). Also, if the media has been used several times, and clips simply deleted from the card without ever reformatting the card in the Ki Pro Mini, that could also potentially lead to an issue as the media directory becomes fragmented over time. Finally, a compromised signal - one that is valid enough to pass over SDI but is problematic in other signal contexts - could potentially cause such an alert, but that is very, very uncommon.
- The Ki Pro Mini doesn't switch menu structures on it's own. We have never seen that in very extensive testing of the units; it's always possible that the menu was inadvertently changed while moving through the MEDIA menu when looking to change some other menu parameter. Also if you are concerned about the Custom Clip Take parameter, know that it was removed from the menu structure due to people misunderstanding it's operation beginning with the current version 2.6 firmware; you must be running earlier firmware on the unit in question then. You might also considering running the latest firmware for the unit as new features have been added with each new version.

Again, I would encourage you to contact AJA Technical Support for assistance; they will be happy to help you. They are available from 9am to 5pm, Monday through Friday via phone or email. I'm sorry that you experienced any issue with the unit, but as I said, this sounds out of the norm; I'm sure the support staff at AJA can help you sort things out.

Hope this is helpful,
Jon Thorn
Senior Product Manager
AJA Video Systems, Inc.

[William Hohauser]

Thanks for responding.

Yes, the unit was in it's stand but it was very warm, certainly around 120 degrees on the sides. How much space do you recommend beyond the confines of the base? I have about two inches to one side and nothing to the other side. Regarding the fan, I had set the unit to Normal before we started the production as it is being used in a control room where the fan noise isn't an issue. Once I turned a room fan on it, it never got as warm.

The Delkin card wouldn't mount in my iMac running 10.6 but I haven't tried to mount it on any of my other computers yet. I am in backup mode now converting the Tricaster footage to ProRes for the editor. My routine is to dump the cards to a hard drive (and then back up the hard drive) and then format the cards before using them again.

The SanDisk card came from B&H and I have already set an RMA in for it. This was one of the new price reduced SanDisk Extreme Pro cards that B&H announced two weeks ago, perhaps there is something wrong with the new batch of cards that AJA might find useful to look into for their customers' benefit. Or perhaps I just lucked out and got a bad card.

The menu change was very odd as the card was only 25% used and the only buttons being pushed were record and stop. The config menu wasn't even near the Custom setting when I went to turn it off, a power surge perhaps?
On Monday I will call tech support and update the firmware, thanks for the advice.

[Charles Papert]

Note: I moved my post and the three responses from another thread into this one as it describes a very similar issue to what William has experienced. Because they are older posts, they appear in the chronological order here, now at the top of this thread. William, no intention to hijack your thread, I think it's worth it to pool experiences in these matters.

[Jon Thorn]

Charles P. - Where did you move this thread from? I never saw it here in the AJA sub-forum before or I would have likely replied. Second, when you ran into your issue, did you contact AJA Technical Support? If not, that's a great place to start; our tech support is provided free of charge and they are happy to help. Finally, as noted in what is now a combined thread (somehow), there isn't information about how the units you used were handled: Were they laid on their sides or used in the stands? Stacked on top of each other if laid on their side or items stacked on them? Was the fan speed set to Normal due to the environment? Best to relay such information to AJA Technical Support.

Again, I would stress to everyone that when you experience an issue, your best course of action would be to contact AJA Technical Support; we go so far with our support that we will perform advanced exchanges where, if you suspect there is an issue with a piece of AJA equipment, we will ship a replacement before you ship your unit in for inspection. We endeavor to help our customers... but it is difficult to properly help our users if they don't contact us directly... it's just not realistic for me to try to find every post on the internet where someone might need assistance... mind you, I make an effort, but it's impossible. I appreciate dialogue or questions on these and other forums, but know that if you need help, a dedicated support service is provided by the manufacturer of the product.

Hope this is helpful,
Jon Thorn
Senior Product Manager
AJA Video Systems, Inc.

[Charles Papert]

Jon:

It was in this thread but after I posted, Nick Hiltgen pointed out that that was referring to the Ki Pro and not the Mini.

In the heat of the day (literally, figuratively and otherwise) we didn't contact AJA and I wish that we would have. The whole story is a long one and involves some personnel-related aspects that I don't want to go in to, but suffice to say that had it been my job to problem-solve the issue directly, I'm sure I would have done just that.

I did indeed indicate in my first post how the units were stacked. Not in stands and initially on their sides, but once we had issues, we propped them up to allow airflow. The most diabolical issues (indicating recording but not actually doing so) came after they were better set up physically. As far as the fan settings, I don't know. I believe we reset the units to factory default so presumably Quiet Recording--obviously they would have been better off in Normal or Quiet Auto since they were not on board the cameras.

There were simply too many variables that day to know exactly what went wrong and why. What I do know is that two of the units indicated that they were recording without errors, but the net result was that they weren't.

Unfortunately we couldn't afford to take any further risks and had to move to a different technology. It's not to say that I won't try the Ki Pro Mini's again under different circumstances. My intent to tell this tale is not to badmouth the unit but to see if others have experienced this issue, gather anecdotes, learn what solutions may or may not help. The fan speed adjustment is definitely a new piece of information that I wish we knew that day (although for all I know, the DIT made it and I wasn't aware).

[William Hohauser]

Yes I will second that my long term intention was always to call AJA tech but this problem occurred, just as in Charles case, at a time of production stress in addition to being at the end of a long work week in a building that was already 80 degrees at nine in the morning. I find that these forums are a great place to share work techniques and sometimes good companies like AJA and JVC use them to keep an eye out for their customers. Thank you for being here. So first of all my intention was not to bad mouth the product but to discover if I was using it in a way that was detrimental to it's proper functioning. Over the past 30 years I have worked with excellent equipment that had heat or cold or other issues and a helpful tip was all that was needed to have a trouble free production.

In the three months of using the AJA Ki Pro Mini in a normal environment it has been trouble free and a great alternative to expensive HD deck rentals. Now if somehow I could convince Panasonic to fix the 24p HDMI output of the GH2 DSLR I would have a great 4:2:2 remote combo.

[Jon Thorn]

Charles P. - "Gathering anecdotes" from internet forums seems to be the norm these days... but it comes with a lot of pitfalls I'm afraid, not just for the manufacturers, but also, and most importantly, for the users of the products. Forum posts tend to be like "soundbites" when compared to the type of detailed dialogue that can be had when working with a manufacturer's technical support staff. Posts on forums rarely go into the level of detail about each and every parameter involved in a potential issue the way working with a technical support department might. The end result is an incomplete picture of a particular, and often, specific situation.

In the future I would urge everyone to learn from this: if you have an issue with an AJA product, contact AJA Technical Support immediately. They are your best resource for trying to determine what might be causing an issue in a timely fashion. In some markets, there are even field system engineers that might be able to visit your location "in the heat of the day."

At this point, this particular example is largely anecdotal I'm afraid: we don't know what the fan settings were during the shoot, we don't know what type of CF media was used and if it was potentially problematic to the recordings (Lexar media for example, proved to be problematic, and was removed from our qualified CF card list), etc. We do know that the units were placed on their sides in a hot environment which likely impacted their operation because that isn't the optimal or recommended way to operate Ki Pro Minis and that likely contributed to at least some of the trouble that was experienced.

This statement, "There were simply too many variables that day to know exactly what went wrong and why," makes me really feel for you. Production isn't easy... I know, I was a 1st A.C. for several years; sometimes things move fast - too fast - on set and sometimes not all of the personnel on a project have the level of knowledge or experience to problem solve a situation adequately... and that's not a judgement about the quality of a person's work or character (nobody can know everything) - just know that some companies, like AJA, are there if you do need help.

Again Charles, sorry you experienced an issue. Always feel free to contact AJA Technical Support, or as might have been the case with this particular project, encourage others to contact AJA if that isn't your role or responsibility on the project.

Regards,
Jon Thorn
Senior Product Manager
AJA Video Systems, Inc.

[Charles Papert]

Thank you Jon for the thoughtful response.

This event occurred above five weeks ago and I certainly had no intention to immediately jump onto numerous forums and scream "nobody use Ki Pro Minis! They lose footage!". Certainly not my style and we discourage that sort of thing on DVI anyway. In fact, I had moved on from it until I read of the fairly similar circumstances experienced by William.

The "blame" for this issue lay in many hands but they all came down to one thing: a budget that didn't allow for the right prep. Ideally we would have started that show with an experienced DIT who had ample time to spec the proper physical setup of the recorders (and assure his familiarity with them) and work with the rental house to mount them onto the cart properly. As to the cards--I can't remember the brand (one of the majors), but it's certainly possible they were not on the AJA approved list. It's possible that we could have avoided the issue if everything was properly set up and as soon as the error messages began to appear, we could have headed them off at the pass.

As you can imagine, the pressure of losing 1/3 or more of your first day's footage on a network series was intense to say the least. Under some circumstances, a mass firing would occur. Thankfully there was an inherent understanding in this case that the budget had forced us to cut corners, a number of which likely factored into the end result. Only one person was let go and we moved on (and reshot enough to salvage the scenes). Had there been more of a witch-hunt, I would have demanded a serious post-mortem including sending the units back to AJA along with the media to attempt a recreation of the issue. It was decided to place our energies elsewhere, so the mystery remains.

Ultimately, it's a sad sign of the times that even at the level of prime time network production, such risks are forced upon us. We are shooting great stuff at a breakneck pace and everyone is happy with the material, but we are always on the verge of "having the wheels come off" if there are unforeseen issues such as these.

So: agreed that the moral of the story is to call the manufacturer. Some are more available than others, some will jump through hoops to ensure that you are well-taken care of (these are the companies that gain my trust). It's worth considering when one buys gear: that inexpensive knockoff on eBay that saves you a few hundred over the name brand may not be such a bargain when you can't get parts or service for it without serious delays (if at all) and your project and reputation suffer as a result.

[Ed David]

Always dual record. My PIX 240 was having issues on a shoot I was doing in Vegas and I was lucky I also recorded to the SxS cards. And overall SxS is not the end of the world. I'm always suprised by how well XDCAM EX holds up. And anyway, once the thing is broadcast the signal is so heavily compressed and still I heard 80% of people watch tv still in sd - so does Pro Res HQ really make that much of a difference in the end? Of course we would like to think it does, but hey, SxS holds up fine.

I have a KI Pro Mini - done probably 15 shoots with it - overall I am quite pleased with it. It hasn't failed me yet. The problems I have with it - battery meter issues are being resolved in the new firmware update. It gets the job done. And when it hypothetically doesn't there is the on board recorder to bail me out if I need to be bailed out.

[Scott Stoneback]

Backup to internal camera media is integral to my workflow. It is so easy to misstep and record incorrectly on an external recorder. I've re-run media off tape in my HDX a few times, and re-recorded on my NanoFlash. It saved my butt.

You know, I have to say that the Nano has been a real workhorse and does not ever have heat issues... The AJA stuff is nice but the Nano is really getting the job done. If anything, it has too many functions and is complicated to learn. But as far as mechanical reliability, it is superb. Never a problem with heat or cold.