'Buffer Overflow' problem: New possible explanation

[Adam Welz]

Hi All

I've just been alerted to a post by Curtis Anderson dated 20 March 2010 in a thread I started which, to me, gives a good lead on what might be causing these buffer overflows and crashes. In short, the crashes are not triggered by actual buffer overflows -- it's an incorrect error message -- but rather by corrupted/bad sectors in flash memory that the camera can't deal with.

Curtis writes "Based on my personal experiences, data to flash memory units can become corrupt during a read/write cycle if power is interrupted. One corrupted, the unit must be reformatted. Some bit errors can still remain after a format and the unit may not work, or work until the bad spot is addressed again during usage, and in some cases may not reformat at all. I have been able to try another camera and correct the corrupted data flash unit with a good format. My thought is, one or both of your units got corrupted and the camera, with limited error reporting, limited on-board diagnostics, and no error correction feature was giving you a ‘catch-all’ error message with little to no meaningful explanation of the real trouble."

Prior to me experiencing the deep crashes on both my NXCAMs I did in fact have clips that were interrupted, in other words while recording the camera was shut down before I hit a REC button to end the clip 'normally'. Neither of my NXCAMs crashed until after I'd recorded clips that had been non-standardly interrupted.

With the first camera I turned the main power switch off while recording to test the cam, and with the second camera the battery went completely flat in the middle of a clip. This may have resulted in corruption of a small section of the flash memory in either or both the SD memory cards and/or the FMU128 flash unit. These bad sections will in theory survive a standard reformat/erase procedure, and then when the camera tries to re-record data over the bad sections, it fails and this causes a deep crash of the entire system.

This would explain the lack of correlation between card read-write speed and the crashes, and clip length and complexity and the crashes.

So, perhaps the HXR-NX5 does not have data bottlenecks or even any specific issues with writing data to SD cards (which some have thought to be the problem). The problem might rather be that the software cannot recognise and skip over bad sectors in flash memory, and also not robust enough to recover from this encounter OR (in many cases) to recover the interrupted clip.

I will now be testing my idea by 'deep reformatting' an SD card (to make sure it's as clean as it can be, so it starts recording 'at the beginning' so to speak, what Sony calls a MEDIA EMPTY operation on page 65 of the NX5's manual) and then interrupting a clip by being really awful and pulling the battery off in the middle of it, and then doing a 'standard' reformat and using the card again and seeing if the cam freezes up when it encounters the bad sector that should theoretically be lurking on the card.

As I said before, at this point it's unclear whether the cam struggles with bad sectors on the SD card or FMU128 or both because all my crashes have happened when recording to both simultaneously. I'll be trying to crash the cam using only an SD card initially.

I'd love feedback from other users!

Cheers

Adam

[Fred Davis]

Dude you are the best it sounds like it you found the problem, but keep me posted

[Daniel Paquin]

Excellent information Adam,

These bad sections will in theory survive a standard reformat/erase procedure, and then when the camera tries to re-record data over the bad sections, it fails and this causes a deep crash of the entire system.

I've read the MEDIA EMPTY instructions you are referring in the manual. It says this operation may take minutes and hours. Also says, if you stop this operation you must make sure to continue the next time prior using this card.

This operation cannot be done when you shooting, you've got to plan your time and have sufficient memory card with you. Might be better to have 16gib card rather than 32, 64... and so on. Bigger you card is, probably longer this operation would take.

I am wondering if reformatting the card with a MAC would do it faster and in the meantime repair the section in error. Just a thought.

I do not know if Sony has a deck which would do all of those functions et let the cameras does what it is supposed to, I,e film.

Have you talked to your contact with Sony. Meaning which would be the best route to reformat the card, EMPTY with the camera, reformat with Windows xx or a MAC, or a special deck.

Thanks from all of us, for you performing all of those operations.

P.S.: I hope the card is not permanently damaged, that would be awful.

[Ron Evans]

The instruction are clear that formating MUST be done on the camera. I expect there are specific formating that prepares the card for use including setting up directories etc. Base format is FAT32 though.
Just as a test I have done a MEDIA EMPTY on my 32G Patriot Class 10 card. I first emptied the card and then did the clean. It only took about a minute, just a little slower than a format.
Sony do say that Memorystick PRoDuo HG are considerably faster than SD card but of course cost a LOT more.
I expect one must take care with stopping before removing power as this is just the same as removing a USB drive from a PC before its has finished writing.( in that case the FAT table can be corrupt and data lost on the drive or the next time it is plugged in will not be recognized). I would have thought that Sony would have built in a means of avoiding this. Pulling out the battery would definitely do it though!!!
Switching off power on a tape camcorder will just corrupt the last frame not the whole file though.

Ron Evans

[Daniel Paquin]

With the type of filming that I do I cannot wait after the camera reformatting a card.

When I start my day, I turn on the camera and leave it on for a period of 10 to 14 hours.

I film small clips of 3 to 6 minutes each with a break of two minutes between each clips. Sometime, only a second between each clips. Each clips are part of an event which has subjects to film that are one after the other. Provincial and national events, I do have two minutes, local events I stop and go between each subjects. This type of filming goes on for three to five days.

At the current time, with any of cameras, I've never filmed on miniDV tape, VHS tapes..., I always had my camera hooked up to a HDD via the firewire. I produce DVDs on the fly, customer leaves with their DVD in their hands no more than 30 minutes after their performance.

This is the first time, I am having a workflow which makes me recording on the camera's media. This why I need to trust the camera from A to Z.

All of this to say, I've got no time to use the camera to format or repair a SDHC card with bad sector. I must then find a tool, a deck, a function which would allow me to find in advance if the SDHC cards I've got has bad sector. If so I must repair them to ensure their OK before starting to use them. All of this must be done prior starting those three to five days.

I will be switching SDHC cards about 50 times a day on my cameras. Someone else at the front will be emptying the cards and produce the DVDs.

This is why I must find another solution then the one recommended by Sony. When I've talked to the Sony Montreal Rep he had told my this camera was exactly designed for people who has similar needs then mine. 99% of my needs are not to perform video editing. I shoot, I deliver and the clients goes. There are the odd time, where I would be filming year end show and then perform video editing,

I wish someone would know another reliable solution than using the camera to repair a SDHC card with bad sector.

Regards!

[Ron Evans]

Just did a Media Empty on the FMU to see how long it would take. It took 1 hour and 40 mins.

Ron Evans

[Marshall Levy]

Interesting info, Adam. It's quite parallel to the Z7U when that came out with regards to the testing I did on CF cards and their performance and stability. Many people I spoke with during the time I did the testing - and those who had problems - were using knock-off cards and/or were not properly formatting the cards prior to recording on them with the camera. Once the people (and even in my testing) formatted their cards and/or used better cards in general, many of the initial problems were no longer.

[Dave Blackhurst]

Good job troubleshooting!! If you're correct, I'll bet that a firmware update would fix this - something that either prevents the corruption, fixes it on "reboot", or ignores it would do the trick.

We all know that batteries never die mid clip, and we never accidently turn a camera off... but those are the sorts of things might be missed/overlooked/forgotten when developing the firmware.

Can you intentionally "crash" while recording and repro the problem, because that would put the squish on the bug for certain, but I think you may have it, in which case you should get the "most valuable user" medal from Sony!

[Ron Evans]

Interesting point on formatting. I started by formatting both the FMU and the SD card twice ( it takes less than 10sec anyway) and its a habit I have got into with cards for most things.
Ron Evans

[Ron Evans]

FAT or file error will occur on cards or hard drives if power or connection is remove prematurely. In this case there are two instances. In the camera or while connected to a PC. What PC's are being used to capture from the cards? And is the PC being used to release the USB connection properly before disconnecting? This requires a specific action on the PC . Dont have a MAC so do not know how this is done on a MAC. These are possible occassions where FAT or file errors could arise on the cards or FMU. Are files being deleted in the camera or on the PC?
I am on a PC running Vista 64 using the Content Manager to transfer files and deleting files on the camera. So far I have had no problems. Only obvious indication to me is that the red activity light on the SD card takes longer to go off when I stop recording than the red light on the FMU. So it must take longer to close the session.

Ron Evans

[Adam Welz]

Earlier I wrote

"In short, the crashes are not triggered by actual buffer overflows -- it's an incorrect error message -- but rather by corrupted/bad sectors in flash memory that the camera can't deal with."

What is of course a possibility is that a corruption in the media could hold up a write operation which in turn could cause a buffer overflow -- so perhaps the error message is correct.

Cheers

Adam