9/23/2006

09-23-06 - 1

How can I write data to disk quickly/incrementally and be sure that it is safely being flushed if my app crashes?

What I'd like to do is have my file handle open, prep all my data for writing a whole valid record, then flush out that record using an OS call that I know can't fail (or if it fails, the whole thing fails, never a partial failure). I'd like that write to be buffered, HOWEVER, I want the buffers to never flush partial records, or if my app crashes, I want Windows to completely flush the buffers to disk. Worst case if the computer is struck by lightning and suddenly powers off, I don't want partial records to be written. (I guess if the HD was actually in the process of writing when you power off that's going to screw you no matter what, but I want other failure cases to not happen).

eg. say my records are 3000 bytes long. Pretend Windows flushes buffers in 4k chunks. I write two records, the first 4k is flushed and the last 2k of the end record stays in the buffers. Now my computer powers off. The disk is left with a partial record written which is very bad.

I can't really find good information on how Windows manages it's file buffering and when exactly things are flushed and how it handles catastrophic shutdowns. It just says "when an app is terminated normally the buffers are flushed". Umm, what exactly is "normal" and how do you handle abnormal terminations? I think I can write my own buffered layer which is aware of the requirement of only writing integer number of records. In my case it's not a huge problem if I simply fail to write some records under abnormal termination, that's just losing the last few records, not corrupting my database.

Addendum : Two ideas :

1) Use "rename" ; basically write to a temp file then swap it ( details here ). My data file is huge (500 MB), so I'd have to double-buffer or something like that; there's some nasty step where you have to catch the mirror up with the other side which itself could fail.

2) Write separate files then occasionally collate to a copy. Instead of appending to one file, you have your base data file, "F1", then write all your records from a session to a new file "F2", etc.. each time you start up you read F1..FN. Occasionally you collate, F1+F2+F3.. -> Collated. At the end you rename Collated to F1 and delete F2->FN. If you ever crash while writing a file it might be corrupted and you just kill that file from the set.

No comments:

old rants