dbda8d6ce9 2007-07-21 drh: <html> dbda8d6ce9 2007-07-21 drh: <head> dbda8d6ce9 2007-07-21 drh: <title>Fossil Repository Integrity Self-Checks</title> dbda8d6ce9 2007-07-21 drh: </head> dbda8d6ce9 2007-07-21 drh: <body bgcolor="white"> 469002ccdf 2007-09-12 aku: <p>[ <a href="index.html">Index</a> ]</p> 469002ccdf 2007-09-12 aku: <hr> dbda8d6ce9 2007-07-21 drh: <h1 align="center"> dbda8d6ce9 2007-07-21 drh: Fossil Repository Integrity Self-Checks dbda8d6ce9 2007-07-21 drh: </h1> dbda8d6ce9 2007-07-21 drh: dbda8d6ce9 2007-07-21 drh: <p> dbda8d6ce9 2007-07-21 drh: Even though fossil is a relatively new project and still contains dbda8d6ce9 2007-07-21 drh: many bugs, it is designed with features to give it a high level dbda8d6ce9 2007-07-21 drh: of integrity so that you can have confidence that you will not dbda8d6ce9 2007-07-21 drh: lose your files. This note describes the defensive measures that dbda8d6ce9 2007-07-21 drh: fossil uses to help prevent file loss due to bugs. dbda8d6ce9 2007-07-21 drh: </p> dbda8d6ce9 2007-07-21 drh: 6680679c2e 2007-11-24 drh: <p><i>Follow-up as of 2007-11-24:</i> 6680679c2e 2007-11-24 drh: Fossil has been hosting itself and several other projects for 6680679c2e 2007-11-24 drh: months now. Many bugs have been encountered. But, thanks in large 6680679c2e 2007-11-24 drh: part to the defensive measures described here, no data has been 6680679c2e 2007-11-24 drh: lost. The integrity checks are doing their job well.</p> b807acf62e 2007-07-24 drh: dbda8d6ce9 2007-07-21 drh: <h2>Atomic Check-ins With Rollback</h2> dbda8d6ce9 2007-07-21 drh: dbda8d6ce9 2007-07-21 drh: <p> dbda8d6ce9 2007-07-21 drh: The fossil repository is an b807acf62e 2007-07-24 drh: <a href="http://www.sqlite.org/">SQLite version 3</a> database file. b807acf62e 2007-07-24 drh: SQLite is very mature and stable and has been in wide-spread use for many dbda8d6ce9 2007-07-21 drh: years, so we have little worries that it might cause repository dbda8d6ce9 2007-07-21 drh: corruption. SQLite dbda8d6ce9 2007-07-21 drh: databases do not corrupt even if a program or system crash or power dbda8d6ce9 2007-07-21 drh: failure occurs in the middle of the update. If some kind of crash dbda8d6ce9 2007-07-21 drh: does occur in the middle of a change, then all the changes are rolled dbda8d6ce9 2007-07-21 drh: back the next time that the database is accessed. dbda8d6ce9 2007-07-21 drh: </p> dbda8d6ce9 2007-07-21 drh: dbda8d6ce9 2007-07-21 drh: <p> dbda8d6ce9 2007-07-21 drh: A check-in operation in fossil makes many changes to the repository dbda8d6ce9 2007-07-21 drh: database. But all these changes happen within a single transaction. dbda8d6ce9 2007-07-21 drh: If something goes wrong in the middle of the commit, then the transaction dbda8d6ce9 2007-07-21 drh: is rolled back and the database is unchanged. dbda8d6ce9 2007-07-21 drh: </p> dbda8d6ce9 2007-07-21 drh: dbda8d6ce9 2007-07-21 drh: <h2>Verification Of Delta Encodings Prior To Transaction Commit</h2> dbda8d6ce9 2007-07-21 drh: dbda8d6ce9 2007-07-21 drh: <p> dbda8d6ce9 2007-07-21 drh: The content files that comprise the global state of a fossil respository dbda8d6ce9 2007-07-21 drh: are stored in the repository as a tree. The leaves of the tree are dbda8d6ce9 2007-07-21 drh: stored as zlib-compressed BLOBs. Interior nodes are deltas from their 6458f020fc 2008-05-14 drh: decendants. A lot of encoding is going on. There is dbda8d6ce9 2007-07-21 drh: zlib-compression which is relatively well-tested but still might dbda8d6ce9 2007-07-21 drh: cause corruption if used improperly. And there is the relatively dbda8d6ce9 2007-07-21 drh: new delta-encoding mechanism designed expressly for fossil. We want dbda8d6ce9 2007-07-21 drh: to make sure that bugs in these encoding mechanisms do not lead to dbda8d6ce9 2007-07-21 drh: loss of data. dbda8d6ce9 2007-07-21 drh: </p> dbda8d6ce9 2007-07-21 drh: dbda8d6ce9 2007-07-21 drh: <p> dbda8d6ce9 2007-07-21 drh: To increase our confidence that everything in the repository is dbda8d6ce9 2007-07-21 drh: recoverable, fossil makes sure it can extract an exact replicate dbda8d6ce9 2007-07-21 drh: of every content file that it changes just prior to transaction 6680679c2e 2007-11-24 drh: commit. So during the course of check-in (or other repository 6680679c2e 2007-11-24 drh: operation) many different files dbda8d6ce9 2007-07-21 drh: in the repository might be modified. Some files are simply dbda8d6ce9 2007-07-21 drh: compressed. Other files are delta encoded and then compressed. dbda8d6ce9 2007-07-21 drh: While all this is going on, fossil makes a record of every file b807acf62e 2007-07-24 drh: that is encoded and the SHA1 hash of the original content of that dbda8d6ce9 2007-07-21 drh: file. Then just before transaction commit, fossil re-extracts dbda8d6ce9 2007-07-21 drh: the original content of all files that were written, computes b807acf62e 2007-07-24 drh: the SHA1 checksum again, and verifies that the checksums match. dbda8d6ce9 2007-07-21 drh: If anything does not match up, an error dbda8d6ce9 2007-07-21 drh: message is printed and the transaction rolls back. dbda8d6ce9 2007-07-21 drh: </p> dbda8d6ce9 2007-07-21 drh: dbda8d6ce9 2007-07-21 drh: <p> dbda8d6ce9 2007-07-21 drh: So, in other words, fossil always checks to make sure it can 6680679c2e 2007-11-24 drh: re-extract a file before it commits a change to that file. dbda8d6ce9 2007-07-21 drh: Hence bugs in fossil are unlikely to corrupt the repository in dbda8d6ce9 2007-07-21 drh: a way that prevents us from extracting historical versions of dbda8d6ce9 2007-07-21 drh: files. dbda8d6ce9 2007-07-21 drh: </p> dbda8d6ce9 2007-07-21 drh: b807acf62e 2007-07-24 drh: <h2>Checksum Over All Files In A Baseline</h2> dbda8d6ce9 2007-07-21 drh: dbda8d6ce9 2007-07-21 drh: <p> 6680679c2e 2007-11-24 drh: Manifest artifacts that define a baseline have two fields (the 6680679c2e 2007-11-24 drh: R-card and Z-card) that record MD5 hashs of the manifest itself b807acf62e 2007-07-24 drh: and of all other files in the manifest. Prior to any check-in b807acf62e 2007-07-24 drh: commit, these checksums are verified to ensure that the baseline b807acf62e 2007-07-24 drh: checked in agrees exactly with what is on disk. Similarly, b807acf62e 2007-07-24 drh: the repository checksum is verified after a checkout to make b807acf62e 2007-07-24 drh: sure that the entire repository was checked out correctly. b807acf62e 2007-07-24 drh: Note that these added checks use a different hash (MD5 instead b807acf62e 2007-07-24 drh: of SHA1) in order to avoid common-mode failures in the hash b807acf62e 2007-07-24 drh: algorithm implementation. dbda8d6ce9 2007-07-21 drh: </p>
File Annotation