Overview
| SHA1 Hash: | 16094f7ebc5a6bdf91c10e51252dcbe273151a00 |
|---|---|
| Date: | 2008-05-16 15:31:05 |
| User: | drh |
| Comment: | Resolve broken hyperlinks and other minor cleanup in the documentation. |
| Timelines: | ancestors | descendants | both | trunk |
| Other Links: | files | ZIP archive | manifest |
Tags And Properties
- branch=trunk inherited from [a28c83647d]
- sym-trunk inherited from [a28c83647d]
Changes
[hide diffs]Modified www/build.wiki from [d16ea958f5] to [61aa0809d1].
@@ -4,11 +4,11 @@ whole process is designed to be very easy.</p> <h2>0.0 Using A Precompiled Binary</h2> <p>You can skip steps 1.0 and 2.0 below by downloading -a <a href="http://www.fossil-scm.org/download.html">FIXME precompiled binary</a> +a <a href="http://www.fossil-scm.org/download.html">precompiled binary</a> appropriate for your platform. If you use a precompiled binary jump immediate to step 3.0.</p> <h2>1.0 Obtaining The Source Code</h2>
Modified www/concepts.wiki from [a61e6b72f9] to [076fde5d6b].
@@ -207,10 +207,11 @@ token after the name of the fossil executable, as shown above.</p> <h2>4.0 Workflow</h2> <img src="concept2.gif" align="right" hspace="10"> + <ol> <li><p> Establish a local repository using either the <b>new</b> command to start a new project, or the <b>clone</b> command to make a clone of a repository for an existing project.
Modified www/delta_encoder_algorithm.wiki from [83801262d2] to [a5b01a2840].
@@ -1,5 +1,6 @@ +<nowiki> <h1 align="center"> Fossil Delta Encoding Algorithm </h1> <p>A key component for the efficient storage of multiple revisions of @@ -7,21 +8,21 @@ store only the changes between revisions instead of the whole file.</p> <p>This document describes the encoding algorithm used by Fossil to generate deltas. It is targeted at developers working on either -<a href="index.html">fossil</a> itself, or on tools compatible with +<a href="index.wiki">fossil</a> itself, or on tools compatible with it. The exact format of the generated byte-sequences, while in general not necessary to understand encoder operation, can be found in the -companion specification titled "<a href="delta_format.html">Fossil +companion specification titled "<a href="delta_format.wiki">Fossil Delta Format</a>". </p> <p>The entire algorithm is inspired by <a href="http://samba.anu.edu.au/rsync/">rsync</a>.</p> -<a name="argresparam"><h2>1.0 Arguments, Results, and Parameters</h2> +<a name="argresparam"></a><h2>1.0 Arguments, Results, and Parameters</h2> <p>The encoder takes two byte-sequences as input, the "original", and the "target", and returns a single byte-sequence containing the "delta" which transforms the original into the target upon its application.</p> @@ -33,28 +34,28 @@ "sliding window" for the "rolling hash", in bytes. These two terms are explained in the next section. The value of this parameter has to be a power of two for the algorithm to work. For Fossil the value of this parameter is set to "16".</p> -<a name="operation"><h2>2.0 Operation</h2> +<a name="operation"></a><h2>2.0 Operation</h2> <p>The algorithm is split into three phases which generate -the <a href="delta_format.html#header">header</a>, -<a href="delta_format.html#slist">segment list</a>, -and <a href="delta_format.html#trailer">trailer</a> of the delta, per -its general <a href="delta_format.html#structure">structure</a>.</p> +the <a href="delta_format.wiki#header">header</a>, +<a href="delta_format.wiki#slist">segment list</a>, +and <a href="delta_format.wiki#trailer">trailer</a> of the delta, per +its general <a href="delta_format.wiki#structure">structure</a>.</p> <p>The two phases generating header and trailer are not covered here as their implementation trivially follows directly from the -specification of the <a href="delta_format.html">delta format</a>.</p> +specification of the <a href="delta_format.wiki">delta format</a>.</p> <p>This leaves the segment-list. Its generation is done in two phases, a pre-processing step operating on the "original" byte-sequence, followed by the processing of the "target" byte-sequence using the information gathered by the first step.</p> -<a name="preprocessing"><h3>2.1 Preprocessing the original</h3> +<a name="preprocessing"></a><h3>2.1 Preprocessing the original</h3> <p>A major part of the processing of the "target" is to find a range in the "original" which contains the same content as found at the current location in the "target".</p> @@ -83,11 +84,11 @@ <li>A hashtable is filled, mapping from the hashes of the chunks to the list of chunk locations having this hash. </li> </ol> -<a name="processing"><h3>2.1 Processing the target</h3> +<a name="processing"></a><h3>2.1 Processing the target</h3> <p>This, the main phase of the encoder, processes the target in a loop from beginning to end. The state of the encoder is captured by two locations, the "base" and the "slide". "base" points to the first byte of the target for which no delta output has been generated yet, and @@ -96,15 +97,15 @@ <p>Initially both "base" and "slide" point to the beginning of the "target". In each iteration of the loop the encoder decides whether to <ul> <li>emit a single instruction -to <a href="delta_format.html#copyrange">copy a range</a>, or +to <a href="delta_format.wiki#copyrange">copy a range</a>, or </li> <li>emit two instructions, first -to <a href="delta_format.html#insertlit">insert a literal</a>, then -to <a href="delta_format.html#copyrange">copy a range</a>, or +to <a href="delta_format.wiki#insertlit">insert a literal</a>, then +to <a href="delta_format.wiki#copyrange">copy a range</a>, or </li> <li>move the window forward one byte. </li> </ul> </p> @@ -167,29 +168,29 @@ <p>If the processing loop left bytes unencoded, i.e. "base" not exactly at the end of the "target", as is possible for both end conditions, then one last insert instruction is emitted to put these bytes into the delta.<p> -<a name="exceptions"><h2>3.0 Exceptions</h2> +<a name="exceptions"></a><h2>3.0 Exceptions</h2> <p>If the "original" is at most NHASH bytes long no compression of changes is possible, and the segment-list of the delta consists of a single literal which contains the entire "target".</p> <p>This is actually equivalent to the second end condition of the processing loop described in the previous section, just checked before actually entering the loop.</p> -<a name="rollhash"><h2>4.0 The rolling hash</h2> +<a name="rollhash"></a><h2>4.0 The rolling hash</h2> <p>The rolling hash described below and used to compute content signatures was chosen not only for good hashing properties, but also to enable the easy (incremental) recalculation of its value for a sliding window, i.e. where the oldest byte is removed from the window and a new byte is shifted in.<p> -<a name="rhdef"><h3>4.1 Definition</h3> +<a name="rhdef"></a><h3>4.1 Definition</h3> <p>Assuming an array Z of NHASH bytes (indexing starting at 0) the hash V is computed via</p> <p align=center><table><tr><td> @@ -199,11 +200,11 @@ </td></tr></table></p> where A and B are unsigned 16-bit integers (hence the <u>mod</u>), and V is a 32-bit unsigned integer with B as MSB, A as LSB. -<a name="rhincr"><h3>4.2 Incremental recalculation</h3> +<a name="rhincr"></a><h3>4.2 Incremental recalculation</h3> <p>Assuming an array Z of NHASH bytes (indexing starting at 0) with hash V (and components A and B), the dropped byte <img src="encode4.gif" align="center">, and the new byte <img src="encode5.gif" align="center"> , the new hash can
Modified www/delta_format.wiki from [03e4eaca4d] to [15113b9293].
@@ -1,30 +1,43 @@ +<nowiki> <h1 align="center"> Fossil Delta Format </h1> -<p>A key component for the efficient storage of multiple revisions of -a file in fossil repositories is the use of delta-compression, i.e. to -store only the changes between revisions instead of the whole -file.</p> - -<p>This document describes the format used to encode such changes, -also known as "delta". It is targeted at developers working on either -<a href="index.html">fossil</a> itself, or on tools compatible with -it.</p> - -<a name="structure"><h2>1.0 Structure</h2> +<p>Fossil achieves efficient storage and low-bandwidth synchronization +through the use of delta-compression. Instead of storing +or transmitting the complete content of an artifact, fossil stores or +transmits only the changes relative to a related artifact. +</p> + +<p>This document describes the delta-encoding format used by fossil. +The intended audience is developers working on either +<a href="index.wiki">fossil</a> itself, or on tools compatible with +fossil.</p> + +<p>Note that the delta-encoding is not a fundamental element of the +state of a fossil repository. A state of a fossil repository is +defined by the uncompressed and undeltaed content of all artifacts. +The fact the artifacts +are stored on disk using this delta-encoding format is merely an +optimization. One could, in theory, create an entirely new and +compatible implementation of fossil that used a different delta-encoding +or did no delta-encoding at all. However, experience has shown that +the delta-encoding described here is both efficient to compute and +results in very small deltas, so its continued use is recommended.</p> + +<a name="structure"></a><h2>1.0 Structure</h2> <img src="delta1.gif" align="left" hspace="10"> <p>A delta consists of three parts, a "header", a "trailer", and a "segment-list" between them.</p> <p>Both header and trailer provide information about the target helping the decoder, and the segment-list describes how the target can be constructed from the original.</p> -<a name="header"><h3>1.1 Header</h3> +<a name="header"></a><h3>1.1 Header</h3> <img src="delta6.gif" align="left" hspace="10"> <p>The header consists of a single number followed by a newline character (ASCII 0x0a). The number is the length of the target in bytes.</p> @@ -32,11 +45,11 @@ <p>This means that, given a delta, the decoder can compute the size of the target (and allocate any necessary memory based on that) by simply reading the first line of the delta and decoding the number found there. In other words, before it has to decode everything else.</p> -<a name="trailer"><h3>1.2 Trailer</h3> +<a name="trailer"></a><h3>1.2 Trailer</h3> <img src="delta5.gif" align="left" hspace="10"> <p>The trailer consists of a single number followed by a semicolon (ASCII 0x3b). This number is a checksum of the target and can be used by a decoder to verify that the delta applied correctly, reconstructing the @@ -49,11 +62,11 @@ <p>By putting this information at the end of the delta a decoder has it available immediately after the target has been reconstructed fully.</p> -<a name="slist"><h3>1.3 Segment-List</h3> +<a name="slist"></a><h3>1.3 Segment-List</h3> <img src="delta2.gif" align="left" hspace="10"> <p>The segment-list of a delta describes how to create the target from the original by a combination of inserting literal byte-sequences and copying ranges of bytes from the original. This is there the @@ -62,20 +75,20 @@ <p>The target is constructed from beginning to end, with the data generated by each instruction appended after the data of all previous instructions, with no gaps.</p> -<a name="insertlit"><h4>1.3.1 Insert Literal</h4> +<a name="insertlit"></a><h4>1.3.1 Insert Literal</h4> <p>A literal is specified by two elements, the size of the literal in bytes, and the bytes of the literal itself.</p> <img src="delta4.gif" align="left" hspace="10"> <p>The length is written first, followed by a colon character (ASCII 0x3a), followed by the bytes of the literal.</p> -<a name="copyrange"><h4>1.3.2 Copy Range</h4> +<a name="copyrange"></a><h4>1.3.2 Copy Range</h4> <p>A range to copy is specified by two numbers, the offset of the first byte in the original to copy, and the size of the range, in bytes. The size zero is special, its usage indicates that the range extends to the end of the original.</p> @@ -82,11 +95,11 @@ <img src="delta3.gif" align="left" hspace="10"> <p>The length is written first, followed by an "at" character (ASCII 0x40), then the offset, followed by a comma (ASCII 0x2c).</p> -<a name="intcoding"><h2>2.0 Encoding of integers</h2> +<a name="intcoding"></a><h2>2.0 Encoding of integers</h2> <p> The format currently handles only 32 bit integer numbers. They are written base-64 encoded, MSB first, and without leading "0"-characters, except if they are significant (i.e. 0 => "0"). @@ -95,13 +108,13 @@ <p> The base-64 coding is described in <a href="http://www.ietf.org/rfc/rfc3548.txt">RFC 3548</a>. </p> -<a name="examples"><h2>3.0 Examples</h2> - -<a name="examplesint"><h3>3.1 Integer encoding</h3> +<a name="examples"></a><h2>3.0 Examples</h2> + +<a name="examplesint"></a><h3>3.1 Integer encoding</h3> <table border=1> <tr> <th>Value</th> <th>Encoding</th> @@ -118,11 +131,11 @@ <td>-1101438770</td> <td>2zMM3E</td> </tr> </table> -<a name="examplesdelta"><h3>3.2 Delta encoding</h3> +<a name="examplesdelta"></a><h3>3.2 Delta encoding</h3> <p>An example of a delta using the specified encoding is:</p> <table border=1><tr><td><pre> 1Xb @@ -194,20 +207,18 @@ </pre></td></tr></table> -<a name="notes"><h2>Notes</h2> +<a name="notes"></a><h2>Notes</h2> <ul> <li>Pure text files generate a pure text delta. </li> <li>Binary files generate a delta that may contain some binary data. </li> -<li>Instead of putting special instructions for general compression -into the delta-format itself, specifically the segment-list, like -run-length encoding of literals, etc. it was considered to be much -more sensible to keep the various concern separate and use a general -compression library, like <a href="http://www.zlib.net">zlib</a>, to -compress the full delta after its generation. +<li>The delta encoding does not attempt to compress the content +It was considered to be much +more sensible to do compression using a separate general-purpose +compression library, like <a href="http://www.zlib.net">zlib</a>. </li> </ul>
Modified www/pop.wiki from [aaab800d38] to [f15d98d5e1].
@@ -65,10 +65,10 @@ <li><p>Every baseline has a special file at the top-level named "manifest" which is an index of all other files in that baseline. The manifest is automatically created and maintained by the system.</p></li> -<li><p>The <a href="fileformat.html">file formats</a> +<li><p>The <a href="fileformat.wiki">file formats</a> used by Fossil are all very simple so that with access to the original content files, one can easily reconstruct the content of a baseline without the need for any special tools or software.</p></li>
Modified www/quickstart.wiki from [4381e8a48d] to [5e0abce227].
@@ -1,14 +1,16 @@
+<nowiki>
<h1 align="center">Fossil Quick Start</h1>
<p>This is a guide to get you started using fossil quickly
and painlessly.</p>
<h2>Installing</h2><blockquote>
<p>Fossil is a single self-contained C program. You need to
- either download a <a href="download.wiki">FIXME precompiled binary</a>
+ either download a
+ <a href="http://www.fossil-scm.org/download.html">precompiled binary</a>
or <a href="build.wiki">build it yourself</a> from sources.
Install fossil by putting the fossil binary
someplace on your PATH environment variable.</p>
</blockquote>
@@ -270,6 +272,5 @@
<p>Explore and have fun!</p>
</blockquote>
-
Modified www/selfcheck.wiki from [e5d3578c4b] to [61460bdeef].
@@ -1,5 +1,6 @@ +<nowiki> <h1 align="center"> Fossil Repository Integrity Self-Checks </h1> <p> @@ -9,11 +10,12 @@ lose your files. This note describes the defensive measures that fossil uses to help prevent file loss due to bugs. </p> <p><i>Follow-up as of 2007-11-24:</i> -Fossil has been hosting itself and several other projects for +<i>Reiterated on 2008-05-16:</i> +Fossil has been hosting itself and many other projects for months now. Many bugs have been encountered. But, thanks in large part to the defensive measures described here, no data has been lost. The integrity checks are doing their job well.</p> <h2>Atomic Check-ins With Rollback</h2> @@ -20,11 +22,11 @@ <p> The fossil repository is an <a href="http://www.sqlite.org/">SQLite version 3</a> database file. SQLite is very mature and stable and has been in wide-spread use for many -years, so we have little worries that it might cause repository +years, so we are confident it will not cause repository corruption. SQLite databases do not corrupt even if a program or system crash or power failure occurs in the middle of the update. If some kind of crash does occur in the middle of a change, then all the changes are rolled back the next time that the database is accessed.