d87ca60c58 2008-05-15 stephan: <h1 align="center">The Fossil Sync Protocol</h1> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>Fossil supports commands <b>push</b>, <b>pull</b>, and <b>sync</b> d87ca60c58 2008-05-15 stephan: for transferring information from one repository to another. The d87ca60c58 2008-05-15 stephan: command is run on the client repository. A URL for the server repository d87ca60c58 2008-05-15 stephan: is specified as part of the command. This document describes what happens d87ca60c58 2008-05-15 stephan: behind the scenes in order to synchronize the information on the two d87ca60c58 2008-05-15 stephan: repositories.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <h2>1.0 Transport</h2> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>All communication between client and server is via HTTP requests. d87ca60c58 2008-05-15 stephan: The server is listening for incoming HTTP requests. The client d87ca60c58 2008-05-15 stephan: issues one or more HTTP requests and receives replies for each d87ca60c58 2008-05-15 stephan: request.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>The server might be running as an independent server d87ca60c58 2008-05-15 stephan: using the <b>server</b> command, or it might be launched from d87ca60c58 2008-05-15 stephan: inetd or xinetd using the <b>http</b> command. Or the server might d87ca60c58 2008-05-15 stephan: be launched from CGI. The details of how the server is configured d87ca60c58 2008-05-15 stephan: to "listen" for incoming HTTP requests is immaterial. The important d87ca60c58 2008-05-15 stephan: point is that the server is listening for requests and the client d87ca60c58 2008-05-15 stephan: is the issuer of the requests.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>A single push, pull, or sync might involve multiple HTTP requests. d87ca60c58 2008-05-15 stephan: The client maintains state between all requests. But on the server d87ca60c58 2008-05-15 stephan: side, each request is independent. The server does not preserve d87ca60c58 2008-05-15 stephan: any information about the client from one request to the next.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <h3>1.1 Server Identification</h3> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>The server is identified by a URL argument that accompanies the d87ca60c58 2008-05-15 stephan: push, pull, or sync command on the client. (As a convenience to d87ca60c58 2008-05-15 stephan: users, the URL can be omitted on the client command and the same URL d87ca60c58 2008-05-15 stephan: from the most recent push, pull, or sync will be reused. This saves d87ca60c58 2008-05-15 stephan: typing in the common case where the client does multiple syncs to d87ca60c58 2008-05-15 stephan: the same server.)</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>The client modifies the URL by appending the method name "<b>/xfer</b>" d87ca60c58 2008-05-15 stephan: to the end. For example, if the URL specified on the client command d87ca60c58 2008-05-15 stephan: line is</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <blockquote> d87ca60c58 2008-05-15 stephan: http://fossil-scm.hwaci.com/fossil d87ca60c58 2008-05-15 stephan: </blockquote> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>Then the URL that is really used to do the synchronization will d87ca60c58 2008-05-15 stephan: be:</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <blockquote> d87ca60c58 2008-05-15 stephan: http://fossil-scm.hwaci.com/fossil/xfer d87ca60c58 2008-05-15 stephan: </blockquote> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <h3>1.2 HTTP Request Format</h3> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>The client always sends a POST request to the server. The d87ca60c58 2008-05-15 stephan: general format of the POST request is as follows:</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <blockquote><pre> d87ca60c58 2008-05-15 stephan: POST /fossil/xfer HTTP/1.0 d87ca60c58 2008-05-15 stephan: Host: fossil-scm.hwaci.com:80 d87ca60c58 2008-05-15 stephan: Content-Type: application/x-fossil d87ca60c58 2008-05-15 stephan: Content-Length: 4216 d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <i>content...</i> d87ca60c58 2008-05-15 stephan: </pre></blockquote> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>In the example above, the pathname given after the POST keyword d87ca60c58 2008-05-15 stephan: on the first line is a copy of the URL pathname. The Host: parameter d87ca60c58 2008-05-15 stephan: is also taken from the URL. The content type is always either d87ca60c58 2008-05-15 stephan: "application/x-fossil" or "application/x-fossil-debug". The "x-fossil" d87ca60c58 2008-05-15 stephan: content type is the default. The only difference is that "x-fossil" d87ca60c58 2008-05-15 stephan: content is compressed using zlib whereas "x-fossil-debug" is sent d87ca60c58 2008-05-15 stephan: uncompressed.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>A typical reply from the server might look something like this:</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <blockquote><pre> d87ca60c58 2008-05-15 stephan: HTTP/1.0 200 OK d87ca60c58 2008-05-15 stephan: Date: Mon, 10 Sep 2007 12:21:01 GMT d87ca60c58 2008-05-15 stephan: Connection: close d87ca60c58 2008-05-15 stephan: Cache-control: private d87ca60c58 2008-05-15 stephan: Content-Type: application/x-fossil; charset=US-ASCII d87ca60c58 2008-05-15 stephan: Content-Length: 265 d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <i>content...</i> d87ca60c58 2008-05-15 stephan: </pre></blockquote> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>The content type of the reply is always the same as the content type d87ca60c58 2008-05-15 stephan: of the request.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <h2>2.0 Fossil Synchronization Content</h2> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>A synchronization request between a client and server consists of d87ca60c58 2008-05-15 stephan: one or more HTTP requests as described in the previous section. This d87ca60c58 2008-05-15 stephan: section details the "x-fossil" content type.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <h3>2.1 Line-oriented Format</h3> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>The x-fossil content type consists of zero or more "cards". Cards d87ca60c58 2008-05-15 stephan: are separate by the newline character ("\n"). Leading and trailing d87ca60c58 2008-05-15 stephan: whitespace on a card is ignored. Blank cards are ignored.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>Each card is divided into zero or more space separated tokens. d87ca60c58 2008-05-15 stephan: The first token on each card is the operator. Subsequent tokens d87ca60c58 2008-05-15 stephan: are arguments. The set of operators understood by servers is slightly d87ca60c58 2008-05-15 stephan: different from the operators understood by clients, though the two d87ca60c58 2008-05-15 stephan: are very similar.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <h3>2.2 Login Cards</h3> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>Every message from client to server begins with one or more login d87ca60c58 2008-05-15 stephan: cards. Each login card has the following format:</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <blockquote> d87ca60c58 2008-05-15 stephan: <b>login</b> <i>userid nonce signature</i> d87ca60c58 2008-05-15 stephan: </blockquote> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>The userid is the name of the user that is requesting service d87ca60c58 2008-05-15 stephan: from the server. The nonce is the SHA1 hash of the remainder of d87ca60c58 2008-05-15 stephan: the message - all text that follows the newline character that d87ca60c58 2008-05-15 stephan: terminates the login card. The signature is the SHA1 hash of d87ca60c58 2008-05-15 stephan: the concatenation of the nonce and the users password.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>For each login card, the server looks up the user and verifies d87ca60c58 2008-05-15 stephan: that the nonce matches the SHA1 hash of the remainder of the d87ca60c58 2008-05-15 stephan: message. It then checks the signature hash to make sure the d87ca60c58 2008-05-15 stephan: signature matches. If everything d87ca60c58 2008-05-15 stephan: checks out, then the client is granted all privileges of the d87ca60c58 2008-05-15 stephan: specified user.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>Privileges are cumulative. There can be multiple successful d87ca60c58 2008-05-15 stephan: login cards. The session privileges are the bit-wise OR of the d87ca60c58 2008-05-15 stephan: privileges of each individual login.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <h3>2.3 File Cards</h3> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>Repository content records or files are transferred using d87ca60c58 2008-05-15 stephan: a "file" card. File cards come in two different formats depending d87ca60c58 2008-05-15 stephan: on whether the file is sent directly or as a delta from some d87ca60c58 2008-05-15 stephan: other file.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <blockquote> d87ca60c58 2008-05-15 stephan: <b>file</b> <i>uuid size</i> <b>\n</b> <i>content</i><br> d87ca60c58 2008-05-15 stephan: <b>file</b> <i>uuid delta-uuid size</i> <b>\n</b> <i>content</i> d87ca60c58 2008-05-15 stephan: </blockquote> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>File cards are different from all other cards in that they d87ca60c58 2008-05-15 stephan: followed by in-line "payload" data. The content of the file d87ca60c58 2008-05-15 stephan: or the file delta consists of the first <i>size</i> bytes of the d87ca60c58 2008-05-15 stephan: x-fossil content that immediately follow the newline that d87ca60c58 2008-05-15 stephan: terminates the file card. No other cards have this characteristic. d87ca60c58 2008-05-15 stephan: </p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>The first argument of a file card is the UUID of the file that d87ca60c58 2008-05-15 stephan: is being transferred. The UUID is the lower-case hexadecimal d87ca60c58 2008-05-15 stephan: representation of the SHA1 hash of the entire file content. d87ca60c58 2008-05-15 stephan: The last argument of the file card is the number of bytes of d87ca60c58 2008-05-15 stephan: payload that immediately follow the file card. If the file d87ca60c58 2008-05-15 stephan: card has only two arguments, that means the payload is the d87ca60c58 2008-05-15 stephan: complete content of the file. If the file card has three d87ca60c58 2008-05-15 stephan: arguments, then the payload is a delta and second argument is d87ca60c58 2008-05-15 stephan: the UUID of another file that is the source of the delta.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>File cards are sent in both directions: client to server and d87ca60c58 2008-05-15 stephan: server to client. A delta might be sent before the source of d87ca60c58 2008-05-15 stephan: the delta, so both client and server should remember deltas d87ca60c58 2008-05-15 stephan: and be able to apply them when their source arrives.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <h3>2.4 Push and Pull Cards</h3> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>Among of the first cards in a client-to-server message are d87ca60c58 2008-05-15 stephan: the push and pull cards. The push card tell the server that d87ca60c58 2008-05-15 stephan: the client is pushing content. The pull card tell the server d87ca60c58 2008-05-15 stephan: that the client wants to pull content. In the event of a sync, d87ca60c58 2008-05-15 stephan: both cards are sent. The format is as follows:</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <blockquote> d87ca60c58 2008-05-15 stephan: <b>push</b> <i>servercode projectcode</i><br> d87ca60c58 2008-05-15 stephan: <b>pull</b> <i>servercode projectcode</i> d87ca60c58 2008-05-15 stephan: </blockquote> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>The <i>servercode</i> argument is the repository ID for the d87ca60c58 2008-05-15 stephan: client. The server will only allow the transaction to proceed d87ca60c58 2008-05-15 stephan: if the servercode is different from its own servercode. This d87ca60c58 2008-05-15 stephan: prevents a sync-loop. The <i>projectcode</i> is the identifier d87ca60c58 2008-05-15 stephan: of the software project that the client repository contains. d87ca60c58 2008-05-15 stephan: The projectcode for the client and server must match in order d87ca60c58 2008-05-15 stephan: for the transaction to proceed.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>The server will also send a push card back to the client d87ca60c58 2008-05-15 stephan: during a clone. This is how the client determines what project d87ca60c58 2008-05-15 stephan: code to put in the new repository it is constructing.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <h3>2.5 Clone Cards</h3> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>A clone card works like a pull card in that it is sent from d87ca60c58 2008-05-15 stephan: client to server in order to tell the server that the client d87ca60c58 2008-05-15 stephan: wants to pull content. But unlike the pull card, the clone d87ca60c58 2008-05-15 stephan: card has no arguments.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <blockquote> d87ca60c58 2008-05-15 stephan: <b>clone</b> d87ca60c58 2008-05-15 stephan: </blockquote> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>In response to a clone message, the server also sends the client d87ca60c58 2008-05-15 stephan: a push message so that the client can discover the projectcode for d87ca60c58 2008-05-15 stephan: this project.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <h3>2.6 Igot Cards</h3> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>An igot card can be sent from either client to server or from d87ca60c58 2008-05-15 stephan: server to client in order to indicate that the sender holds a copy d87ca60c58 2008-05-15 stephan: of a particular file. The format is:</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <blockquote> d87ca60c58 2008-05-15 stephan: <b>igot</b> <i>uuid</i> d87ca60c58 2008-05-15 stephan: </blockquote> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>The argument of the igot card is the UUID of the file that d87ca60c58 2008-05-15 stephan: the sender possesses. d87ca60c58 2008-05-15 stephan: The receiver of an igot card will typically check to see if d87ca60c58 2008-05-15 stephan: it also holds the same file and if not it will request the file d87ca60c58 2008-05-15 stephan: using a gimme card in either the reply or in the next message.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <h3>2.7 Gimme Cards</h3> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>A gimme card is sent from either client to server or from server d87ca60c58 2008-05-15 stephan: to client. The gimme card asks the receiver to send a particular d87ca60c58 2008-05-15 stephan: file back to the sender. The format of a gimme card is this:</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <blockquote> d87ca60c58 2008-05-15 stephan: <b>gimme</b> <i>uuid</i> d87ca60c58 2008-05-15 stephan: </blockquote> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>The argument to the gimme card is the UUID of the file that d87ca60c58 2008-05-15 stephan: the sender wants. The receiver will typically respond to a d87ca60c58 2008-05-15 stephan: gimme card by sending a file card in its reply or in the next d87ca60c58 2008-05-15 stephan: message.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <h3>2.8 Cookie Cards</h3> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>A cookie card can be used by a server to record a small amount d87ca60c58 2008-05-15 stephan: of state information on a client. The server sends a cookie to the d87ca60c58 2008-05-15 stephan: client. The client sends the same cookie back to the server on d87ca60c58 2008-05-15 stephan: its next request. The cookie card has a single argument which d87ca60c58 2008-05-15 stephan: is its payload.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <blockquote> d87ca60c58 2008-05-15 stephan: <b>cookie</b> <i>payload</i> d87ca60c58 2008-05-15 stephan: </blockquote> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>The client is not required to return the cookie to the server on d87ca60c58 2008-05-15 stephan: its next request. Or the client might send a cookie from a different d87ca60c58 2008-05-15 stephan: server on the next request. So the server must not depend on the d87ca60c58 2008-05-15 stephan: cookie and the server must structure the cookie payload in such d87ca60c58 2008-05-15 stephan: a way that it can tell if the cookie it sees is its own cookie or d87ca60c58 2008-05-15 stephan: a cookie from another server. (Typically the server will embed d87ca60c58 2008-05-15 stephan: its servercode as part of the cookie.)</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <h3>2.9 Error Cards</h3> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>If the server discovers anything wrong with a request, it generates d87ca60c58 2008-05-15 stephan: an error card in its reply. When the client sees the error card, d87ca60c58 2008-05-15 stephan: it displays an error message to the user and aborts the sync d87ca60c58 2008-05-15 stephan: operation. An error card looks like this:</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <blockquote> d87ca60c58 2008-05-15 stephan: <b>error</b> <i>error-message</i> d87ca60c58 2008-05-15 stephan: </blockquote> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>The error message is English text that is encoded in order to d87ca60c58 2008-05-15 stephan: be a single token. d87ca60c58 2008-05-15 stephan: A space (ASCII 0x20) is represented as "\s" (ASCII 0x5C, 0x73). A d87ca60c58 2008-05-15 stephan: newline (ASCII 0x0a) is "\n" (ASCII 0x6C, x6E). A backslash d87ca60c58 2008-05-15 stephan: (ASCII 0x5C) is represented as two backslashes "\\". Apart from d87ca60c58 2008-05-15 stephan: space and newline, no other whitespace characters nor any d87ca60c58 2008-05-15 stephan: unprintable characters are allowed in d87ca60c58 2008-05-15 stephan: the error message.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <h3>2.10 Unknown Cards</h3> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>If either the client or the server sees a card that is not d87ca60c58 2008-05-15 stephan: described above, then it generates an error and aborts.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <h2>3.0 Phantoms And Clusters</h2> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>When a repository knows that a file exists and knows the UUID of d87ca60c58 2008-05-15 stephan: that file, but it does not know the file content, then it stores that d87ca60c58 2008-05-15 stephan: file as a "phantom". A repository will typically create a phantom when d87ca60c58 2008-05-15 stephan: it receives an igot card for a file that it does not hold or when it d87ca60c58 2008-05-15 stephan: receives a file card that references a delta source that it does not d87ca60c58 2008-05-15 stephan: hold. When a server is generating its reply or when a client is d87ca60c58 2008-05-15 stephan: generating a new request, it will usually send gimme cards for every d87ca60c58 2008-05-15 stephan: phantom that it holds.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>A cluster is a special file that tells of the existence of other d87ca60c58 2008-05-15 stephan: files. Any file in the repository that follows the syntactic rules d87ca60c58 2008-05-15 stephan: of a cluster is considered a cluster.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>A cluster is a line oriented file. Each line of a cluster d87ca60c58 2008-05-15 stephan: is a card. The cards are separated by the newline ("\n") character. d87ca60c58 2008-05-15 stephan: Each card consists of a single character card type, a space, and a d87ca60c58 2008-05-15 stephan: single argument. No extra whitespace and no trailing or leading d87ca60c58 2008-05-15 stephan: whitespace is allowed. All cards in the cluster must occur in d87ca60c58 2008-05-15 stephan: strict lexicographical order.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>A cluster consists of one or more "M" cards followed by a single d87ca60c58 2008-05-15 stephan: "Z" card. Each M card holds an argument which is a UUID for a file d87ca60c58 2008-05-15 stephan: in the repository. The Z card has a single argument which is the d87ca60c58 2008-05-15 stephan: lower-case hexadecimal representation of the MD5 checksum of all d87ca60c58 2008-05-15 stephan: preceding M cards up to and included the newline character that d87ca60c58 2008-05-15 stephan: occurred just before the Z that starts the Z card.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>Any file that does not match the specifications of a cluster d87ca60c58 2008-05-15 stephan: exactly is not a cluster. There must be no extra whitespace in d87ca60c58 2008-05-15 stephan: the file. There must be one or more M cards. There must be a d87ca60c58 2008-05-15 stephan: single Z card with a correct MD5 checksum. And all cards must d87ca60c58 2008-05-15 stephan: be in strict lexicographical order.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <h3>3.1 The Unclustered Table</h3> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>Every repository maintains a table named "<b>unclustered</b>" d87ca60c58 2008-05-15 stephan: which records the identity of every file and phantom it holds that is not d87ca60c58 2008-05-15 stephan: mentioned in a cluster. The entries in the unclustered table can d87ca60c58 2008-05-15 stephan: be thought of as leaves on a tree of files. Some of the unclustered d87ca60c58 2008-05-15 stephan: files will be clusters. Those clusters may contain other clusters, d87ca60c58 2008-05-15 stephan: which might contain still more clusters, and so forth. Beginning d87ca60c58 2008-05-15 stephan: with the files in the unclustered table, one can follow the chain d87ca60c58 2008-05-15 stephan: of clusters to find every file in the repository.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <h2>4.0 Synchronization Strategies</h2> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <h3>4.1 Pull</h3> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>A typical pull operation proceeds as shown below. Details d87ca60c58 2008-05-15 stephan: of the actual implementation may very slightly but the gist of d87ca60c58 2008-05-15 stephan: a pull is captured in the following steps:</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <ol> d87ca60c58 2008-05-15 stephan: <li>The client sends login and pull cards. d87ca60c58 2008-05-15 stephan: <li>The client sends a cookie card if it has previously received a cookie. d87ca60c58 2008-05-15 stephan: <li>The client sends gimme cards for every phantom that it holds. d87ca60c58 2008-05-15 stephan: <hr> d87ca60c58 2008-05-15 stephan: <li>The server checks the login password and rejects the session if d87ca60c58 2008-05-15 stephan: the user does not have permission to pull. d87ca60c58 2008-05-15 stephan: <li>If the number entries in the unclustered table on the server is d87ca60c58 2008-05-15 stephan: greater than 100, then the server constructs a new cluster file to d87ca60c58 2008-05-15 stephan: cover all those unclustered entries. d87ca60c58 2008-05-15 stephan: <li>The server sends file cards for every gimme card it received d87ca60c58 2008-05-15 stephan: from the client. d87ca60c58 2008-05-15 stephan: <li>The server sends ihave cards for every file in its unclustered d87ca60c58 2008-05-15 stephan: table that is not a phantom. d87ca60c58 2008-05-15 stephan: <hr> d87ca60c58 2008-05-15 stephan: <li>The client adds the content of file cards to its repository. d87ca60c58 2008-05-15 stephan: <li>The client creates a phantom for every ihave card in the server reply d87ca60c58 2008-05-15 stephan: that mentions a file that the client does not possess. d87ca60c58 2008-05-15 stephan: <li>The client creates a phantom for the delta source of file cards when d87ca60c58 2008-05-15 stephan: the delta source is a file that the client does not possess. d87ca60c58 2008-05-15 stephan: </ol> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>These ten steps represent a single HTTP round-trip request. d87ca60c58 2008-05-15 stephan: The first three steps are the processing that occurs on the client d87ca60c58 2008-05-15 stephan: to generate the request. The middle four steps are processing d87ca60c58 2008-05-15 stephan: that occurs on the server to interpret the request and generate a d87ca60c58 2008-05-15 stephan: reply. And the last three steps are the processing that the d87ca60c58 2008-05-15 stephan: client does to interpret the reply.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>During a pull, the client will keep sending HTTP requests d87ca60c58 2008-05-15 stephan: until it holds all files that exist on the server.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>Note that the server tries d87ca60c58 2008-05-15 stephan: to limit the size of its reply message to something reasonable d87ca60c58 2008-05-15 stephan: (usually about 1MB) so that it might stop sending file cards as d87ca60c58 2008-05-15 stephan: described in step (6) if the reply becomes too large.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>Step (5) is the only way in which new clusters can be created. d87ca60c58 2008-05-15 stephan: By only creating clusters on the server, we hope to minimize the d87ca60c58 2008-05-15 stephan: amount of overlap between clusters in the common configuration where d87ca60c58 2008-05-15 stephan: there is a single server and many clients. The same synchronization d87ca60c58 2008-05-15 stephan: protocol will continue to work even if there are multiple servers d87ca60c58 2008-05-15 stephan: or if servers and clients sometimes change roles. The only negative d87ca60c58 2008-05-15 stephan: effects of these unusual arrangements is that more than the minimum d87ca60c58 2008-05-15 stephan: number of clusters might be generated.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <h3>4.2 Push</h3> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>A typical push operation proceeds roughly as shown below. As d87ca60c58 2008-05-15 stephan: with a pull, the actual implementation may vary slightly.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <ol> d87ca60c58 2008-05-15 stephan: <li>The client sends login and push cards. d87ca60c58 2008-05-15 stephan: <li>The client sends file cards for any files that it holds that have d87ca60c58 2008-05-15 stephan: never before been pushed - files that come from local check-ins. d87ca60c58 2008-05-15 stephan: <li>If this is the second or later cycle in a push, then the d87ca60c58 2008-05-15 stephan: client sends file cards for any gimme cards that the server sent d87ca60c58 2008-05-15 stephan: in the previous cycle. d87ca60c58 2008-05-15 stephan: <li>The client sends igot cards for every file in its unclustered table d87ca60c58 2008-05-15 stephan: that is not a phantom. d87ca60c58 2008-05-15 stephan: <hr> d87ca60c58 2008-05-15 stephan: <li>The server checks the login and push cards and issues an error if d87ca60c58 2008-05-15 stephan: anything is amiss. d87ca60c58 2008-05-15 stephan: <li>The server accepts file cards from the client and adds those files d87ca60c58 2008-05-15 stephan: to its repository. d87ca60c58 2008-05-15 stephan: <li>The server creates phantoms for igot cards that mention files it d87ca60c58 2008-05-15 stephan: does not possess or for file cards that mention delta source files that d87ca60c58 2008-05-15 stephan: it does not possess. d87ca60c58 2008-05-15 stephan: <li>The server issues gimme cards for all phantoms. d87ca60c58 2008-05-15 stephan: <hr> d87ca60c58 2008-05-15 stephan: <li>The client remembers the gimme cards from the server so that it d87ca60c58 2008-05-15 stephan: can generate file cards in reply on the next cycle. d87ca60c58 2008-05-15 stephan: </ol> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>As with a pull, the steps of a push operation repeat until the d87ca60c58 2008-05-15 stephan: server knows all files that exist on the client. Also, as with d87ca60c58 2008-05-15 stephan: pull, the client attempts to keep the size of the request from d87ca60c58 2008-05-15 stephan: growing too large by suppressing file cards once the d87ca60c58 2008-05-15 stephan: size of the request reaches 1MB.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <h3>4.3 Sync</h3> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>A sync is just a pull and a push that happen at the same time. d87ca60c58 2008-05-15 stephan: The first three steps of a pull are combined with the first five steps d87ca60c58 2008-05-15 stephan: of a push. Steps (4) through (7) of a pull are combined with steps d87ca60c58 2008-05-15 stephan: (5) through (8) of a push. And steps (8) through (10) of a pull d87ca60c58 2008-05-15 stephan: are combined with step (9) of a push.</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <h2>5.0 Summary</h2> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <p>Here are the key points of the synchronization protocol:</p> d87ca60c58 2008-05-15 stephan: d87ca60c58 2008-05-15 stephan: <ol> d87ca60c58 2008-05-15 stephan: <li>The client sends one or more PUSH HTTP requests to the server. d87ca60c58 2008-05-15 stephan: The request and reply content type is "application/x-fossil". d87ca60c58 2008-05-15 stephan: <li>HTTP request content is compressed using zlib. d87ca60c58 2008-05-15 stephan: <li>The content of request and reply consists of cards with one d87ca60c58 2008-05-15 stephan: card per line. d87ca60c58 2008-05-15 stephan: <li>Card formats are: d87ca60c58 2008-05-15 stephan: <ul> d87ca60c58 2008-05-15 stephan: <li> <b>login</b> <i>userid nonce signature</i> d87ca60c58 2008-05-15 stephan: <li> <b>push</b> <i>servercode projectcode</i> d87ca60c58 2008-05-15 stephan: <li> <b>pull</b> <i>servercode projectcode</i> d87ca60c58 2008-05-15 stephan: <li> <b>clone</b> d87ca60c58 2008-05-15 stephan: <li> <b>file</b> <i>uuid size</i> <b>\n</b> <i>content</i> d87ca60c58 2008-05-15 stephan: <li> <b>file</b> <i>uuid delta-uuid size</i> <b>\n</b> <i>content</i> d87ca60c58 2008-05-15 stephan: <li> <b>igot</b> <i>uuid</i> d87ca60c58 2008-05-15 stephan: <li> <b>gimme</b> <i>uuid</i> d87ca60c58 2008-05-15 stephan: <li> <b>cookie</b> <i>cookie-text</i> d87ca60c58 2008-05-15 stephan: <li> <b>error</b> <i>error-message</i> d87ca60c58 2008-05-15 stephan: </ul> d87ca60c58 2008-05-15 stephan: <li>Phantoms are files that a repository knows exist but does not possess. d87ca60c58 2008-05-15 stephan: <li>Clusters are files that contain the UUIDs of other files. d87ca60c58 2008-05-15 stephan: <li>Clusters are created automatically on the server during a pull. d87ca60c58 2008-05-15 stephan: <li>Repositories keep track of all files that are not named in any d87ca60c58 2008-05-15 stephan: cluster and send igot messages for those files. d87ca60c58 2008-05-15 stephan: <li>Repositories keep track of all the phantoms they hold and send d87ca60c58 2008-05-15 stephan: gimme messages for those files. d87ca60c58 2008-05-15 stephan: </ol>
File Annotation