Grapevine: An Exercise in Distributed Computing

Arthurs
➗ Xerox Palo Alto Research Center
Summary by: Zuyu Zhang

One-line Summary

Overview/Main Points

Distributed

Involve the use of multiple computers communicating through an internet consisting of a collection of Ethernet local networks, gateways, and long distance links (typically telephone lines at data rates of 9.6 to 56 Kbps).

Replicated

Services (e.g. mail service, file storage, and printing) are provided equally well by any of several distinct computers.

inbox

the buffer for a recipient's messages

distribution list

named sets of recipients

client

The programs that manipulate and interpretate the contents of the messages that Grapevine transports.

• Grapevine System Configuration
  • Grapevine's clients run on various workstations and server computers in a connected manner.
  • Grapevine service runs on dedicated server computers.
  • A client accesses Grapevine service via a middleware called GrapevineUser running on the client's computer.
  • GrapevineUser package has two roles: implement the internet protocols for communicating with particular Grapevine servers; and performs the resource location for a particular function by obtaining the membership of the service group and then the connect site of each member from the registration data base, giving the data distribution and server availability situation of the moment. The package is not trusted by the rest of Grapevine; an incorrect package cannot affect the use of Grapevine by other clients.
• The Message Service
  • Buffered Message Delivery Mechanism for a large, dispersed computer mail system.
  • Character
    • Independence: Grapevine makes no assumptions about the content of the message being transmitted. Integrity / Availability of Grapevine service does not depend on correctness of the clients.
    • Fault-tolerate: A single point failure in Grapevine servers does not affect the availability of the whole services to any clients.
    • Performance: The recipient of the message need not be available at the time the sender wishes to transmit the message. The typical interval from sending a message to its arrival in a recipient's inbox should be a few minutes at most. The typical interactive delay perceived by a client program when delivering or receiving a message should be a few seconds at most.
    • Reliability: If a message is accepted for delivery, then it will either be made available to its intended recipients or returned with an indication of any errors, including user errors (such as invalid names), client errors (such as protocol violations), server problems (such as disk space congestion or hardware failures), or communication difficulties (such as internet link severance or gateway crashes).
    • Administrability: An administrator could implement its decentralized decisions (such as user access control, storage and computing resource management) by interacting directly with Grapevine.
    • Security: Perform authentication of senders and recipients, the content integrity check for message delivery (secure from eavesdropping or content alteration), and control on use and modification of its registration data bases.
  • Communication Protocols
    • UDP: small unreliable individual packets delivery without acknoledgement.
    • TCP: established byte streams for reliable, acknoledged transmission of unlimited amounts of data
  • Functions
    • Accept message: [sender, password, recipients, message-body] → ok. The recipients are individual and group RNames. Any message server is willing to accept any message for delivery (providing a replicated mail submission service).
    • Message polling: [individual] → {empty, nonempty}. Each message server will accept message polling and retrieve requests for inboxes on that server.
    • Retrieve messages: [name, password] → sequence of messages → ok.
• The Registration Service
  • Tasks
    • Naming people, machines, and services
    • Decentralized Access Control Service (Authenticating people and machines includes admission of users, control of the names by which they are known, and their inclusion in distribution list)
    • Resource / Service location on the internet
  • A registration data base that maps names to information about the users, machines, services, distribution lists, and access control lists that those name signify.
    • RName: the name of an entry in the registration data base using a two-level hierarchy (f.r where r is a registry name and f is a name within that registry)
    • An individual RName, represented human users and servers, contains an authenticator (a password), a list of inbox sites (primary inbox, secondary inbox), and a connect site.
    • A group RName, used primarily as distribution lists (may be recursive) and partially as a service, contains a set of RNames of other data base entries. For a service, its members are the RNames of the servers offering the service.
    • Registries, a partitioned naming scheme for RNames, serving as the basis for dividing the administrative responsibility, and for distributing the data base at the grain of a registry among the Grapevine computers.
  • Functions
    • Authenticate: [individual, password] → {authentic, bogus}.
    • Membership: {name, group} → {in, out}. The client is interpretating the group as an access control list. Two forms of the membership function: one indicates direct membership in the named group; the other indicates membership in its closure.
    • Resource location: [group] → members. [individual] → connect site. [individual] → ordered list of inbox sites. An individual may have inboxes on the several message servers (replicating the delivery path for the individual).
    • Registration data base update and inquiry: update and merge (on a complete list basis) primitive operations in atomic manner. Any server that contains the data for a registry can accept a change to that registry, and takes the responsibility for propagating the change to the other relevant servers.
  • Implementation
    • GV (for Grapevine) registry, where each registration server is an individual whose connect site is the internet address and the authenticator and inbox site list are used, is replicated in every registration server to control the distribution and replication of the registration data base and allow clients to locate appropriate registration servers for particular RNames. The groups of the GV registry, r.gv, whose members are the RNames of the registration servers that contain the registry, are the registries themselves; gv.gv includes the RNames of all registration servers.
    • MS (for message servers) registry includes each message server as an individual whose connect site is the internet address of the message server. A group in the MS registry, Maildrop.ms, contains as members some subset (usually, but not necessarily, all) of the message server RNames to find a message server that accepts a message for delivery.
    • Two mechanisms to determine the internet address of a registration server to get started
      • A primitive name lookup server on each local network, connected by a broadcast protocol, contains an infrequently updated data base that maps character strings to internet addresses, for example, GrapevineRServer to the internet addresses of some subset of the registration servers in the internet.
      • A distinctive packet broadcasted on the directly connected local network is used for faster respond and in case the name lookup server is down, and any accessible registration server will respond.
    • Eventually consistency guarantee for updating the registration data base.
    • Each entry in the registration data base contains a collection of lists and a version timestamp that functions as an entry's version number. Each list is represented as two sorted sublists of items, called the active sublist and the deleted sublist. An item consists of the string that exists in either sublist and the timestamp that is a unique identitier whose most significant bits are the server's time when the item placed in the list and least significant bits the server's internet address. The item timestamps in the active sublist are used to imply the preference order for the inbox site list in an individual's entry; order items are preferred. Thus, deleting then adding a site name moves it to the end of the preference ordering. A client uses the cached version timestamp to compare with that in the data base. The total ordering is kept by the registration server that has larger timestamp, thus there is no guarantee for the correctness of concurrent updates.
    • The registration server that accepts an update to its registry (f.reg) changes its local copy of the entry, and then propagates the change to other servers using Grapevine's delivery service to send a change message to the members of the group (reg.gv) that represents the registry, other than itself, as the service gives a guarantee of delivery and provides buffering when other servers are temporarily inaccessible. The change message contains the name of the affected entry and the entire new value for the entry.
    • Registration servers poll their inboxes for new message every 30 seconds, and upon received a change message, uses the merge operation to combine the entry from the change message with its own copy.
    • The client has to be prepared to cope with transient inconsistencies introduced by conflicting updates to the registration data base from multiple administrators.
    • Deteled items are record for further investigations by human administrators. The Grapevine servers purge all deleted items older than the upper bound of 14 days upon the clock asynchrony among the registration servers, on message delivery delay, and on administrative hindsight.
    • A permanent inconsistency that results from destroyed change messages due to software bugs or equipment failures may be resolved by periodically comparing the data base for a registry on each registration server with another in all disconnected regions, and using the merge operation to solve any discovered inconsistencies. An alternative in an anti-entropy manner is by detecting during the merge operation whether the local copy of the entry contained information that was missing from the incoming copy. (But the change message only contains the name of the affected entry and the entire new value for the entry, instead of the complete list.) But this mechanism is too haphazard to be useful; errors were not corrected until the next change to an entry.
    • For creating a new entry of each registry in the data base, there is usually some human-level centralization of name creation.
  • Two benefits of using Grapevine as an authentication service and an administration service that groups individuals by organziation, interest, and function. Simplify administrative operations as users have the same name and password everywhere. The same group structure can be used for many different purposes, such as both for an access control list for file sharing and a distribution list for message delivery.
• Examples of How Grapevine works
  1. After the sender P.Q prepares a message to the receiver X.Y, the client porgram would call the delivery function of the GrapevineUser package on its workstation.
  2. GrapevineUser would contact a registration server such as A and use the Grapevine resource location functions to locate any message server such as B; it would then submit the message to B.
  3. For each recipient, B would use the resource location facilities, and suitable registration servers (such as A) to determine that recipient's best inbox site as message server C, then forward the message to C. If the message had more recipients, B might consult other registration servers and forward the message to multiple message servers. If some of recipients were distribution lists, B would use the registration servers to obtain the members of the appropriate groups.
  4. C would buffer this message locally in the inbox for X.Y.
  5. When X.Y wishes to read the mail, the client program calls the retrieval function of the GrapevineUser package.
  6. GrapevineUser uses a registration server D that contains the Y registry to locate inbox sites for X.Y, then connects to each of these inbox sites to retrieve messages.
  7. C uses a registration server to authenticate X.Y, then allows such retrieval.
  8. If X.Y wanted to access a file on the file server E through FTP program, the file server might authenticate his identity and check access control lists by communicating with a registration server such as A.

Relevance

Flaws