My Project
Chapter 1 – Overview of the Rx RPC system


Section 1.1: Introduction to Rx


The Rx package provides a high-performance, multi-threaded, and secure mechanism by which remote procedure calls (RPCs) may be performed between programs executing anywhere in a network of computers. The Rx protocol is adaptive, conforming itself to widely varying network communication media. It allows user applications to define and insert their own security modules, allowing them to execute the precise end-to-end authentication algorithms required to suit their needs and goals. Although pervasive throughout the AFS distributed file system, all of its agents, and many of its standard application programs, Rx is entirely separable from AFS and does not depend on any of its features. In fact, Rx can be used to build applications engaging in RPC-style communication under a variety of unix-style file systems. There are in-kernel and user-space implementations of the Rx facility, with both sharing the same interface.
This document provides a comprehensive and detailed treatment of the Rx RPC package.

Section 1.2: Basic Concepts

The Rx design operates on the set of basic concepts described in this section.

Section 1.2.1: Security

The Rx architecture provides for tight integration between the RPC mechanism and methods for making this communication medium secure. As elaborated in Section 5.3.1.3 and illustrated by the built-in rxkad security system described in Chapter 3, Rx defines the format for a generic security module, and then allows application programmers to define and activate instantiations of these modules. Rx itself knows nothing about the internal details of any particular security model, or the module-specific state it requires. It does, however, know when to call the generic security operations, and so can easily execute the security algorithm defined. Rx does maintain basic state per connection on behalf of any given security class.

Section 1.2.2: Services

An Rx-based server exports services, or specific RPC interfaces that accomplish certain tasks. Services are identified by (host-address, UDP-port, serviceID) triples. An Rx service is installed and initialized on a given host through the use of the rx NewService() routine (See Section 5.6.3). Incoming calls are stamped with the Rx service type, and must match an installed service to be accepted. Internally, Rx services also carry string names which identify them, which is useful for remote debugging and statistics-gathering programs. The use of a service ID allows a single server process to export multiple, independently-specified Rx RPC services.
Each Rx service contains one or more security classes, as implemented by individual security objects. These security objects implement end-to-end security protocols. Individual peer-to-peer connections established on behalf of an Rx service will select exactly one of the supported security objects to define the authentication procedures followed by all calls associated with the connection. Applications are not limited to using only the core set of built-in security objects offered by Rx. They are free to define their own security objects in order to execute the specific protocols they require.
It is possible to specify both the minimum and maximum number of lightweight processes available to handle simultaneous calls directed to an Rx service. In addition, certain procedures may be registered with the service and called at specific times in the course of handling an RPC request.

Section 1.2.3: Connections

An Rx connection represents an authenticated communication path, allowing a sequence of multiple asynchronous conversations (calls). Each connection is identified by a connection ID. The low-order bits of the connection ID are reserved so that they may be stamped with the index of a particular call channel. With up to RX MAXCALLS concurrent calls (set to 4 in this implementation), the bottom two bits are set aside for this purpose. The connection ID is not sufficient to uniquely identify an Rx connection by itself. Should a client crash and restart, it may reuse a connection ID, causing inconsistent results. Included with the connection ID is the epoch, or start time for the client side of the connection. After a crash, the next incarnation of the client will choose a different epoch value. This will differentiate the new incarnation from the orphaned connection record on the server side.
Each connection is associated with a parent service, which defines a set of supported security models. At creation time, an Rx connection selects the particular security protocol it will implement, referencing the associated service. The connection structure maintains state for each individual call simultaneously handled.

Section 1.2.4: Peers

For each connection, Rx maintains information describing the entity, or peer, on the other side of the wire. A peer is identified by a (host, UDP-port) pair, with an IP address used to identify the host. Included in the information kept on this remote communication endpoint are such network parameters as the maximum packet size supported by the host, current readings on round trip time and retransmission delays, and packet skew (see Section 1.2.7). There are also congestion control fields, including retransmission statistics and descriptions of the maximum number of packets that may be sent to the peer without pausing. Peer structures are shared between connections whenever possible, and, hence, are reference-counted. A peer object may be garbage-collected if it is not actively referenced by any connection structure and a sufficient period of time has lapsed since the reference count dropped to zero.

Section 1.2.5: Calls

An Rx call represents an individual RPC being executed on a given connection. As described above, each connection may have up to RX MAXCALLS calls active at any one instant. The information contained in each call structure is specific to the given call.
"Permanent" call state, such as the call number, is maintained in the connection structure itself.

Section 1.2.6: Quotas

Each attached server thread must be able to make progress to avoid system deadlock. The Rx facility ensures that it can always handle the arrival of the next unacknowledged data packet for an attached call with its system of packet quotas. A certain number of packets are reserved per server thread for this purpose, allowing the server threads to queue up an entire window full of data for an active call and still have packet buffers left over to be able to read its input without blocking.

Section 1.2.7: Packet Skew

If a packet is received n packets later than expected (based on packet serial numbers), then we define it to have a skew of n. The maximum skew values allow us to decide when a packet hasn't been received yet because it is out of order, as opposed to when it is likely to have been dropped.

Section 1.2.8: Multicasting

The rx multi.c module provides for multicast abilities, sending an RPC to several targets simultaneously. While true multicasting is not achieved, it is simulated by a rapid succession of packet transmissions and a collection algorithm for the replies. A client program, though, may be programmed as if multicasting were truly taking place. Thus, Rx is poised to take full advantage of a system supporting true multicasting with minimal disruption to the existing client code base.

Section 1.3: Scope

This paper is a member of a documentation suite providing specifications as to the operation and interfaces offered by the various AFS servers and agents. Rx is an integral part of the AFS environment, as it provides the high-performance, secure pathway by which these system components communicate across the network. Although AFS is dependent on Rx's services, the reverse is not true. Rx is a fully independent RPC package, standing on its own and usable in other environments.
The intent of this work is to provide readers with a sufficiently detailed description of Rx that they may proceed to write their own applications on top of it. In fact, code for a sample Rx server and client are provided.
One topic related to Rx will not be covered by this document, namely the Rxgen stub generator. Rather, rxgen is addressed in a separate document.

Section 1.4: Document Layout

After this introduction, Chapter 2 will introduce and describe various facilities and tools that support Rx. In particular, the threading and locking packages used by Rx will be examined, along with a set of timer and preemption tools. Chapter 3 proceeds to examine the details of one of the built-in security modules offered by Rx. Based on the Kerberos system developed by MIT's Project Athena, this rxkad module allows secure, ecrypted communication between the server and client ends of the RPC. Chapter 5 then provides the full Rx programming interface, and Chapter 6 illustrates the use of this programming interface by providing a fully-operational programming example employing Rx. This rxdemo suite is examined in detail, ranging all the way from a step-by-step analysis of the human-authored files, and the Rxgen-generated files upon which they are based, to the workings of the associated Makefile. Output from the example rxdemo server and client is also provided.

Section 1.5: Related Documents

Titles for the full suite of AFS specification documents are listed below. All of the servers and agents making up the AFS computing environment, whether running in the unix kernel or in user space, utilize an Rx RPC interface through which they export their services.
  • AFS-3 Programmer's Reference: Architectural Overview: This paper provides an architectual overview of the AFS distributed file system, describing the full set of servers and agents in a coherent way, illustrating their relationships to each other and examining their interactions.
  • AFS-3 Programmer's Reference: file Server/Cache Manager Interface: This document describes the workings and interfaces of the two primary AFS agents, the file Server and Cache Manager. The file Server provides a centralized disk repository for sets of files, regulating access to them. End users sitting on client machines rely on the Cache Manager agent, running in their kernel, to act as their agent in accessing the data stored on file Server machines, making those files appear as if they were really housed locally.
  • AFS-3 Programmer's Reference:Volume Server/Volume Location Server Interface: This document describes the services through which "containers" of related user data are located and managed.
  • AFS-3 Programmer's Reference: Protection Server Interface: This paper describes the server responsible for mapping printable user names to and from their internal AFS identifiers. The Protection Server also allows users to create, destroy, and manipulate "groups" of users, which are suitable for placement on access control lists (ACLs).
  • AFS-3 Programmer's Reference: BOS Server Interface: This paper explicates the "nanny" service which assists in the administrability of the AFS environment.
In addition to these papers, the AFS 3.1 product is delivered with its own user, system administrator, installation, and command reference documents.