Captain's log, stardate 11-20-94.
We have been in orbit of Bristola system for 2 weeks. Acting under orders
from W3 headquarters, an away team from engineering has been working on
the planets surface attempting to renovate the native hypertext system.
Their report now follows.
--- Here's some quick notes on how HTTP-NG is coming along - We've now got a simple proof-of-concept implementation up and running, using a minimalist set of requests and responses, and the results have been quite encouraging. Even though the network connection between Bristol and Civillization was extremely congested, initial timings seemed to show that HTTP-NG was capable of using the entire path bandwidth available when multiple requests were used. Initial measurements seemed to show that for typical inlined-objects, HTTP 1.0 would only use around a 10th of the available bandwidth, and that parallel fetches degraded badly under congested conditions.
Architecture.
HTTP-NG is designed to be a replacement for HTTP 1.0 offering much higher performance and adding some extra features needed for use in commercial applications. The protocol is designed to make it easy to implement the basic functionality needed for a simple browser whilst making the addition of more powerful features such as security and authentication much simpler than it is for HTTP.
HTTP-NG uses a different basic model to HTTP. HTTP sets up a new connection for every request, which causes a lot of severe performance problems both in the time taken for each transaction, and the load placed on both networks and servers.
To avoid these problems, HTTP-NG allows many different requests to be sent over a single connection. These requests are asynchronous - there's no need for the client to wait for a response before sending out a different request. The server can also respond to requests in any order it sees fit - it can even interweave the data from multiple objects, allowing several images to be transferred in "parallel".
To make these multiple data streams easy to work with, HTTP-NG sends all it's messages and data using a "session layer". This divides the connection up into lots of different channels. HTTP-NG sends all control messages (GET requests, meta-information etc) over a control channel. Each object is returned over in its own channel.
This separation of data and control information also makes redirection much more powerful - for example, if the object is a video the server can return the meta-information over the same connection, together with a URL pointing to a dedicated video transfer protocol that will fetch the data for the relevant object. This becomes much more important when working with multimedia aware networking technologies, such as ATM or RSVP.
---- Implementation experience.
There are several obvious ways of implementing HTTP-NG. The first approach is to use a simple synchronous RPC model. The client establishes a connection to the server, then sends out a request. The client then waits until it has received all the data corresponding to the request, then returns. This approach is very simple to implement, and corresponds quite closely to current practice in older HTTP clients. As a simple optimization, the implementation could send a number of requests before waiting for all their responses.
The second approach is to use an event based model. A simple dispatcher can be wrapped around the connection. Each request then adds a callback that should be called whenever the response to the request comes in. This model works well with event based browsers such as Arena and Netscape; it is also quite simple to implement.
The final approach is to use a multi-threaded implementation; each request and channel gets allocated its own thread, allowing multiple requests to be processed in parallel. This model works supremely well for servers, especially those serving as proxies. The proxy server can keep a cache of open connections to other proxies or to popular sites. Since each connection resides within a single process, requests that can't be satisfied from the local cache can easily be switched to the appropriate connection. This approach also allows efficient gatewaying of HTTP 1.0 requests (see next section).
Transition ----------
The best transition strategy for moving from HTTP 1.0 to HTTP-NG is through the use of intermediate proxy servers. This allows the existing base of servers and clients to continues operating as they are now, whilst still taking advantage of much of the performance enhancements in the new protocol.
The reason that this works is that most of the performance problems in HTTP 1.0 are caused by delays in the network. If proxy servers are placed close to older clients and servers, then these delays become significant. For example, if two servers are placed at either end of a transatlantic link, communicating with each other using HTTP-NG, but accepting and sending requests to and from other systems using HTTP 1.0, all the HTTP 1.0 delays would all occur within a continent, rather than spanning the intercontinental links. Further, a cacheing server can interpret HTML documents and pre-fetch any inlined objects before an HTTP client requests them.
------- =_aaaaaaaaaa0 Content-Type: text/plain; charset="us-ascii" Content-Description: Architectural overview (text dump from framemaker)
FHTTP - Architectural Overview
Simon Spero, UNC Sunsite/EIT ([email protected])
About FHTTP
FHTTP is an enhanced replacement for HTTP/1.0. It is designed to correct the known performance problems in previous versions of HTTP, and to provide extra support for commercial transactions, including enhanced security and support for on-line payment.
About this document
This document provides an architectural overview of the new protocol. It gives an overall view of how the protocol works, and explains how various operations and features interact with each other and the network. This document is intended for a general audience, and is not a technical specification. This is not an official W3O document.
About this series
This document is part of a series describing FHTTP.
1. Architectural Overview
2. The Session Control Protocol
3. Introduction to ASN.1 and the Packed Encoding Rules
4. FHTTP Specification
5. Security Profiles
6. Payment Profiles
Why do we need a new protocol?
HTTP is the fastest growing protocol on the internet. It is simple to implement, and thousands of people use it every day to browse through gigabytes of on-line hypertext. If HTTP is working so well, why replace it? The driving forces behind this change are the twin needs of performance and commerce.
The Need For Performance
Part of the need for a new protocol is caused by the very fact of HTTP's success. The original protocol was designed to be a simple way of transferring a file between two machines. This led to several very serious performance problems, adding extra delays to the time taken to fetch pages, and preventing browsers from making efficient use of the network.
Commercial Applications
Although many publishers and companies are now starting to explore the possibilities opened up by the World Wide Web, the protocols currently in use do not support several very important features needed for electronic commerce. Currently there is no reliable way to find out the identity of someone trying to access a document. There is also no way to convey information about charging and prices; nor is there a way of dealing with on-line payment.
Can these changes be made by just tweaking HTTP 1.0?
The major problem in making just minor changes HTTP 1.0 is that the modifications which are needed change the fundamental model on which the protocol is based. Each extension adds more and more com- plexity to the protocol; as more and more enhancements are made, the problem becomes much worse. Since HTTP 1.0 is a simple protocol, it is much simpler to recreate the existing functionality as part of a new protocol than to attempt to kludge the new functionality into a protocol not designed to handle such changes.
User Requirements
Any new protocol needs to meet the needs to three different groups. Individual users want to be able to browse the web without being forced to wait for pages to be delivered. Information providers need to be able to support large numbers of users, and to restrict access to authorized users, and to get money from their paying customers. Software developers need a system which is easy to implement, but which can be optimized and enhanced to differentiate between products.
TABLE 1. User requirements
Simplicity
FHTTP must allow simple implementations to be implemented simply without penalising more optimized systems. The protocol should be designed to work well for the commonest cases
Performance:
FHTTP should allow objects to be transferred over wide area networks efficiently.
Asynchronicity:
FHTTP should allow a client or server to initiate a new request with- out waiting for previous requests to complete. It should be possible to transfer multiple objects in parallel over a single connection.
Security:
FHTTP should support the transfer of encrypted objects. The protocol should not impose a single security policy or mechanism.
Authentication:
FHTTP should support mutual authentication between all parties involved in a transaction. It should be possible to relay authentication information through multiple untrusted intermediaries. It should also be possible to use multiple authentication contexts over a single con- nection.
Charging:
FHTTP should provide support for on-line payment schemes such as First Virtual and DigiCash. The protocol should not impose a single payment policy or mechanism.
Intermediate Servers:
FHTTP should provide support for server to server interaction to allow for the maintenance of caches and mirrors, and to allow intermediate servers to relay usage information to originating sites.
Mandatory display:
FHTTP should support the mandatory display of information relating to an object, such as licensing information, copyright, authorship.
Logging Information
FHTTP should support the transfer of logging information between intermediate servers and the original source of a resource. Users should be able to specify restrictions on the use of logging information, and to discontinue transactions if such restrictions cannot be satisfied.
Network requirements - Transports of Delight
FHTTP is transport-layer independent, and can arrange for data transfer using different transports. However, since all uses of FHTTP in the current internet will be over TCP, FHTTP must work well over TCP.
TCP and the Internet
Many performance problems in HTTP are a result of not taking Transport layer interactions into account. Table\x112 lists some important points that need to be taken into account when designing a proto- col.
TABLE 2. Important features of TCP
Connection setup costs
When TCP sets up a connection, it sends connection request to the server, and waits for the connection to be accepted or rejected. This adds a delay of one Round Trip Time.
Slow start limits transfer rates during start up.
When a connection is first started, TCP initially sends only a small amount of data. The amount of data that can be transmit- ted before the sender must wait for a reply is increased until a steady state is reached- if congestion occurs and packets are lost, the server slows down the send rate until a new stable point is reached.
Slow-start particularly affects the first request on a new connec- tion. If the request won't fit into a single segment, the client must wait an extra Round Trip before it can finish sending the request.
Congestion information is not shared between connections
Although slow-start is used to converge upon the correct trans- mission rate for the path between two computers, this informa- tion is not shared between different connections to the same host. Thus is there are several connections running between a pair of hosts, if the path becomes congested, the connections will interfere with each other, leading to poor throughput.
Special purpose transport layers.
As resource reservation and guaranteed bandwidth become more commonplace, and as ATM moves into more general deployment, it becomes important to be able to make use of special purpose transport layers which are optimised for certain types of media- for example, a multicast video transmission, or a ATM telephone channel. FHTTP should be able to refer a user on to another service for actual data trans- fer, whilst still being able to handle the relevant control information.
Protocol Model
HTTP 1.0 works by creating a new transport connection for each request. The client sends a request over the connection; the server replies over the same connection, sending information about the response, followed if possible by the requested data.
FHTTP uses a different model. Instead of creating a separate connection for each request, FHTTP cre- ates a single connection which can be used for many different requests. The connection is divided up into a number of virtual sessions. One of these sessions is used to carry control information - requests from the client, and meta-data from the server. The other channels are used to carry the requested objects.
Requests and Responses
Each request and response is sent as an FHTTP message. Some parts of a message are the same in all cases. For example, each message may need to be signed by the sender to make sure that the request is genuine. Other parts of the message are different for each operation. A GET request will specify a list of the objects to fetch- the response will contain information describing the retrieved objects.
All of this information must be encoded in some way before it can be sent over the network. HTTP 1.0 uses a text based syntax to encode requests, and a modified form of MIME to handle responses. This tex- tural representation is easy for humans to understand, but quickly becomes extremely complicated when used to handle requests with complicated structure.
To avoid this complexity, FHTTP uses a different way of describing and encoding the request message. The scheme used is a simplified form of ASN.1 and PER (Abstract Syntax Notation, and Packed Encod- ing Rules). This scheme allows efficient, compact parsers to be generated automatically, whilst remain- ing simple enough to allow hand-crafted parsers to be built easily.
FHTTP messages can be sent at any time. In the typical case, the client sends a request message to the server, and gets back one or more responses in return. The client doesn't have to wait for a response to one request before sending out another. Since returned objects can be interleaved on a stream, this allows clients such as Mosaic Communications NetScapeTM to request objects as soon as they discover they are needed, and allows intermediate servers to handle requests from different clients to the same server in a fair and efficient manner.
Although the usual pattern is for the client to issue requests to the server, there are some cases where this pattern needs to be reversed. One example for this is the need to warn browsers if a requested action would result in a hefty charge. In this case, the server needs to be able to check to see if the client still wishes to perform that action, and to discuss with the client how sir wishes to pay.
Negotiations - The Art of The Deal
Because FHTTP doesn't restrict the types of objects which can be requested, and does not impose a single security or payment policy, there needs to be some way of negotiating which types and mecha- nisms a client and server can support.
HTTP 1.0 allowed clients to propose a list of types which they were able to accept; unfortunately the mechanism used had several problems. Because the client had to sent a list of all possible types with each request, requests typically contained over 1K of type information. This caused serious performance problem, and required a lot of effort to process correctly. As a result of this complexity, very few servers properly supported this type negotiation.
FHTTP tries to simplify negotiation by adding an extra mechanism to support the most common cases without affecting more complicated situations. This new form of negotiation is based on the observation that the vast majority of all WWW traffic involves the exchange of just a few well-known object types. FHTTP defines a small list of these well-known types, and allows sets of these types to be encoded in a short bitmap. A text only browser would sent a bitmap indicating support for just HTML and plain text. A graphical brower would claim support for several graphics and sound formats. A server or browser which only supports these types need only support this simple form of negotiation.
To indicate support for other types. FHTTP also allows the client and server to add extra items to the bit-sets. The proposer send a message containing the option being proposed (for example, text/ms- word), together with a numeric code which will be used in later messages to refer to the proposed value.
This kind of indirect reference is useful, because it allows the proposed types and values to become more structured without affecting performance. This extra freedom can be used to support parameterized types; for example, if a client is running on a 4-bit display, a server can avoid generating a 24-bit deep image.
The same method is used to indicate supported security schemes, authentication information, and pay- ment mechanisms.
Security
Because there are so many different security schemes and policies, FHTTP provides a general security framework into which the various security components can be fitted. This results in several architectural decisions. One such is the absence of any special support for certificate exchange; this exchange is han- dled using the Fetch request (which is after all the most fundamental operation in FHTTP).
The FHTTP message wrapper has fields which can carry arbitrary authentication and security informa- tion. This allows each message to be individually authenticated.
FHTTP also allows an intermediate server to relay authorisation and verification to and from another server on behalf of one of its clients. This feature allows untrusted proxies to cache encrypted data, and to relay the information needed to decrypt this data to the end user without having to be able to decode the document itself.
Charging and Payment
FHTTP allows a server to initiate a payment request in response to an action of the client. This allows the server to request specific authorisation from the client for actions for which charges will be incurred- the challenge includes cost information, and a list of acceptable payment mechanisms; the response may include payment using a mutually acceptable method of on-line payment.
The Client may also use the negotiation mechanisms to authorise a series of payments without requir- ing explicit approval for each item. The server can also provide pricing information as part of an objects meta-data.
Mandatory Display
The client may negotiate a set of fields such as licensing information and authorisation which it is pre- pared to automatically display to the user along with the actual requested information. This allows infor- mation providers whose database licences require such information to be displayed at the start of each session to be made available over the web.
Copyright Control
Each transferred object may have associated copying information associated with it. This information can be used to assist honest implementations from accidentally violating the terms of a licence agree- ment.
Transition from HTTP 1.0
FHTTP uses two strategies for transition from HTTP 1.0 to the new protocol. One strategy uses a dual stack approach - the other is based on the use of proxy servers.
To support the dual stack approach, FHTTP has been designed to co-exist with FHTTP on the same port. Each FHTTP connect begins with a syntactically legal HTTP 1.0 request, using a non-existent method. If an FHTTP client tries to talk to an old HTTP server, the HTTP server will send back a `Method not supported' message; the client can then cache this information, and repeat the request using the old protocol. If an old server tries to access a new server, the server can either handle the transaction using HTTP 1.0, or return an error message.
The dual stack approach doesn't offer much improvement for clients that don't support FHTTP directly. Proxy servers allow old clients to take advantage of some of the newer performance features. The proxy server can pre-fetch inlined images using the new protocol, so that they are ready and waiting when the client requests them; since the proxy server will usually be on the same network as the client, this reduces latency delays to a minimum.
A proxy server can also multiplex several HTTP 1.0 requests over the same connection. This allows browsers like NetScape to request multiple concurrent images without causing congestion problems.
Historical Notes
This document was produced with assistance of David St. John Raggett (Hewlett Packard), Phillip Hallam-Baker (CERN), Allan Schiffman (EIT), and Tim Berners-Lee (MIT).
FHTTP has been inspired and influenced by several earlier protocols. The roll of honor includes:
HTTP (HyperText Transfer Protocol)
WAIS (Wide Area Information Servers)
NISO Z39.50 (Search and Retrieve)
IMAP (Internet Mail Access Protocol)
TCP (Transport Control Protocol)
T/TCP (Transaction TCP)
Bibliography
HTTP 1.0
HTTP-PROB
------- =_aaaaaaaaaa0 Content-Type: text/plain; charset="us-ascii" Content-Description: Session control protocol
Session Control Protocol (SCP) ------------------------------
Several heavily used Internet applications such as FTP, GOPHER, and HTTP use a protocol model in which every transaction requires a separate TCP connection. Since clients normally issue multiple requests to the same server, this model is quite inefficient, as it incurs all the connection start up costs for every single request.
SCP is a simple protocol which lets a server and client have multiple conversations over a single TCP connection. The protocol is designed to be simple to implement, and is modelled after TCP.
Services. ---------
SCP's main service is dialogue control. This service allows either end of the connection to establish a virtual session over a single transport connection. SCP also allows a sender to indicate message boundaries, and allows a reciever to reject an incoming session.
Design goals. -------------
o Unconfirmed service without negotiation.
SCP allows data to be sent with the session establishment; the recepient does not confirm successful connection establishment, but may reject unsuccessful attempts. This simplifies the design of the protocol, and removes the latency required for a confirmed operation.
o Low overhead
SCP has a fixed overhead of 8 bytes per segment. This overhead is half the size of an IPNG address, and is only incurred once per segment, instead of once per packet.
o Simple design
The session protocol should be simple enough to implement for a single application.
Protocol Description
Header Format: -------------- 32 24 16 8 0 | | | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0|0|0|0|S|F|R|P| SESSION ID | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | SEGMENT LENGTH | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ . . . DATA . . . +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ S = SYN F = FIN R = RST P = PUSH Protocol Operation: -------------------
Session ID allocation.
Each session is allocated a session identifier. Session Identifiers below 1024 are reserved. Session IDs allocated by clients are even; those allocated by servers, odd.
Session establishment.
A session is established by setting the SYN bit in the first message sent on that channel.
Graceful release.
A session is ended by sending a message with the FIN bit set. Each end of a connection may be closed independently.
Disgraceful release.
A session may be terminated by sending a message with the RST bit set. All pending data for that session should be discarded
Message boundaries.
A message boundary is marked by sending a message with the PUSH bit set. The boundary is set at the final octet in this message, including that octet.
------- =_aaaaaaaaaa0--