Charles M. Kozierok The TCP-IP Guide

Подождите немного. Документ загружается.

However, caching is even more essential to HTTP than to most other protocols or technol-

ogies where it used. The reason is that Web documents tend to be structured so that a

request for one resource leads to a request for many others. Even if I load a number of

different documents, they may each refer to common elements that do not change

between user requests. Thus, caching can be of benefit in HTTP even if a user never asks

for the same document twice, or if a single document changes over time so that caching the

document itself would be of little value.

For example, suppose that each morning I load up CNN’s Web site to see what is going on

in the world. Obviously, the headlines will be different every day, so caching of the main

CCN.com Web home page won’t be of much value. However, many of the graphical

elements on the page (CNN’s logo, dividing bars, perhaps a “breaking news” graphic) will

be the same, every day, and these can be cached. Another example would be a set of

discussion forums on a Web site. As I load up different topics to read, each one is different,

but they have common elements (such as icons and other images) that would be wasteful

to have to retrieve over and over again.

Caching in HTTP yields two main benefits. The first is reduced bandwidth use, by elimi-

nating unneeded transfers of requests and responses. The second, equally important, is

faster response time for the user loading a resource. Consider that on many Web pages

today, the image files are much larger than the HTML page that references them. Caching

these graphics will allow the entire page to load far more quickly. Figure 319 illustrates how

caching reduces bandwidth and speeds up resource retrieval by “short-circuiting” the

request/response chain.

The obvious advantages of caching have made it a part of the Web since pretty much the

beginning. However, it was not until HTTP/1.1 that the importance of caching was really

recognized in the protocol itself, and many features added to support it. Where the HTTP/

1.0 standard makes passing mention of caching and some of the issues related to it, HTTP/

1.1 devotes 26 full pages to caching (over 20% of the main body of the document!) I

obviously cannot go into that level of detail here, but I will discuss HTTP caching in general

terms to give you a feel for the subject.

Figure 319: Impact of Caching on HTTP Request/Response Chain

This diagram illustrates the impact of caching on the request/response chain of Figure 316. In this example,

intermediary #2 is able to satisfy the client’s request from its cache. This “short-circuits” the communication

chain after two transfers, which means the client gets its resource more quickly, and the HTTP server is

spared the need to process the client’s request.

HTTP Client

Intermediary #1 HTTP ServerIntermediary #2

Request

Cached

Response

#1 #2

#1#2

Tradeoffs in Cache Location

HTTP caching can be implemented in a variety of places in the request/response chain.

The location of the cache must be chosen based on a fundamental trade-off that always

occurs in caching: proximity versus universality. Simply put, the closer the cache is to the

requestor of the information, the more savings that result when data is pulled from the

cache rather than being fetched from the source. However, the further the cache is from the

requestor (and thus closer to the source), the greater the number of devices that can benefit

from the cache. Let’s see how this manifests itself in the three classes of devices where

caches may be found.

Web Client Caching

The cache with which most Internet users are familiar is that found on the local client. It is

usually built into the Web browser software, and for this reason called a Web browser

cache. This cache stores recent documents and files accessed by a particular user, so that

they can be made quickly available if that user requests them again.

Since the cache is in the user’s own machine, a request for an item that the cache contains

is filled instantly, resulting in no network transaction and “instant gratification” for the user.

However, that user is the only one who can benefit from the cache, which is for this reason

sometimes called a private cache.

Intermediary (Proxy) Caching

Devices such as proxy servers that reside between Web clients and servers are also often

equipped with a cache. If a user wants a document not in his or her local client cache, the

intermediary may be able to provide it, as shown in Figure 319. This is not as efficient as

retrieving from the local cache, but far better than going back to the Web server. However,

the intermediary has the advantage that all devices using it can benefit from its cache,

which may be termed public or shared. This can be useful, because members of an organi-

zation often access similar documents.

For example, in an organization developing a hardware product to be used on Apple

computers, many different people might be accessing documents on Apple’s Web site. With

a shared cache, a request from user A would often result in items being cached that could

be used by user B.

Web Server Caching

Web servers themselves may also implement a cache. While it may seem a bit strange to

have a server maintain a cache of its own documents, this can be of benefit in some circum-

stances. A resource might require a significant amount of server resources to create; for

example, consider a Web page that is generated using a complex database query. If this

page is retrieved frequently by many clients, there can be a large benefit to creating it

periodically and caching it rather than generating it on the fly for each request.

Since the Web server is “farthest” from the users, this results in the least savings for a

cache hit, as the client request and server response must still travel the full path over the

network between client and server. However, this distance from the client also means that

all users of the server can benefit from it.

Key Concept: The most important feature that improves the efficiency of operation

of HTTP is caching—the storing of recently-requested resources in a temporary

area. If the same resource is then needed again a short time later, it can be retrieved

from the cache rather than requiring a fresh request to the server, resulting in a savings of

both time and bandwidth. Caching can be performed by Web clients, servers and intermedi-

aries. The closer the cache is to the user, the greater the efficiency benefits; the farther from

the user, the greater the number of users that can benefit from the cache.

Cache Control

The control of caching in clients and servers is accomplished in the same manner that most

other types of control are implemented in HTTP: through the use of special headers. The

most important of these is the Cache-Control general header, which has a number of direc-

tives that allow the operation of caches to be managed. There are other important caching-

related headers, including Expires and Vary. For a great deal of more specific information

related to HTTP caching, please consult RFC 2616, section 13.

Important Caching Issues

While the performance advantages of caching make it a "no-brainer", there is no denying

that caching has one significant drawback: it complicates the operation of HTTP in a

number of ways. Below is a description of some of the more important issues that HTTP/1.1

clients, servers and intermediaries need to deal with. As long as it is, this list is not

exhaustive, which gives you an idea of what is involved with caching in HTTP (as well as

why the standard needed 26 pages to cover the subject!)

Cache Aging and “Staleness”

When a user retrieves a document directly from its original source on the server, he or she

always is assured of getting the current version of that resource. When caching is used, that

is no longer the case. While many resources change infrequently, almost all will change at

some point in time. To take the example above of CNN’s Web site, it is probable that the

CNN logo won’t change very often, but it’s possible that the site may be redesigned period-

ically and the logo modified in some way, such as its size or color.

For this reason, a device cannot keep items in an HTTP cache indefinitely. The longer an

item is held in a cache—a process called aging—the more likely it is that the resource on

the server has changed and the cache has become “stale”. To make matters even more

complex, some resources become stale more quickly than others. As a result, much of the

caching-related functionality of HTTP is related to dealing with this matter of cache aging.

Cache Expiration and Validation

One of the ways that HTTP deals with the cache aging issue is through headers and logic

that allow caches, clients and servers to specify how long items should be cached before

they expire and must be refreshed. A process of validation is also defined that allows a

cache to check with a server at appropriate times to see if an item it has stored has been

modified.

Communication of Cache Status to the User

In most cases, the fact that an item has been retrieved from a cache rather than its source

is transparent to the user (though he or she may notice that the resource loads faster than

expected!) In certain cases, however, the user may need to be informed that a resource

came from a cache and not its original source. This is especially true when a cached item

may be stale, in which case the client should warn the user that the information might be out

of date.

Header Caching

Caching in HTTP is complicated by the fact that it can occur in multiple places, and some

HTTP headers are treated differently than others. HTTP headers are divided into two

general categories: end-to-end headers that are intended to accompany a resource all the

way to its ultimate recipient, and hop-by-hop headers that are used only for a particular

communication between two devices (be they client, server, or intermediary device). End-

to-end headers must be stored with a cached resource, where hop-by-hop headers only

have meaning for a particular transfer and are not cached.

Impact of Resource Updates

Some HTTP methods will automatically cause cache entries to become invalidated,

because they inherently cause a change to the underlying resource. For example, if a user

performs a PUT on a resource that was previously retrieved using GET, any cached copies

of that resource should be automatically invalidated to prevent the old version from being

supplied from the cache.

Privacy Concerns

In the case of shared caches (such as might exist in a proxy) there are potential privacy

issues to be concerned with. While in most cases having user A’s cached resource be

made available to user B is advantageous, we must be careful not to cache any items that

might be specific to user A, which user B should not see.

HTTP Proxy Servers and Proxying

In my overview of the HTTP operational model, I described how HTTP was designed to

support not just communication between a client and server, but also the inclusion of inter-

mediaries that may sit in the communication path between them. One of the most important

types of intermediary is a device called a proxy server, or more simply, just a proxy.

A proxy is a “middleman” that acts as both a client and a server. It accepts requests from a

client as if it were a server, then forwards them (possibly modifying them) to the real server,

which sees the proxy as a client. The server responds back to the proxy, which forwards the

reply back to the client. Proxies can be either transparent, meaning that they do not modify

requests and responses, or non-transparent, if they do so in order to provide a particular

service.

Note: The term “transparent proxy” can also be used to refer to a proxy that is

interposed automatically between a client and server—such as an organization-

wide firewall—as opposed to one that a user manually configures.

Benefits of Proxies

Since proxies have the ability to fully process all client requests and server responses, they

can be extremely useful in a number of circumstances. They can be used to implement or

enhance many important capabilities.

Security

Proxies can be set up to examine both outgoing requests and incoming responses, to

address various security concerns. For example, filtering can be set up to prevent users

from requesting “objectionable” content, or to screen out harmful replies such as files

containing hidden viruses.

Caching

As I mentioned in the previous topic, it can be advantageous to set up a “shared cache” that

is implemented on an intermediary, so resources requested by one client can be made

available to another. This can be done within a proxy server.

Performance

In some circumstances, the existence of a proxy server can significantly improve perfor-

mance, particularly by reducing latency. An excellent example of this is the proxy server that

is used by my own satellite Internet connection.

Due to the distance from the earth to the satellite, it takes over 500 milliseconds for a round

trip request/response cycle between my PC and an Internet server. If I load a Web page

containing images, I would have to wait 500+ milliseconds to get the HTML page, at which

point my browser would then have to generate new requests for each graphical element,

meaning another 500+ millisecond delay for each.

Instead, my ISP has a proxy server to which I send my requests for Web pages. It looks

through the HTML of these pages and automatically requests any elements such as

graphics for me. It then sends them straight back to my machine, cutting the time required

to display a full Web page drastically.

Key Concept: One of the most important types of intermediary devices in HTTP is a

proxy server, which acts as a middleman between the client and server, handling

both requests and responses. A proxy server may either transport messages

unchanged or may modify them to implement certain features and capabilities. Proxies are

often used to increase the security and/or performance of Web access.

Comparing Proxies and Caches

Proxying and caching are concepts that have a number of similarities, especially in terms of

the impact that they have on basic HTTP operation. Like caching, proxying has become

more important in recent years, and also complicates HTTP in a number of ways. The

HTTP/1.1 standard includes a number of specific features to support proxies, and also

addresses a number of concerns related to proxying.

The fact that both proxying and caching represent ways in which basic HTTP client/server

communication is changed, combined with the ability of proxies to perform caching,

sometimes leads people to think caches and proxies are the same, when they are not. A

proxy is actually a separate element that resides in the HTTP request/response chain,

where caches can be implemented within any device in that chain, including a proxy.

Another key way that caches and proxies differ is that caches are used “automatically”

when they are enabled, where proxies are not. To use a proxy, client software must be told

to use the proxy, and supplied with its IP address or domain name. The client then sends all

requests to the proxy rather than to the actual server that the user specifies.

Note: Most of my explanations here have focused on hardware proxy servers, but

proxies are also commonly implemented as software in a client device. A software

proxy performs the same tasks of processing requests and responses; it is much

cheaper to implement than a hardware proxy, but cannot be shared by many devices.

Important Proxying Issues

As I mentioned above, there are a few different issues that come into play when proxies are

used in HTTP. Here are some of the more important ones, and how HTTP deals with them.

Again, for much more information on proxying, please refer to your trusty copy of RFC

2616.

Capability Inconsistencies:

Issues arise when a client and server don’t use the same version of HTTP, or don’t support

the same features; for example, some servers may not support all of the methods that a

client may try to use. This is made all the more complex when a proxy enters the picture. Of

particular concern is the situation where a client and server may agree on a particular

feature that the proxy does not. The proxy must make sure that it passes along headers or

other elements that it may not comprehend.

Authentication Issues

The use of proxy servers often introduces new authentication or security requirements. In

addition to authenticating with an end server, the proxy may specify that the client needs to

present separate authentication credentials to it as well. This is done using the HTTP

Proxy-Authorization and Proxy-Authenticate headers; see the topic on HTTP security for

details.

Caching Interaction

Not only do both caching and proxying both complicate HTTP, they can complicate each

other. Many of the issues in handling caching, such as header caching, expiration and

validation, become more complex when proxies are involved. Some of the Cache-Control

general header directives are specific to proxying.

Another issue is that the use of proxying and caching together can lead to distortions in the

apparent number of times that a Web resource is accessed. This is important in situations

where Web pages are supported by advertising, based on the number of times the page is

accessed. Sometimes in this situation, special codes are placed in URLs called “cache

busters” are used to force pages not to be stored in shared caches.

Encodings

Content encodings are applied end-to-end and so should not be affected by proxies.

Transfer encoding is done hop-by-hop, so a proxy may use different encodings in handling

different transfers of a single request or response.

Tracing Proxy Handling

It is useful in some circumstances, especially when multiple proxies may be in the request/

response chain, to be able to trace what proxies have processed a particular message. To

this end, HTTP/1.1 requires that each proxy that handles a message identify itself in the Via

header.

HTTP Security and Privacy

There are a number of different protocols in this Guide where I address security consider-

ations. Usually, I start out by saying something to the effect that the protocol doesn’t include

much in the way of security, because when it was first developed, the Internet was small

and used by a tight-knit group, so security wasn’t a big concern. Today, the Internet is

globe-spanning and used by millions of strangers, making security a big deal indeed, blah

blah blah.

☺

Well, in the case of the World Wide Web this is true, but the issue is even more important

due to the significance of the changes in the content of what HTTP messages carry. HTTP

has become the vehicle for transporting any and every kind of information, including a large

amount of personal data. HTTP was initially designed to carry academic documents such

as memos about research projects, but today is more likely to carry someone’s mortgage

application, credit card details or medical details. Thus, not only does HTTP have the usual

security issues such as preventing unauthorized access, it needs to deal with privacy

concerns as well.

HTTP Authentication Methods

The main HTTP/1.1 standard, RFC 2616, also does not deal extensively with security

matters. These are addressed in detail instead in the companion document, RFC 2617,

which explains the two methods of HTTP authentication. Highly summarized, they are:

☯ Basic Authentication: This is a conventional user/password type of authentication.

When a client sends a request to a server that requires authentication to access a

resource, the server sends a response to the client’s initial request that contains a

WWW-Authenticate header. The client then sends a new request containing the

Authorization header, which carries a base64-encoded username and password

combination.

☯ Digest Authentication: Basic authentication is not considered strong security

because it sends credentials “in the clear”, which means that they can be intercepted.

Digest authentication uses the same headers as basic authentication, but employs

more sophisticated techniques, including encryption, that protect against a malicious

person “snooping” credentials information. Digest authentication is not considered as

strong as public key encryption, but is a lot better than basic authentication. It’s also a

darn sight more complicated. The full details of how it works are in RFC 2617.

Security and Privacy Concerns and Issues

Both RFC 2616 and 2617 also address some of the specific security concerns and threats

that can potentially affect HTTP clients and servers. These include actions such as

spoofing, counterfeit servers, replay attacks and much more. One concern addressed is the

potential for “man-in-the-middle” attacks, where an attacker interposes between the client

and server. Since proxies are inherently “men in the middle”, they represent a security

concern in this area. The same authentication methods used for servers can also be

applied to authentication with proxies. The Proxy-Authenticate and Proxy-Authorization

headers are used instead of WWW-Authenticate and Authorization.

The standards also discuss a number of privacy issues. Some that are worthy of note:

☯ Handling of Sensitive Information: The HTTP protocol can carry any type of infor-

mation, and it does not inherently protect the privacy of data in HTTP message

entities. To ensure the privacy of sensitive information, other techniques must be used

(which we will discuss shortly).

☯ Privacy of Information in URLs: One issue that sometimes arises in HTTP is that

poorly-designed Web sites may inadvertently encode private information into URLs.

These URLs may be recorded in Web logs, where they could fall into the hands of

people who could abuse them. An example of this would be a Web site that submits a

user login and password to a server by encoding them as parameters of a GET

request such as this:

GET http://www.somesite.com/login?name=xxx&password=yyy”

The POST method should be used instead for this sort of functionality, because it

transmits its data in the body of the message instead of putting it into the URL.

☯ Private Information in Accept Headers: While this may seem strange at first, it is

possible that private information about the user could be transmitted through the use

of certain “Accept-” headers used for content negotiation. For example, some users

might not want others to know what languages they speak, so they may be concerned

about who looks at the Accept-Language header.

☯ Information Obtained From the Referer Header: The Referer [sic] request header is

a double-edged sword; it can be very useful to those who operate Web sites because

it lets them see the sources of links to their resources. At the same time, it can be

abused by those who might employ it to study users’ Web access patterns. There are

also potential privacy issues that the HTTP standard raises. For example, a user might

not want the name of a private document that references a public Web page to be

transmitted in a Referer header.

Methods for Ensuring Privacy in HTTP

As mentioned earlier, HTTP does not include any mechanism to protect the privacy of

transmitted documents or messages. There are two different methods by which this is

normally accomplished. The simplest way is to encrypt the resource on the server and

supply valid decryption keys only to authorized users; even if the entire message is inter-

cepted, the entity itself will still be secured. The level of protection here depends on the

quality of the encryption.

Another more common method is to use an “add-on” protocol designed specifically to

ensure the privacy of HTTP transactions. The one often used today is called Secure

Sockets Layer (SSL). Servers employ SSL to protect sensitive resources, such as those

associated with financial transactions. They are accessed by using the URL scheme “https”

rather than “http” in a Web browser that supports the protocol. SSL was originally

developed by Netscape and is now widely used across the World Wide Web.

HTTP State Management Using "Cookies"

Even though the modern Hypertext Transfer Protocol has a lot of capabilities and features,

it is still, at its heart, a simple request/reply protocol. One of the unfortunate problems that

results from this is that HTTP is entirely stateless. This means that each time a server

receives a request from a client, it processes the request, sends a response, and then

forgets about the request. The next request from the client is treated as independent of any

previous ones.

Note: The persistent connection feature of HTTP/1.1 does not change the

stateless nature of the protocol. Even though multiple requests and responses can

be sent on a single TCP connection, they are still not treated as being related in

any way.

So why is this a problem? Isn’t this what we would expect of a protocol designed to allow a

client to quickly and efficiently retrieve resources from a server? Well, this is, yet again,

another place where HTTP’s behavior was well-suited to its original intended uses, but not

to how the Web is used today. Sure, if all we want to do is to say “hey server, please give

me that file over there”, then the server doesn’t have to care about whether or not it may

have previously provided that client with any other files in the past. This is how HTTP was

originally intended to be used.

Today, as most of us know, the Web is much more than a simple resource-retrieval protocol.

If you go to an online store, you want to be able to select a number of items to put into a

“shopping cart”, and have the store’s server remember them. You might also want to partic-

ipate in a discussion forum, which requires you to provide a user name and password in

order to post a message. Ideally, the server should let you log in once and then remember

who you are so you can post many messages without having to enter your login information

each and every time. (I have used forums where the latter is required—it gets old very

quickly, believe me.)

Cookies: Storing HTTP State Information

For these and other interactive applications, the stateless nature of HTTP is a serious

problem. The solution was the addition of a new technology, called state management, that

allows the state of a client session with a server to be maintained across a series of HTTP

transactions. Initially developed by Netscape, this technique was later made a formal

Internet standard in RFC 2109, later revised in RFC 2965 (HTTP State Management

Mechanism). Note that this feature is actually not part of HTTP; it is an optional element, but

one that has been implemented in pretty much all Web browsers due to its usefulness.

The idea behind state management is very simple. When a server implements a function

that requires state to be maintained across a set of transactions, it sends a small amount of

data to the Web client called a “cookie”. The cookie contains important information relevant

to the particular Web application, such as a customer name, items in a shopping cart, or a

user name and password. The client stores the information in the cookie, and then uses it in

subsequent requests to the server that set the cookie. The server can then update the

cookie based on the information in the new request and send it back to the client. In this

manner, state information can be maintained indefinitely, allowing the client and server to

have a “memory” that persists over a period of time.