Charles M. Kozierok The TCP-IP Guide

Подождите немного. Документ загружается.

Key Concept: The Network News Transfer Protocol (NNTP) is the protocol used to

implement message communication in modern Usenet. It is used for two primary

purposes: to propagate messages between NNTP servers, and to permit NNTP

clients to post and read articles. It is a standalone protocol, but shares many characteristics

with e-mail’s Simple Mail Transfer Protocol (SMTP).

NNTP Inter-Server Communication Process: News Article Propagation

The Network News Transfer Protocol (NNTP) is used for all of the transfer steps in the

modern Usenet communication process. However, NNTP is most often associated with the

process of Usenet article propagation. This is arguably the most important function of

NNTP: providing an efficient means of moving large volumes of Usenet articles from one

server to another. It is thus a sensible place to start looking at the protocol.

Understanding the Usenet Server Structure

To see how NNTP propagation works, we must begin with a look at the way that the modern

Usenet network itself is organized. Usenet sites are now all on the Internet, and theoreti-

cally, any NNTP server can contact any other to send and receive Usenet articles. However,

it would be ridiculous to have a new article submitted to a particular server need to be sent

via separate NNTP connections to each and every other NNTP server. For this reason, the

Usenet logical network continues to be very important, even in the Internet era.

In theory, all that is required of the Usenet structure is that each site be connected to at

least one other site in some form. The logical network could be amorphous and without any

formal structure at all, as long as every site could form a path through some sequence of

intermediate servers to each other one. However, modern Usenet is very large, with

thousands of servers and gigabytes of articles being posted every day. This calls for a more

organized structure for distributing news.

For this reason, the modern Usenet logical network is structured loosely in a hierarchy. A

few large Internet Service Providers and big companies with high-speed Internet connec-

tions and large servers are considered to be at the top of the hierarchy, in what is

sometimes called the Usenet backbone. Smaller organizations connect to the servers run

by these large organizations; these organizations are considered to be downstream from

the backbone groups. In turn, still smaller organizations may connect further downstream

from the ones connected to the large organizations.

This hierarchical structure means that most Usenet servers only maintain a direct

connection to their upstream neighbor, and to any downstream sites to which they provide

service. A server is said to receive a news feed from its upstream connection, since that is

the place from which it will receive most of its news articles. It then provides a news feed to

all the servers downstream from it. I illustrated this in Figure 310 in the topic describing the

Usenet communication model.

As an example, suppose Company A runs a large Usenet server called largenews that is

connected to the backbone. Downstream from this server is the NNTP server mediumnews.

That server in turn provides service to smallnews. If a user posts an article to mediumnews,

it will be placed on that server immediately. That server will send the article downstream, to

smallnews, so that it can be read by that server's users. mediumnews will also, at some

point, send the article to largenews. From largenews, the message will be distributed to

other backbone sites, which will in turn pass the message down to their own downstreams.

In this way, all sites eventually get a copy of the message, even though mediumnews only

needs to connect to two other servers directly.

The term used to describe how news is propagated with NNTP is flooding. This is because

of the way that a message begins in one server and “floods” outward from it, eventually

reaching the backbone sites and then going down all the downstream “rivers” to reach

every site on Usenet.

Even though I described the logical Usenet network as a hierarchy, it is not a strict

hierarchy. For redundancy, many NNTP servers maintain connections to multiple other

servers to ensure that news propagates quickly. The transmission of articles can be

controlled by looking at message IDs to avoid duplication of messages that may be

received simultaneously by one server from more than one neighbor.

Basic NNTP Propagation Methods

Now let's look at how messages are actually propagated between servers using NNTP.

There are two techniques by which this can be done:

☯ “Pushing” News: In the push model, as soon as a server receives a new message, it

immediately tells its upstream and downstream neighbors about the message and

asks them if they want a copy of it.

☯ “Pulling” News: In the pull model, servers do not offer new articles to their neighbors.

The neighboring servers must ask for a list of new messages if they want to see what

has arrived since the last connection was established, and then request that the new

messages be sent to them.

There are advantages and disadvantages to both of these techniques, but pushing is the

model most commonly used today.

Key Concept: One important role that NNTP plays is its propagation of articles

between Usenet servers, which is what makes the entire system possible. Two

models are used for article propagation: the push model, in which a server that

receives a new message offers it to connected servers immediately, and the pull model,

where servers that receive new messages hold them until they are requested by other

servers. The push model is usually preferred since it allows for quicker communication of

messages around the system.

Article Propagation Using the "Push" Model

Here’s how the push model works. When the administrators of an NNTP server establish a

service relationship with an upstream Usenet service provider, they furnish the provider with

a list of newsgroups that the downstream server wants to carry. Whenever a new article

arrives at the upstream server within that list of groups, it is automatically sent to the

downstream site. This saves the downstream server from constantly having to ask “has

anything arrived?”

In the classical NNTP protocol as defined in RFC 977, the exchange of articles is based on

the push model, and performed using the IHAVE command. Suppose three new messages

arrive at the largenews server. It would establish an NNTP connection to mediumnews and

use IHAVE to provide the message IDs of each of the three new messages, one at a time.

The mediumnews server would respond to each one indicating whether it already had that

message or not. If not, largenews would send it the message. An example article trans-

action using the push model of propagation is illustrated in Figure 312.

The main advantage of this technique is that it ensures that a server is not sent a duplicate

copy of a message that it already has. The problem with it in modern Usenet is that it is

slow, because the server must respond to the IHAVE command before the message or the

next command can be sent by the client.

Improving Push Propagation Efficiency With Streaming Mode

Due to the inefficiency problem just described, one of the more important NNTP extensions

is streaming mode, which changes how news pushing is done. When this mode is enabled,

the client machine uses the CHECK command instead of IHAVE to ask the server if it wants

a particular message. The server responds back as for IHAVE to indicate if it wants the

message; if it does, the client sends the message with the TAKETHIS command.

The benefit of CHECK/TAKETHIS is that the client does not have to wait for a reply to

CHECK before sending the next command. While the client is waiting for a reply to the first

CHECK command, it can do something else, like sending the next CHECK command,

allowing commands to be streamed for greater efficiency. So the client could send a

CHECK command for the first new message, then a CHECK for the second while waiting

for a reply from the server to the first one. Many CHECK commands could be sent in a

stream and then TAKETHIS commands sent for each reply received to CHECKs sent

earlier indicating that the message was wanted by the server.

Article Propagation Using the "Pull" Model

The pull model is implemented using the NEWNEWS and ARTICLE commands. The client

connects to the server and sends the NEWNEWS command with a date specifying the date

and time that it last checked for new messages. The server responds with a set of message

IDs for new articles that have arrived since that date. The client then requests each new

message using the ARTICLE command.

Figure 312: NNTP Article Propagation Using The “Push” Model

This example shows how Usenet articles are moved between servers using the conventional “push” model of

propagation. Here, the device acting as an NNTP client (which recall may in fact be an NNTP server) has two

messages available to offer to the server. It sends the IHAVE command specifying the message ID of the first

message, but the server already has that message so it sends a 435 “do not send” reply. The client then

issues an IHAVE with the second message ID; the server wants this one, so it sends a 335 reply; the client

sends the Usenet message, ending with a single period on a line by itself. The server indicates that it received

the message and the client, done with its transactions, quits the session.

NNTP ServerNNTP Client

(TCP)

1. Establish TCP

Connection To Server

2. Establish TCP Connection,

Send 200 "Ready" Reply

4. Receive IHAVE; Already

Have Message, Send 435 "Do

Not Send" Reply

3. Receive "Ready" Reply,

Send IHA VE For Message #1

200

IHAVE <msg1>

5. Receive "Do Not Send"

Reply; Send IHAVE For

Me ssa g e #2

435

IHAVE <msg2>

6. Receive IHAVE; M e ssa ge I s

New, Send 335 "Send

Message" Reply

335

Send "." By Itself To Indicate

End of Message Body

235

8. Receive and Store Message

Lines

9. Receive "OK" Reply; Done

With Transfers, Quit

(message line)

...

Receive "."; Store E-Mail

Message, Send 235 "OK" Reply

...

QUIT

205

10. Receive QUIT, Send 205

"Goodbye" Reply, Close

Transmission Channel

11. Receive "Goodbye" Reply,

Close Transmission Channel

7. Receive Intermediate

Reply; Send Usenet Message

Headers and Body

Note that the push and pull models can be combined in a single session. A client can

connect to a server, use NEWNEWS to check for new messages on that server, and then

IHAVE or CHECK to inform the server about new messages the client wants to send. In

practice, it is more common for only one or the other of the models to be used between a

pair of servers on any given exchange.

In addition to propagating new messages, NNTP is also used to allow servers to commu-

nicate information about new newsgroups that have been created. This is done using the

NEWGROUPS command, which is specified with a date and time like NEWNEWS. In

response, the server sends to the client a list of new newsgroups that have been created

since the specified date/time.

NNTP Client-Server Communication Process: News Posting and Access

The propagation of Usenet articles is indeed a defining function of the Network News

Transfer Protocol (NNTP)—would I lie to you? ☺ However, one critical area where NNTP

differs from its progenitor, SMTP, is that NNTP is used not just for inter-server communi-

cation. It is also the protocol utilized for the initial posting of Usenet messages, and reading

them as well. In fact, the majority of NNTP commands deal with the interaction between

user client machines and NNTP servers, not communication between servers.

An NNTP client is any software program that knows the NNTP protocol and is designed to

provide user access to Usenet. NNTP clients are usually called newsreaders, and provide

two main capabilities to a user: posting (creating) and reading Usenet messages. Usenet

newsreaders exist for virtually all hardware and software platforms, and range greatly in

terms of capabilities, user interface and other characteristics. Most people today use a

Usenet newsreader on a client computer that must make NNTP connections to a separate

NNTP server to read and post news. These programs are analogous to e-mail clients, and

in fact, many e-mail clients also function as NNTP clients.

News Posting

Posting a Usenet message is the first step in the overall Usenet communication process

(although many Usenet articles are actually replies to other articles, so it's a bit of a

“chicken and egg” thing.) Article posting is quite straight-forward with NNTP. The client

establishes a connection to the server and issues the POST command. If the server is

willing to accept new articles it replies with a prompt for the client to send it the article. The

article is then transmitted by the client to the server. Some newsreaders may “batch” new

articles so they can be sent in a single NNTP session rather than submitting them one at a

time.

News Access and Article Reading

Newsreaders also establish an NNTP connection to a server to read Usenet articles. NNTP

provides a large number of commands to support a variety of different article access and

retrieval actions that may be taken by a user. The first step in reading is sometimes to

examine the list of available newsgroups. Using the LIST command, the client requests

from the server a list of the newsgroups available for reading and posting. RFC 977 defined

the basic LIST command, which returns a list of all groups to the client. RFC 2980 defines

numerous extensions to the command to allow a client to retrieve only certain types of infor-

mation about groups on the server. Since the number of Usenet newsgroups is so large

today, many times this listing of newsgroups is now skipped unless the user specifically

requests it.

The next step in Usenet message access is typically to select a newsgroup to read from the

list of groups available. Again, since there are so many groups today, most newsreaders

allow a user to search for a group name using a pattern or partial name string. The GROUP

command is then sent to the server with the name of the selected group. The server returns

the first and last current article numbers for the group to the client.

Messages are identified in two ways, one absolute and the other site-specific. The article’s

Message ID is a fixed identifier that can be used to uniquely represent it across Usenet; this

is what is used in inter-server communication to determine whether or not each site has a

copy of a given message. In contrast, article numbers are server-specific; they represent

the numbers assigned to those articles as they arrived at that server and are used as a

“short hand” to more easily refer to articles in a newsgroup. Thus, the same message will

have a different article number on each NNTP server. They are used for convenience, since

an article number is much shorter than a message ID. During a session, the NNTP server

also maintains a “current article pointer” that can be used for stepping sequentially through

a newsgroup.

News Access Methods

There are several different ways that the newsreader can access messages in a group,

depending on how it is programmed and on what the user of the software wants.

Full Newsgroup Retrieval

This is the “brute force” technique: the client simply requests that the server send it all the

messages in the group. This is done by issuing the ARTICLE command to select the first

current message in the group, using the first article number returned by the GROUP

command. This sets the server's internal pointer for the session to point to the first article,

so it can be retrieved. The NEXT command is then used to advance the pointer to the next

message, and the ARTICLE command to retrieve it. This continues until the entire group

has been read. Figure 313 illustrates the process.

The retrieved messages are stored by the newsreader and available for instant access by

the user. This method is most suitable for relatively small newsgroups and/or users with fast

Internet connections.

Figure 313: NNTP Full Newsgroup Retrieval Process

There are many ways that an NNTP client can access and read Usenet messages on a server. One common

method is to retrieve the entire contents of a newsgroup. In this example, the client uses the GROUP

command to select the newsgroup comp.protocols.tcp-ip for reading; the server responds with a 211 “group

selected” reply, which includes important statistics about the group. The client uses the ARTICLE command

with the number of the first article in the group, 177, to read it from the server. The server then sends the

message line by line, ending it with a single period on a line. The client uses the NEXT command to tell the

server to advance its internal article pointer to the next message, which often will not be the next consecutive

number after the one just read; here it is 179. The client can then read that message by sending the ARTICLE

command by itself; since no parameters are given the server returns the current message (#179).

NNTP ServerNNTP Client

(TCP)

1. Establish TCP

Connection To Server

2. Establish TCP Connection,

Send 200 "Ready" Reply

4. Receive GROUP; Send 211

"Group Selected" Reply With

Group Statistics: 23 Articles,

First #177, Last #202

3. Receive "Ready" Reply,

Send GROUP Command to

Select Newsgroup

200

GROUP comp.protocols.tcp-ip

5. Receive "Group Selected"

Reply; Read First Message

With ARTICLE Command

211 23 177 202

comp.protocols.tcp-ip

ARTICLE 177

6. Receive ARTICLE; Send 220

"Article Retrieved" Message,

Followed by Message Lines

220

Receive "."; Store Usenet

Message, Move To Next

Me ssa ge

223 179

9. Receive "Article Retrieved"

Reply; Get Article Using

ARTICLE Command (No

Arguments Required)

...

Send "." By Itself To Mark End

of Message Body

...

10. Receive ARTICLE; Send

220 "Article Retrieved"

Message, Followed by

Message Lines

7. Receive "Article Retrieved"

Reply; Receive and Store

Message Lines

(message line)

8. Receive NEXT; Advance

Pointer, Send 223 "Article

Retrieved" Reply Indicating

New Current Article Is #179

ARTICLE

220

(message line)

...

Newsgroup Header Retrieval

Since downloading an entire newsgroup is time-consuming, many newsreaders

compromise by downloading the headers of all messages instead of the full message. The

process is the same as for full newsgroup retrieval, but the HEAD command is used to

retrieve just an article's headers instead of the ARTICLE command. This takes less time

than retrieving each message in its entirety.

The XHDR command extension can also be used, if the server supports it, to more

efficiently retrieve only a subset of the headers for the messages, such as the subject line

and author.

Individual Article Retrieval

It is also possible to retrieve a single message from a group, using the ARTICLE command

and specifying the article's message identifier.

Key Concept: While NNTP is best known for its role in inter-server propagation, it is

also used by Usenet clients to write and read articles. A number of different

commands provides flexibility in how articles can be read by a client device. A client

can retrieve an entire newsgroup, only a set of newsgroup headers, or individual articles.

Other commands also support various administrative functions.

Other Client-Server Functions

In addition to reading and posting, NNTP includes commands to support other “miscella-

neous” tasks that a Usenet user may wish to perform. The client can ask the server for help

information using the HELP command, or get a list of new newsgroups using the

NEWGROUPS command, as described in the inter-server communication topic.

Most modern newsreaders include capabilities that go far beyond the basic posting and

reading functions described above. Most maintain their own sets of configuration files that

allow a user to maintain a set of favorite “subscribed” newsgroups, rather than having to

choose a group to read from the master list each time Usenet is accessed. Newsreaders

also keep track of which articles have been read by a user in each subscribed newsgroup,

so one need not wade through a whole newsgroup to see new messages that have been

posted.

Article Threading

One particularly useful enhancement to basic Usenet article reading is threading. This

feature allows a newsreader to display articles not strictly in either alphabetical or chrono-

logical order, but rather grouped into conversations using the information in the articles'

References headers. Threading is especially useful in busy newsgroups, as it allows you to

see all the articles in a particular discussion at once, rather than trying to juggle messages

from many conversations simultaneously.

A problem with threading is that it takes a long time for a newsreader to sift through all those

References lines and construct the article threads. To speed up this process, many servers

now cache extra threading or overview information for newsgroups, which can be retrieved

by the client to save time when a newsgroup is opened. This is done using the XTHREAD

or XOVER NNTP command extensions.

NNTP Commands

One of the great strengths of the open, cooperative process used to develop Internet

standards, is that new protocols are usually designed by building upon successes in older

ones. This both saves development time and effort, and promotes compatibility between

technologies. As I explained in the overview of NNTP, it was based in many ways on

principles from SMTP; SMTP in turn borrowed ideas from earlier protocols, Teln et and FTP.

This legacy can be seen in the similarities between NNTP commands and those of these

earlier protocols.

Command Syntax

As in SMTP, all NNTP commands are ASCII text that are sent over the NNTP TCP

connection to an NNTP server, from the device acting as the client (which may be a

newsreader client or an NNTP server itself). These are standard text strings adhering to the

Telnet Network Virtual Terminal (NVT) format, terminated by the two-character “CRLF”

sequence. As is the case with SMTP and FTP, you can conduct an interactive session with

an NNTP server by using Telnet to connect to it on port 119.

The basic syntax of an NNTP command is as follows:

<command-code> <parameters>

Unlike SMTP, NNTP commands are not restricted to a length of four characters. The

parameters that follow the command are separated by one or more space characters, and

are used to provide necessary information to allow the server to execute the command.

NNTP commands are not case-sensitive.

Base Command Set

The main NNTP specification, RFC 977, describes the base set of commands supported by

NNTP clients and servers. They are not broken into categories but rather listed alphabeti-

cally, as I have done in Table 267. Additional details on how most of these commands are

used can be found in the topics on news article propagation and news posting and access.

Table 267: NNTP Base Commands (Page 1 of 2)

Command

Code

Command Parameters Description

ARTICLE

Retrieve

Article

Message ID or

server article

number.

Tells the server to send the client a particular Usenet

article. The article to be retrieved may be specified

either using its absolute, universal message ID, or its

locally-assigned article number.

When the command is issued with an article number,

this causes the server's internal message pointer to be

set to the specified article. If the message pointer is

already set to a particular article, the ARTICLE

command can be issued without an article number and

the current message will be retrieved.

HEAD

Retrieve

Article

Headers

Message ID or

server article

number.

Same as the ARTICLE command, but retrieves only

the article's headers.

BODY

Retrieve

Article Body

Message ID or

server article

number.

Same as the ARTICLE command, but returns only the

body of the article.

STAT

Retrieve

Article

Statistics

Server article

number

Conceptually the same as the ARTICLE command, but

does not return any message text, only the message ID

of the article. This command is usually used for the

purpose of setting the server's internal message

pointer, so STAT is normally invoked only with an article

number (and not a message ID).

GROUP

Select

Newsgroup

Newsgroup name

Tells the server the name of the newsgroup that the

client wants to access. Assuming the group specified

exists, the server returns to the client the numbers of

the first and last articles currently in the group, along

with an estimate of the number of messages in the

group. The server's internal article pointer is also set to

the first message in the group.

HELP

Get Help

Information

None

Prompts the server to send the client help information,

which usually takes the form of a list of valid commands

that the server supports.

IHAVE

Offer Article

To Server

Message ID

Used by the client in an NNTP session to tell the server

that it has a new article that the server may want. The

server will check the message ID provided and

respond to the client indicating whether or not it wants

the client to send the article.