Charles M. Kozierok The TCP-IP Guide

Подождите немного. Документ загружается.

Usenet Newsgroup Hierarchies

The total collection of newsgroup hierarchies is in many ways similar to the domain name

tree structure used in DNS. Each Usenet hierarchy is like a collection of all the domain

names within a DNS top-level domain. Just as a domain name like www.pcguide.com is

formed by appending the label of the top level domain “com” to the second-level domain

name “pcguide” and the subdomain “www”, newsgroup names are created in the same

way. They are created from a top-level newsgroup hierarchy name, to which are attached a

set of descriptive label that describe the newsgroup's place in the hierarchy.

One difference between DNS and Usenet hierarchies is that while DNS names are created

from right to left as you go “down the tree”, Usenet newsgroup names are formed in the

more “natural” (for English speakers) left to right order. For example, one of the main

Usenet hierarchies is the “comp” hierarchy, devoted to computer topics. Within “comp” is a

sub-hierarchy on data communications called “dcom”, and within that a group that

discusses data cabling. This group is called comp.dcom.cabling. Almost all newsgroups are

structured in this manner.

The "Big Eight" Newsgroup Hierarchies

One problem with the “decentralized, anarchistic” nature of Usenet is ensuring coordination

in certain areas where we want everyone to be on the same page; one of these is

newsgroup naming. If we let just anyone create a newsgroup, we might well end up with

many, even dozens of groups that all discuss the same topic. Imagine that someone had a

question on data cabling and didn't realize that comp.dcom.cabling existed, so they created

a new group called comp.datacomm.cabling. The two groups could coexist, but this would

lead to both confusion and fragmenting of the pool of people interested in this topic.

To avoid this problem, many of the administrators of large Usenet systems collaborated on

a system for organizing many of the more commonly-used Usenet groups into eight hierar-

chies, and devised a specific procedure for creating new newsgroups within them. There is

a actually a long history behind how this came about; these are today called the Big Eight

Usenet hierarchies, and are summarized in Table 264.

Table 264: Usenet “Big Eight” Newsgroup Hierarchies

Hierarchy Description

comp.*

Newsgroups discussing computer-related topics, including hardware, software,

operating systems and techniques.

humanities.* Groups discussing the humanities, such as literature and art.

misc.* Miscellaneous topics that don't fit into other Big Eight hierarchies.

news.* Groups discussing Usenet itself and its administration.

rec.* Recreation topics, such as games, sports and activities.

sci.*

Science newsgroups, covering specific areas such as physics and chemistry,

research topics and so forth.

These eight hierarchies contain many of the most widely-used groups on Usenet today. For

example, professional baseball is discussed in rec.sport.baseball, Intel computers in

comp.sys.intel and Middle East politics in talk.politics.mideast.

The Big Eight hierarchies are rather tightly controlled in terms of their structure and the

newsgroups they contain. The process to create a new Big Eight newsgroup is democratic

and open; anyone can propose a new group and if there is enough support, it will be

created by the cooperating system administrators who agree to follow the Big Eight system.

However, this creation process is rather complex and time-consuming. Some people find

this unacceptable and even object to the entire concept of this restricted process; others

consider it advantageous, as it keeps the Big Eight hierarchies relatively “orderly” by

slowing the rate of change to existing newsgroups and the number of new groups added.

Other Newsgroup Hierarchies

For those who prefer a more free-wheeling environment and do not want to submit to the

Big Eight procedures, there is the “alternative” Usenet hierarchy, which begins with the

hierarchy name alt. This hierarchy includes many thousands of groups. Some are quite

popular, but many are not used at all; this is a side-effect of the relative ease with which an

alt group can be created.

In addition to these nine hierarchies there are dozens of additional, smaller hierarchies.

Many of these are regional or even company-specific. For example, the “ne.” hierarchy

contains a set of newsgroups discussing issues of relevance to New England; “fr.*” covers

France, and “de.*” Germany. Microsoft has its own set of public newsgroups in the

“microsoft.*” hierarchy. Figure 311 shows the Big Eight hierarchies and some of the other

hierarchies that exist.

Key Concept: Usenet messages are not addressed to individual users; rather, they

are posted to newsgroups. Each newsgroup represents a topic; those with an

interest in the subject of a group can read messages in it, and reply to them as well.

Usenet newsgroups are arranged into tree-like hierarchies that are similar in structure to

DNS domains. Many of the most widely-used newsgroups are found in a collection of

general-interest hierarchies called the Big Eight. There are also many regional and special-

purpose hierarchies.

soc.* Society and social discussions, including groups on specific cultures.

talk.*

Groups primarily oriented around discussion and debate of current events and

happenings.

Table 264: Usenet “Big Eight” Newsgroup Hierarchies

Hierarchy Description

Unmoderated and Moderated Newsgroups

Most newsgroups are open to all to use, and are called unmoderated because a message

sent to them goes directly out to the whole Usenet server internetwork. In contrast, a small

percentage of newsgroups is moderated, which means that all messages sent to the group

are screened and only the ones that are approved by a moderator (or moderator team) are

really posted.

The purpose of moderated groups is to ensure that discussions in a particular group remain

on-topic. They are often created to handle topics that are controversial, to ensure that

debates remain constructive and disruption is avoided. For example, rec.guns is moderated

to ensure that discussions focus on the use of guns and not on endless political arguments

related to gun control and the like (which has a place, in talk.politics.guns).

Moderated groups are also sometimes used for specialty groups intended only for

announcements, or for groups where the content is restricted. For example,

rec.food.recipes is moderated so that it only contains recipes and recipe requests; this

helps people find recipes easily without having to wade through lots of discussion. Finally,

moderated versions of unmoderated groups are sometimes created when a few disruptive

elements choose to post large volumes in the unmoderated groups, making normal

discussion difficult.

Figure 311: Usenet Newsgroup Hierarchies

Usenet newsgroups are arranged into tree-like structures called hierarchies. Eight of these are centralized,

widely-used, general-purpose hierarchies, which are today called the “Big Eight”. The “alternate” (“alt”)

hierarchy is a very loosely-structured set of thousands of groups covering every topic imaginable. In addition

to these, there are many hundreds of regional, private and special-purpose hierarchies.

Root

comp.*

humanities.*

misc.*

new s.*

rec.*

sci.*

soc.*

talk.* alt.*

can.*, ne.*, ... adobe.*,

microsoft.*, ...

"Big Eight" Usenet Hiearchies

"Alternate"

Hierarchy

Regional

Hierarchies

Private / Special-

Purpose

Hierarchies

Cross-Posting to Multiple Newsgroups

It is possible for a single article to be posted to multiple newsgroups. This process, called

cross-posting, is used when a message pertains to two topics, or to allow a sender to reach

a wider audience. For example, if I live in the Seattle area and have a problem with my

house, I might legitimately cross-post to seattle.general and misc.consumers.house.

This is more efficient than posting the same message to each group independently for two

reasons. First, only one copy of the message will be stored on each Usenet server rather

than two. Second, Usenet participants who happen to read both groups won't see the

message twice. However, cross-posting to very large numbers of newsgroups is usually

considered disruptive and a breach of Usenet etiquette.

Usenet Message Format and Special Headers

Usenet is designed to permit users to exchange information in the form of messages that

are sent from one computer to another. Like any message-based networking application, it

is necessary that all Usenet client and server software agree to use a common message

format. This ensures that all devices and programs are able to interpret all Usenet articles in

a consistent manner.

General Message Format

While Usenet was created as an alternative to e-mail, and there are obviously differences in

how each treats messages, there are also many similarities. Both are text-oriented

messaging systems with similar needs for communicating both content and control infor-

mation. The creators of Usenet realized that there would be many advantages to basing the

Usenet message format on the one used for e-mail, rather than creating a new format from

scratch. For one thing, the e-mail message format was already widely used, and adopting it

for Usenet would save implementation time and effort. It would also enhance compatibility

between e-mail and Usenet messages, allowing software designed to process or display e-

mail to also work with Usenet articles.

For this reason, the Usenet message format was defined based on the RFC 822 standard

for e-mail messages, discussed thoroughly in its own subsection of the section on e-mail in

this Guide. RFC 822 messages begin with a set of headers that contain control and

descriptive information about the message, followed by a blank line and then the message

body, which contains the actual content.

One important attribute of the RFC 822 standard is the ability to define custom headers that

add to the regular set of headers defined in the standard itself. Usenet articles require some

types of information not needed by e-mail, and these can be included in specially-defined

headers while still adhering to the basic RFC 822 format. At the same time, headers

specific to e-mail that are not needed for Usenet can be omitted.

Thus, there is no structural difference at all between a Usenet article and an e-mail

message; they differ only in the kinds of headers they contain, and the values for those

headers. For example, a Usenet message will always contain a header specifying the

newsgroup(s) to which the article is being posted, but will not carry a “To:” line as an e-mail

message would.

Key Concept: Usenet articles use the same RFC 822 message format as electronic

mail messages. The only difference between a Usenet article and e-mail message is

in the header types and values used in each.

Usenet Header Categories and Common Headers

All Usenet headers are defined according to the standard header format defined in RFC

822:

As with e-mail messages, headers may extend on to multiple lines, following the indenting

procedure described in the RFC 822 standard.

The current standard for Usenet messages, RFC 1036, describes the header types for

Usenet messages. The headers are divided into two categories: mandatory headers and

optional headers. Some are the same as headers of the equivalent name used for e-mail,

some are similar to e-mail headers but used in a slightly different way, while others are

unique to Usenet. Table 265 describes these header fields and how they are used.

Table 265: Usenet Header Categories and Headers (Page 1 of 2)

Header

Category

Header Name Description

Mandatory

Headers

From: The e-mail address of the user sending the message, as for e-mail.

Date:

The date and time that the message was originally posted to Usenet. This

is usually the date/time that the user submitted the article to his or her

local NNTP server.

Newsgroups:

Indicates the newsgroup or set of newsgroups to which the message is

being posted. Multiple newsgroups are specified by separating them with

a comma; for example: “Newsgroups: news.onegroup,rec.secondgroup”.

Subject:

Describes the subject or topic of the message. Note that this header is

mandatory on Usenet despite being optional for e-mail; it is important

because it is used by readers to decide what messages to open.

Message-ID:

Provides a unique code for identifying a message; normally generated

when a message is sent. The message ID is very important in Usenet,

arguably more so than in e-mail. The reason is that delivery of e-mail is

performed based on recipient e-mail addresses, while the propagation of

Usenet messages is controlled using the message ID header.

Path:

This is an informational field that shows the path of servers that a

particular copy of a message followed to get to the server where it is being

read. Each time a server forwards a Usenet article, it adds its own name to

the list in the Path header. The entries are usually separated by excla-

mation points.

For example, if a user on Usenet server A posts a message, and it is

transported from A to G, then X, then F and finally to the server Q where a

second user reads it, the person on server Q would see something like

this in the Path header: “Q!F!X!G!A”.

Optional

Headers

(part 1 of 2)

Reply-To:

It is possible to reply back to a Usenet article author using e-mail, which by

default goes to the address in the From: line. If this header is present, the

address it contains is used instead of the default From: address.

Sender:

This header indicates the e-mail address of the user who is sending the

message, if different from the message originator. This is functionally the

same as the Sender: header in e-mail messages, but is used in a slightly

different way. Normally, when a Usenet message is posted, the sender's

e-mail address is automatically filled in to the From: line. If the user

manually specifies a different From: line, the address from which the

message was actually sent is usually included in the Sender: line. This is

used to track the true originating point of articles.

Followup-To:

A reply to a Usenet message is usually made back to Usenet itself, and is

called a follow-up. By default, a follow-up goes to the newsgroup(s)

specified in the original message's Newsgroups: header. However, if the

Followup-To: header is included, follow-ups to that message go to the

newsgroups specified in the Followup-To: header instead.

This header is sometimes used to “route” replies to a message to a

particular group. Note, however, that when a user replies to a message,

this field only controls what appears in the new message's Newsgroups:

line by default. The user can override the Newsgroups: header manually.

Additional Usenet Headers

Usenet messages may also contain additional headers, just as is the case with e-mail

messages. Some of these headers are entirely “custom” and are included by individual

users to provide extra information about an article. Others are used in many or even most

Optional

Headers

(part 2 of 2)

Expires:

All Usenet messages are maintained on each server for only a certain

period of time, due to storage limitations. The expiration interval for each

newsgroup is controlled by the administrator of each site. If present, this

line requests a different expiration for a particular message; it is usually

used only for special articles. For example, if a weekly announcement is

posted every Monday morning, each article might be set to expire the

following Monday morning, to make sure that people see the most current

version.

References:

This header lists the message IDs of prior messages in a conversation.

For example, if someone posts a question to a newsgroup with message

ID “AA207”, and a reply to that message is made, the software will

automatically insert the line “References: AA207” into the reply. This is

used by software to group together articles into conversations (called

threads) to make it easier to follow discussions on busy newsgroups.

Control:

Indicates that the article is a control message and specifies a control

action to be performed, such as creating a new newsgroup.

Distribution:

By default, most messages are propagated on Usenet worldwide. If

specified, this line restricts the distribution of a message to a smaller area,

either geographical or organizational.

Organization:

Describes the organization to which the article sender belongs. Often filled

in automatically with the name of the user’s Internet Service Provider.

Keywords:

Contains a list of comma-separated keywords that may be of use to the

readers of the message. Keywords can be useful when searching for

messages on a particular subject matter. This header is not often used.

Summary: A short summary of the message; again, rarely used in practice.

Approved:

This header is added by the moderator of a moderated newsgroup to tell

the Usenet software that the message has been approved for posting.

Lines: A count of the number of lines in the message.

Xref:

While Usenet articles are identified by message ID, they are also given a

number by each Usenet server as they are received. These article

numbers, which differ from one system to the next, are usually listed in

this cross-reference header.

This information is used when a message is cross-posted to multiple

groups. In that case, as soon as a user reads the message in one group, it

is marked as having been read in all the others where it was posted. This

way, if the user later reads one of those other groups, they will not see the

message again.

Table 265: Usenet Header Categories and Headers (Page 2 of 2)

Header

Category

Header Name Description

current Usenet articles, and have become almost “de facto” standard headers through

common use. Many of these custom headers are preceded by “X-”, indicating that they are

“experimental” or “extra” headers.

Some of the more frequently encountered “additional” Usenet headers are shown in Ta bl e

266.

Usenet MIME Messages

Since Usenet follows the RFC 822 standard, Multipurpose Internet Mail Extensions (MIME)

can be used to format Usenet messages. When this is done, you will see the usual MIME

headers (such as MIME-Version, Content-Type and so forth) in the message.

Note that the use of MIME in Usenet messages is somewhat controversial. Some

newsreaders are not MIME-compliant and make a mess when trying to display some of

these messages, and many Usenet veterans object to the use of anything but plain text in

Usenet messages. Despite this, MIME messages are becoming more common, for better or

worse.

Table 266: Common Additional Usenet Headers

Header Name Description

NNTP-Posting-Host:

Specifies the IP address or the DNS domain name of the host used to originally

post the message. This is usually either the address of the client that the author

used for posting the message, or the sender's local NNTP server.

User-Agent:

(or)

X-Newsreader:

The name and version number of the software used to post the message.

X-Trace: Provides additional information that can be used to trace the message.

X-Complaints-To:

An e-mail address to use to report abusive messages. This header is now included

automatically by many Internet Service Providers.

TCP/IP Network News Transfer Protocol (NNTP)

Usenet started out as an informal network of UNIX computers using dial-up UUCP connec-

tions to transmit messages between servers. This arrangement arose out of necessity, and

it worked fairly well, though it had a number of problems. Once the Internet became widely

used in the 1980s, it provided the ideal opportunity for a more efficient means of distributing

Usenet articles. A special TCP/IP protocol was developed for sending these messages,

called the Network News Transfer Protocol (NNTP). NNTP is used today to carry billions of

copies of Usenet messages from computer to computer every day.

In this section I provide a detailed description of the operation of the Network News Transfer

Protocol. I begin with an overview of how NNTP works in general terms. I then explain the

two fundamentals ways that NNTP is used: for the propagation of news articles between

servers, and for client article posting and access. I then provide more information about

standard NNTP commands, as well as the newer NNTP command extensions, and also

illustrate NNTP status responses and the common response codes.

Background Information: NNTP is similar to the Simple Mail Transfer Protocol

(SMTP) in many ways, including its basic operation and command set and reply

format. You may therefore find this section easier to understand if you are familiar

with SMTP.

NNTP Overview and General Operation

Usenet began as a logical internetwork of cooperating hosts that contacted each other

directly. In the early Usenet, a user would post a message to his or her local server. There it

would stay until that server either contacted or was contacted by another server. The

message would then be transferred to the new server, where it would stay until the second

server contacted a third one, and so on.

This transport mechanism was functional, but seriously flawed in a number of ways.

Servers were not continually connected to each other; they could only communicate by

making a telephone call using an analog modem. Thus, messages would often sit for hours

before they could be propagated. Modems in those days were also very slow by today's

standards—2400 bits per second or even less—so it took a long time to copy a message

from one server to another. Worst of all, unless two sites were in the same city, these phone

calls were long distance, making them quite expensive.

Why was this system used despite all of these problems? Simple: there was no alternative.

In the late 1970s and early 1980s, there was no Internet as we know it, and no other

physical infrastructure existed to link Usenet sites together. It was either use UUCP over

telephone lines, or nothing.

The Development of NNTP

Everything changed as the fledgling ARPAnet grew into the modern Internet. As the Internet

expanded, more and more sites connected to it, including many sites that were participating

in Usenet. Once a pair of sites were both on the Internet, it was an easy decision to use the

Internet to send Usenet articles rather than slow, expensive telephone calls. Over time,

more and more Usenet sites joined the Internet, and it became clear that just as e-mail had

moved from UUCP to the TCP/IP Internet, the future of Usenet was on the Internet as well.

The shifting of Usenet from UUCP connections to TCP/IP internetworking meant that some

rethinking was required in how Usenet articles were moved from server to server. On the

Internet, Usenet was just one of many applications, and the transfer of messages had to be

structured using one of the two TCP/IP transport protocols, TCP or UDP.

Thus, like other applications, Usenet required an application-level protocol to describe how

to carry Usenet traffic over TCP/IP. Just as Usenet had borrowed its message format from

e-mail's RFC 822, it made sense to model its message delivery protocol on the one used by

e-mail: the Simple Mail Transfer Protocol (SMTP). The result was the creation of the

Network News Transfer Protocol (NNTP), published as RFC 977 in February, 1986.

Overview of NNTP Operation

The general operation of NNTP is indeed very similar to that of SMTP. NNTP uses TCP,

with servers listening on well-known TCP port 119 for incoming connections, either from

client hosts or other NNTP servers. As in SMTP, when two servers communicate using

NNTP, the one that initiates the connection plays the role of client for that exchange.

After a connection is established, communication takes the form of commands sent by the

client to the server, and replies returned from the server to the client device. NNTP

commands are sent as plain ASCII text, just like those used by SMTP, FTP, HTTP and other

protocols. NNTP responses take the form of three-digit reply codes as well as descriptive

text, again just like SMTP (which in turn borrowed this concept from FTP).

NNTP was designed to be a comprehensive vehicle for transporting Usenet messages. It is

usually most often considered as a delivery protocol for moving Usenet articles from one

server to another, but is also used for connections from client hosts to Usenet servers for

posting and reading messages. Thus, the NNTP command set is quite extensive, and

includes commands to handle both inter-server and client-server communication. For

message propagation, a set of commands is provided to allow a server to request new

articles from another server, or to provide new articles to another server. For message

posting and access, commands allow a client to request lists of new newsgroups and

messages, and to retrieve messages for display to a user.

The commands defined in RFC 977 were the only “official” ones for over a decade.

However, even as early as the late 1980s, implementors of NNTP server and client

software were adding new commands and features to make NNTP both more efficient and

useful to users. These NNTP extensions were eventually documented in RFC 2980,

published in 2000. I describe them in more detail later in this section.