Charles M. Kozierok The TCP-IP Guide

Подождите немного. Документ загружается.

The strings “=?” and “?=” are used to bracket the non-ASCII header, which flags it as a

MIME encoded header to the recipient's e-mail client. The other elements, separated by “?”,

indicate how the non-ASCII text is encoded:

☯ <charset>: The character set used, such as “iso-8859-1”.

☯ <encoding>: Two different encoding types are defined, each represented by a single

letter for brevity:

☯ “B”: This indicates base64 encoding, as described in the previous topic.

☯ “Q”: This indicates quoted-printable encoding, again as in the prior discussion.

☯ <encoded-text>: The non-ASCII text that has been encoded as ASCII using the

encoding type indicated.

As you can see, this method is analogous to how a non-ASCII message body or body part

would be encoded, but the information about the encoding has been condensed so every-

thing can fit in a single header line. The “<charset>” parameter is somewhat analogous to

the Content-Type header for a message body, but since headers can only contain text, it

specifies what kind of text it is. The “<encoding>” parameter is clearly equivalent to the

Content-Transfer-Encoding header.

Example Non-ASCII MIME Header

Here's an example of a non-ASCII header, using the GB2312 character set (for Chinese

characters) and base64 encoding:

Subject: =?GB2312?B?u7bTrbLOvNPDwLn61bm74Q==?=

I hope that doesn't say anything inappropriate; I took it from a piece of spam e-mail I

received once! ☺

Key Concept: In addition to its many functions for encoding a variety of data in e-

mail message bodies, MIME provides a feature that allows non-ASCII information to

be placed into e-mail headers. This is done by encoding the data using either

quoted-printable or base64 encoding, and then using a special format for the header value

that specifies its encoding and character set. This technique is especially useful for e-mail

sent in languages that cannot be represented easily in standard ASCII, such as many Asian

languages.

TCP/IP Electronic Mail Delivery Protocol: The Simple Mail Transfer

Protocol (SMTP)

I have emphasized in my overall description of TCP/IP electronic mail that communication

using e-mail requires the interaction of various protocols and elements. One mistake that

some people make is to equate the method used for delivering e-mail with the entire

system. This is, however, an understandable mistake: just as the postal service is only a

part of the whole system of mailing a letter; it is nonetheless a very big part. Likewise, the

delivery of e-mail from sender to recipient is arguably the most important part of e-mail as a

whole. In modern TCP/IP, this task is the responsibility of the Simple Mail Transfer Protocol

(SMTP).

In this section, I describe in detail the operation of the Simple Mail Transfer Protocol

(SMTP). I begin with an overview and history of SMTP and a discussion of the standards

that define it. I then examine the way that SMTP client/server communication and message

transport works. I explain the way that SMTP servers establish connections and transaction

sessions, and then the process by which mail is transferred from one to another. I describe

some of the special features implemented in SMTP, and discuss SMTP security issues as

well. I conclude with a reference summary of SMTP commands and replies.

Background Information: My discussion of SMTP assumes that you already have

a basic understanding of the general concepts of TCP/IP e-mail, as well as famil-

iarity with TCP/IP e-mail addressing and message formatting.

SMTP Overview, History and Standards

The overview and history of the TCP/IP electronic mail system describes how TCP/IP

evolved from its early beginnings to its current form. Since the mechanism used to deliver

e-mail is such a big part of the system as a whole, any overview of the system must of

necessity discuss how delivery mechanisms have changed as well. In the case of TCP/IP, I

explained how the delivery of mail evolved through many forms during the 1970s as devel-

opers sought to find effective ways of communicating e-mail messages between systems.

Most of these efforts involved attempts to transmit mail using existing protocols; this makes

sense, since it is easier to adapt a technology than design one from scratch.

Early SMTP History

One important achievement was the publishing of the Mail Transfer Protocol (MTP), which

was first defined in RFC 772 in September 1980, then updated in RFC 780 in May 1981.

MTP describes a set of commands and procedures by which two devices can connect

using TCP to exchange e-mail messages. Its operation is described largely using elements

borrowed from two early TCP/IP application protocols that were already in use at that time:

Teln et and FTP. The commands of MTP are in fact based directly on those of FTP.

There wasn't anything inherently wrong with basing e-mail delivery on something like FTP,

but defining it this way made MTP somewhat of a “hack”. It was also restricted to the

capabilities defined by FTP, a general file transfer protocol, so it was not possible to include

features in the protocol that were specific to sending and receiving mail. Due to the impor-

tance of e-mail, a specific protocol designed for the purpose of delivering e-mail was

warranted. This protocol was first defined in RFC 788, published in November 1981: the

Simple Mail Transfer Protocol (SMTP).

The name suggests that SMTP is “simpler” than the “non-simple” MTP that it replaced.

Whether this is true or not is somewhat a matter of opinion; I do note that RFC 788 is 61

pages long, while the earlier RFC 780 was only 43 pages. What SMTP definitely has over

MTP is elegance; the protocol is designed specifically for the transport of electronic mail.

While it retains certain similarities to FTP, it is an “independent” protocol running over TCP.

So, from a conceptual standpoint, it can be considered simpler than MTP. In terms of

mechanics, the process SMTP uses to transfer an e-mail message is indeed rather simple,

especially compared to some other protocols.

RFC 788 described the operation of SMTP carrying e-mail messages corresponding to the

ARPAnet text message standard as described in RFC 733. Development of both e-mail

messages and the SMTP protocol continued, of course. In August 1982, a milestone in

TCP/IP e-mail was achieved when RFCs 821 and 822 were published. RFC 821 revised

SMTP, and became the defining standard for the protocol for the next two decades. RFC

822, its companion standard, became the standard for TCP/IP electronic mail messages

carried by SMTP.

Key Concept: The most important component of the TCP/IP electronic mail system

is the Simple Mail Transfer Protocol (SMTP). SMTP was derived from the earlier Mail

Transfer Protocol (MTP), and is the mechanism used for the delivery of mail between

TCP/IP systems and users. The only part of the e-mail system for which SMTP is not used

is the final retrieval step by an e-mail recipient.

SMTP Extensions and Revisions

As the 1980s progressed and TCP/IP and the Internet both grew in popularity, SMTP

gradually overtook other methods to become the dominant method of e-mail message

delivery. For a number of years, the protocol was used mostly “as is”, with no new RFCs

published to define new versions or formally change its behavior.

This changed in February 1993, when RFC 1425, SMTP Service Extensions

, was

published. As the name suggests, this standard describes a process for adding new

capabilities to extend how SMTP works, while maintaining backward-compatibility with

existing systems. SMTP with these extensions is sometimes called Extended SMTP or

ESMTP (though use of this term seems to be not entirely universal). As development of

SMTP continued, RFC 1425 was revised in RFC 1651 in July 1994 and then RFC 1869 in

November 1995. Along with these, a number of other RFCs defining particular SMTP exten-

sions such as pipelining and message size declaration were defined.

In April 2001, another major milestone in TCP/IP e-mail was reached when revisions of

RFC 821 and RFC 822 were published, as RFCs 2821 and 2822 respectively. Both

documents are “consolidations” of updates and changes that had been made to RFCs 821

and 822 between 1982 and 2001. And no, I don't think it is a coincidence that the old and

new RFC numbers are exactly “2000” apart. RFCs 2820 and 2823 were both published in

May 2000, so it looks like 2821 and 2822 were reserved for the e-mail standards. I think this

was a great idea, as it makes more clear that the new RFCs are revisions of the old ones.

RFC 2821 is the current base standard for SMTP. It incorporates the base protocol

description from RFC 821, and the latest SMTP extensions as defined in RFC 1869.

Perhaps more importantly, it updates the description of the e-mail communication model to

reflect the realities of modern TCP/IP networks, especially the e-mail features built into the

Domain Name System (DNS). We'll examine this in more detail in the next topic.

SMTP Communication and Message Transport Methods, Client/Server Roles

and Terminology

The TCP/IP electronic mail communication model describes the way e-mail messages are

conveyed from the sender to the recipient. In most cases, this involves the sender's client

machine sending the e-mail to its local SMTP server, which in turn sends it to the recipient's

local SMTP server, and finally to the recipient's local host. All of these steps except for the

last one are performed by SMTP. In fact, the overall e-mail communication model is largely

described by the RFC 821 and 2821 SMTP standards.

The initial communication takes place between the sender's client machine and a local

SMTP server that the sender is allowed to access. After submission of the e-mail message,

that SMTP server becomes responsible for delivering the message to the SMTP server

responsible for the recipient's mailbox. There are two different ways that this can happen.

Early E-Mail Communication Using Relaying

In the first years of electronic mail, when RFC 821 and its predecessors were initially

defined, the Internet was very different than it is today. There was no Domain Name

System, and this made electronic mail delivery complex, because there was no way to map

a mailbox address to the IP address of the SMTP server that managed that mailbox. Also,

there were many proprietary networks connected to the Internet, which meant that it was no

always possible for any given system to communicate with any other.

Given this, how could e-mail be delivered? The most common way in the early days of

SMTP was through a process called relaying. SMTP routing information was included along

with the e-mail address, to specify a sequence of SMTP servers that the mail should be

relayed through to get to its destination. For example, if a sender using SMTP server A

wanted to send e-mail to someone whose mailbox was on SMTP server Z, they might have

needed to specify that the mail be sent through intermediate SMTP servers at sites D, P

and U to get there. An SMTP connection would be established from A to D to send the

message on one leg of its journey; then it would go from D to P, P to U and then U to Z. The

process is analogous to how IP routing works, but at the application layer (actually using IP

routing at a lower level, of course.)

You can probably see the problems with this quite easily: it's cumbersome, requires many

devices to “handle” the mail, results in delays in communication, and also requires the

communication of source routes between SMTP servers. It was certainly functional, but far

from ideal.

Modern E-Mail Communication Using DNS and Direct Delivery

The creation of DNS radically changed how e-mail delivery worked. DNS includes support

for a special mail exchanger (MX) record that allows easy mapping from the domain name

in an e-mail address to the IP address of the SMTP server that handles mail for that

domain. I explain this in the topic on the regular e-mail address format, as well as the

dedicated topic on DNS e-mail support.

In the new system, SMTP communication is much simpler and more direct. The sending

SMTP server uses DNS to find the MX record of the domain to which the e-mail is

addressed. This gives the sender the DNS name of the recipient's SMTP server. This is

resolved to an IP address, and a connection can be made directly from the sender's SMTP

server to the recipient's to deliver the e-mail. While SMTP still supports relaying, direct e-

mail delivery using MX records is faster and more efficient, and RFC 2821 makes clear that

this is now the preferred method.

In this new system, SMTP is generally only used for two transfers: first, from the sender's

client machine to the sender's local SMTP server, and then from that server to the

recipient's local SMTP server, as shown in Figure 301. (A distinct mail access protocol or

method is used by the recipient for the last leg of the journey.) Each transfer of an e-mail

message between SMTP servers involves the establishment of a TCP connection and then

the transfer of the e-mail headers and body using the SMTP mail transfer process. The next

two topics describe in detail how this occurs.

Key Concept: In the early days of SMTP, mail was delivered using the relatively

inefficient process of relaying from server to server across the internetwork. Today,

when an SMTP server has mail to deliver to a user, it determines the server that

handles the user’s mail using the Domain Name System (DNS) and sends the mail to that

server directly.

SMTP Terminology: Client/Server and Sender/Receiver

The original RFC 821 standard referred to the device that initiates an SMTP e-mail transfer

as the sender and the one that responds to it as the receiver. These terms were changed to

client and server in RFC 2821 to “reflect current industry terminology”. Strictly speaking, this

is correct, but in some ways the more current terminology is significantly less clear.

As I explained in the general discussion of TCP/IP client/server operation, the terms “client”

and “server” are used in many different sense in networking, and this often leads to

confusion. In common parlance, the computers that handle e-mail on the Internet are

usually all called SMTP servers. This is for two reasons. First, they run SMTP server

software to provide SMTP services to client machines, such as end-user PCs. Second,

these devices are usually dedicated hardware servers running in network centers, typically

managed by Internet Service Providers.

However, the terms “client” and “server” are now used to refer to the roles in a particular

SMTP communication as well. Since all SMTP servers both send and receive e-mail, they

all act as both clients and servers transactionally at different times. An SMTP server that is

relaying an e-mail will act as both for that message, receiving it as a server and then

sending it to the next server as a client.

Adding to this potential confusion is the fact that the initial stage in sending an e-mail is from

the sender's client machine to the sender's local SMTP server. Thus, the client role in an

SMTP transaction may not be an actual SMTP server, but the server role will always be a

server.

Confused yet? ☺ Me too. For all of these reasons, the old terms “sender” and “receiver” are

still used in places in RFC 2821, where needed for clarity. I consider them much more

straight-forward and use them in the topics that follow.

Key Concept: SMTP servers both send and receive e-mail; the device sending mail

acts as a client for that transaction; the one receiving it acts as a server. To avoid

confusion, it is easier to refer to the device sending e-mail as the SMTP sender and

the one receiving as the SMTP receiver; these were the terms used when SMTP was origi-

nally created.

SMTP Connection and Session Establishment and Termination

The delivery of electronic mail using the Simple Mail Transfer Protocol (SMTP) involves the

regular exchange of e-mail messages between SMTP servers. SMTP servers are respon-

sible for sending e-mail that users of the server submit for delivery. They also receive e-mail

either intended for local recipients, or in some cases for forwarding or relaying to other

servers.

Overview of Connection Establishment and Termination

All SMTP communication is done using the TCP. This allows SMTP servers to make use of

TCP's many features that ensure efficient and reliable communication. SMTP servers

generally must be kept running and connected to the Internet 24 hours a day, seven days a

week, to ensure that mail can be delivered at any time. (This is a big reason why most end-

users employ access protocols such as POP3 to access their received e-mail rather than

running their own SMTP servers.) The server listens continuously on the SMTP server port,

well-known port number 25, for any TCP connection requests from other SMTP servers.

As explained in the previous topic, an SMTP server that wishes to send e-mail normally

begins with a DNS lookup of the MX record corresponding to the domain name of the

intended recipient's e-mail address, to get the name of the appropriate SMTP server. This

name is then resolved to an IP address; for efficiency, this IP address is often included as

an Additional record in the response to the MX request, to save the sending server from

needing to perform two explicit DNS resolutions.

The SMTP sender then establishes a SMTP session with the SMTP receiver. Once the

session is established, mail transactions can be performed, to allow mail to be sent

between the devices. When the SMTP sender is done, it terminates the connection. All of

these processes involve specific exchanges of commands and replies, which are illustrated

in Figure 304.

Connection Establishment and Greeting Exchange

Let’s take a look at these processes in more detail, starting of course with SMTP session

establishment. The SMTP sender begins by initiating a TCP connection to the SMTP

receiver. The sending SMTP server uses an ephemeral port number, since it is playing the

role of the client in the transaction. Assuming that the server is willing to accept a

connection, it will indicate that it is ready to receive instructions from the client by sending

reply code 220. This is called the “greeting” or “service ready” response. It commonly

includes the full domain name of the server machine, the version of the SMTP server

software it is running, and possibly other information.

Now, it would be rude for the server acting as a client to just start sending commands to the

responding server without saying hello first, wouldn't it? So that's exactly what comes next:

the client says “hello”. In the original SMTP protocol, this is done by issuing a HELO

command, which includes the domain name of the sending (client) SMTP server as a

courtesy. The receiving device then responds back with a return “hello” message using an

SMTP reply code 250.

For example, if the SMTP server “smtp.sendersite.org” was making a connection to the

SMTP server “mail.receiversplace.com”, it would say:

HELO smtp.sendersite.org

After receiving this, “mail.receiversplace.com” would respond back with a “hello” message

of its own, something like this:

250 mail.receiversplace.com Hello smtp.sendersite.org, nice to meet you.

The “chatty” text is of course purely optional; most of the time SMTP communication is

between software programs, so all the pleasantries are mostly just programmers having a

sense of humor. Still, isn't such politeness a pleasant thing to see in this sometimes difficult

world of ours? ☺

Connection Establishment Using SMTP Extensions

The SMTP extensions defined at first in RFC 1425 and then in subsequent standards up to

RFC 2821 define an alternative “hello” message for the client to use: EHLO (extended

hello). An SMTP sender supporting SMTP extensions (and most do) uses EHLO instead of

HELO in response to the 220 greeting. This serves both to say “hello” to the SMTP receiver,

and to tell it that the sender supports SMTP extensions.

Figure 304: SMTP Transaction Session Establishment and Termination

An SMTP session begins with the SMTP sender establishing a TCP connection to the SMTP receiver. The

receiver sends a ready message; the sender sends a HELO or EHLO command, to which the receiver

responds. Assuming no difficulties, the session is established and mail transactions take place. When the

sender is done, it sends a QUIT command; the receiver responds with a 221 reply and closes the session.

SMTP ReceiverSMTP Sender

(TCP)

1. Establish TCP

Connection To Receiver

2. Establish TCP Connection,

Send 220 "Ready" Reply

4. Receive EHLO, Send 250

"OK" Reply With List Of

Supported SMTP Extensions

3. Receive "Ready" Reply,

Send EHLO Command

220

EHLO

5. Receive "OK" Reply,

Process Acceptable

Extensions; Connection Open

(Mail Transactions)

250

(Extension List)

...

i. Done With Mail Transfers,

Send QUIT Command

QUIT

221

ii. Receive QUIT, Send 221

"Goodbye" Reply, Close

Transmission Channel

iii. Receive "Goodbye" Reply,

Close Transmission Channel

If the SMTP receiver supports the extensions, it replies back with the usual 250 reply, as

well as a series of extra 250 responses. Each of these lists an EHLO keyword that indicates

a particular SMTP extension the receiver supports. If the receiving server doesn't support

the extensions, it will reject the EHLO command with a 500 reply code (“syntax error,

command not recognized”). This tells the SMTP sender that it cannot use extensions; it will

then either issue a conventional HELO command, or QUIT the connection if it requires the

SMTP extensions to be present. (In practice, it is rare for a server to require the use of

SMTP extensions.)

Here's the same example as above, but using EHLO. The sender says:

EHLO smtp.sendersite.org

Assuming “mail.receiversplace.com” supports the SMTP extensions, a typical reply might

be:

250-mail.receiversplace.com Hello smtp.sendersite.org, nice to meet you.

250-SIZE

250-DSN

250 PIPELINING

Each of these additional replies identifies a particular SMTP extension supported by

“mail.receiversplace.com”; in this case, message size declaration (“SIZE”), delivery status

notification (“DSN”) and command pipelining. (The dashes after the “250” indicate a

multiple-line response to a command.)

Once the HELO or EHLO command has been sent and the receiving device has

responded, the session is initiated. Further commands can be sent by the sending SMTP

server to the responding server. These usually take the form of e-mail message transfer

transactions using the process described in the following topic, and other command/reply

exchanges as needed.

Connection Termination

When the sending device is finished sending all the e-mail it has to transfer to the receiving

device, and done with all other activities, it terminates the session by issuing the QUIT

command. This normally results in a 221 “goodbye” message from the SMTP receiver,

saying something like “closing transmission channel”. The TCP connection is then

terminated.

A server may also terminate prematurely in special cases. If it is given a local command to

shut down (for example, due to imminent rebooting of the hardware server upon which it is

running), it may respond to any routine command with a 421 response (“Service not

available, closing transmission channel”). A server is not supposed to terminate a session

simply due to receipt of an invalid command, however, only in special cases where session

termination cannot be avoided.

Key Concept: An SMTP session consists of three basic phases. The session is first

established through the creation of a TCP connection and the exchange of identity

information between the SMTP sender and receiver using the HELO command.

Once established, mail transactions can be performed. When the SMTP sender is done

with the session, it terminates it using the QUIT command. If SMTP extensions are

supported, the SMTP sender uses the EHLO (extended hello) command instead of HELO,

and the SMTP receiver replies with a list of extensions it will allow the SMTP sender to use.

SMTP Mail Transaction Process

The delivery of e-mail message begins with the establishment of an SMTP session between

the devices sending and receiving the message. The SMTP sender initiates a TCP

connection to the SMTP receiver, and then sends a HELO or EHLO command, to which the

receiver responds. Assuming there are no problems, the session is then established and

ready for actual e-mail message transactions.

SMTP Mail Transaction Overview

The SMTP mail transaction process itself consists of three steps:

1. Transaction Initiation and Sender Identification: The SMTP sender tells the SMTP

receiver that it wants to start sending a message, and gives the receiver the e-mail

address of the message's originator.

2. Recipient Identification: The sender tells the receiver the e-mail address(es) of the

intended recipients of the message.

3. Mail Transfer: The sender transfers the e-mail message to the receiver. This is a

complete e-mail message meeting the RFC 822 specification (which may be in MIME

format as well).

That's it! So you can see that the word “Simple” in “Simple Mail Transfer Protocol” definitely

has at least some merit. Especially when compared with other protocols that claim to be

simple, such as SNMP. ☺

The Rationale for A Separate E-Mail Message and Envelope

In fact, one question that sometimes comes up when examining SMTP is why couldn't this

process be even simpler? The first two steps identify the sender of the e-mail and the

intended recipient(s). But all of this information is already contained in headers in the

message itself. Why doesn't SMTP just read that information from the message, which

would in fact make the mail transaction a one-step process?

The explanation for this isn't specifically addressed in the SMTP standards, but I believe

there are several reasons: