Charles M. Kozierok The TCP-IP Guide

Подождите немного. Документ загружается.

Reasons Why Long URLs are Common

The point that many elements in URL syntax are optional is important. The majority of the

time, most of these optional parts are omitted, which makes URLs much simpler in practical

use than they are “on paper”. For example, even though an HTTP URL theoretically

contains a user name, password, host, port, path, query and bookmark, most URLs use

only a host name and a path. This is what helps keep URLs short and easy to use.

Despite this, you will still find some rather long URLs used on the Internet. Here are some of

the most common reasons why.

Long DNS Domain and Host Names

Some people don't realize that long host names are hard to remember. If you run the “Super

Auto Body Shop & Pizza Parlor”, having a Web site called “www.superauto.com” will make

it easier for your customers to find you than trying to register “www.superautobodyshopand-

pizza.com”. Yet DNS names of 15, 20 or even more characters are surprisingly common.

Long Document or Directory Names

Similarly, short file names are better than long ones, and again, many people don't think

about this before putting files on the Internet, which makes things more difficult for those

who must access them.

Use of “Unsafe” Characters

URLs have a mechanism for dealing with “unsafe” characters, but it makes them longer and

harder to decipher. Suppose you have a file named:

“{ABC Corp} budget; draft #3; third quarter 2004.htm”

The URL for this will have to be:

“%7BABC%20Corp%7D%20budget%3B%20draft%20%233%3B%20third%20q

uarter%202004.htm”

The original long file name was readable, but the URL is a mess because of the special

character encodings.

Naming the file “ABC budget draft 3, 3Q2004.htm” would be a better choice, and still

includes enough information to be understandable. Even better, you could replace the

spaces with underscores, to avoid the need for “%20”s entirely: “ABC_budget_draft

3,_3Q2004.htm”.

Parameter Strings

In HTTP URLs, the syntax for specifying a query (following a question mark character) is

often used to allow a Web browser to send various types of information to a Web server,

especially parameters for interactive queries. These parameter strings can get quite

lengthy. For example, I typed in a query to the great Web search engine Google to find

recipes for potato salad. This is what the URL for one of the recipe files looks like:

“http://groups.google.com/groups?q=%22potato+salad%22&hl=en&lr=&ie=UTF-

8&safe=off&selm=B826FB57.89C0%25sbrooks%40ev1.net&rnum=2”.

Almost all of that consists of parameters that tell the Google server exactly what document I

want based on my query. It is necessary, but still cumbersome.

URL Wrapping and Delimiting Issues

For humans, long and complex URLs are hard to remember and use. In addition to the

sheer difficulty of remembering all those characters, there is the issue of URL wrapping,

which occurs when they are presented in certain forms. Most programs can only display 78

or 80 characters in a single line. If a URL is longer than this, the characters of the URL will

“wrap” onto multiple lines; when you read that Google example just above, you probably

noticed that.

URL wrapping can lead to mistakes when copying a URL from one form to another, such as

if you copied it from this document into your Web browser. If a URL is 81 characters long

and 80 are on the first line and the last character on the second line, many users may not

realize that the URL has “wrapped” at all. I have seen URLs that are hundreds of characters

long, requiring several manual “copy and paste” operations to get the URL to work.

Perhaps surprisingly, some software may not handle this “wrapping” properly either. While

this is not a problem when a hyperlink is used in something like an HTML document, it can

be troublesome when links are included in an e-mail message or Usenet article.

Another issue is delimiting where a URL starts and ends when it appears. A URL begins

with a scheme name that could in theory be used in other contexts that are not URLs.

Without a clear way of labeling a URL as being a URL, a software program might not

recognize it. Consider discussion of a URL in a document like this one; if I say “Please visit

http://www.thissite.com; you will see the information you need there”, then we all know the

semicolon is part of the sentence and not part of the URL, but a computer program might

not be so sure. And again, this problem is worse when a URL is long and complex and

wraps on to multiple lines of text?—how does the program recognize the end of the URL?

Explicit URL Delimiting and URL Redirectors

To resolve both the wrapping and delimiting problems, a special URL “super-syntax” is

sometimes employed, especially when URLs are used in other text. This is done by

surrounding the URL in angle brackets, possibly including the label “URL:”. before the

scheme name. For example, all of the following are equivalent:

http://www.networkingistoodarnedcomplicated.com

<http://www.networkingistoodarnedcomplicated.com>

<URL:http://www.networkingistoodarnedcomplicated.com>

The angle brackets indicate clearly where the URL begins and ends, making it easier for

both programs and humans to deal with long URLs.

Another solution sometimes used for long URLs are redirection services, provided by many

Web sites. For example, http://www.tinyurl.com is a free service that allows someone to

create a short URL that automatically loads a resource at a much longer URL.

URL Abbreviation

One final issue I want to discuss isn't related directly to long or complex URLs, but is related

indirectly to the matter of length: URL abbreviation. Many people use URLs so often that

they become, well, lazy when it comes to specifying URLs. They tend to leave off portions

of the full URL syntax to save time and energy. I don't mean by this that they specify relative

URLs, but rather, they specify absolute URLs with “missing pieces”.

For example, rather than type “http://www.sitename.com”, they might type

“http:www.sitename.com”, leaving off the two slashes. More commonly, people omit the

scheme name entirely, just entering “www.sitename.com”. Technically, this is not a URL, it

is just a domain name. However, most Web browsers can handle this, assuming by default

that the scheme is “http://” if none is provided. (Don’t feel bad, I do it too! ☺)

URL Obscuration, Obfuscation and General Trickery

Most of the time, the owner of a resource wants the URL that refers to the resource to be

short, simple and easily-understood. Thus, long and complex URLs are usually the result of

necessity, accident or ignorance. Some resources need to have long names for whatever

reason, such as the use of the long query string in the Google example; other times, URLs

are made long because the owner of the resource doesn't realize that using a long DNS

host name or file name will make for a long and unwieldy URL.

Whatever the reasons for these situations, they are not deliberate. Recent years, however,

have seen a dramatic rise in the use of intentionally long, complex, confusing and deliber-

ately deceptive URLs. These URLs are either structured so that it is impossible to tell what

they are, or worse, they are made to appear as if they point to one resource when they

really go to another.

Why would people do this? Simple: because they fear being open and honest about their

“resources”. And who would these people be? Why, they would be the spammers and con

artists who overload our Internet e-mail boxes with offers of every sort imaginable, from

making you rich beyond your wildest dreams, to inflating the dimensions of certain body

parts to unnatural sizes…

They are afraid that if the URL indicated clearly what the “resource” was, you might not click

on the link; or that if you identify them as a spammer you might filter out their e-mail. They

also figure that if they can make the URL appear to be something interesting, you'll load it.

Even if it turns out to be something you didn't expect, maybe you'll pay attention anyway.

(You mean you are too smart to be tricked into buying a product through a deceptive URL?

And you would never support a spammer anyway? What a coincidence, same with me! Yet

the spam keeps coming; it must work or they wouldn't keep doing it… would they? ☺)

Common Obscuration and Obfuscation Techniques

It is a cruel irony that the complex syntax that was built into URLs to allow them to be so

flexible, has been exploited by the obnoxious into tricking people. They know that most

people are used to seeing simple URLs like “http://www.myfavoritesite.com” and do not

realize that the full URL syntax allows the same resource to be specified in literally millions

of different ways.

So these people, desperate for hits to their Web sites at any cost, keep coming up with new

tricks for manipulating URLs. These are focused on HTTP scheme URLs, though in theory

the tricks can be applied to several other types as well (though obviously they won't work

with some schemes). Here are some of the more common gimmicks that have been used,

past and present (note that if you are trying these out as you read, some examples may not

work on certain browsers):

Excessive Length

In some cases, a URL is just made really long by the addition of lots of gibberish as a “query

string”, so that the user's eyes glaze over just looking at it. This is a relatively “unsophisti-

cated” technique, however, since one can easily tell what the real host name is by looking at

the start of the URL. Most of the better scammers have moved beyond such simple tricks

today.

Regular IP Address Hosts

Internet users are so accustomed to using DNS names that they don't realize that one can

access a URL using an IP address at all. So most people don't realize that The PC Guide

can be accessed as easily using <http://209.68.14.80> as <http://www.PCGuide.com>.

(Note that this is not true of all Internet hosts; those that use virtual names cannot be

accessed using just an IP address.)

This is not really trickery per se. It is quite legitimate, and in some ways even necessary—

for example, for accessing a site that is having DNS problems. The problem here is that

usually one cannot tell what a site is from the IP address alone, and many people will just

click on an IP address link without bothering to find out what it is.

Numeric Domain Names

It is possible to register a DNS domain name consisting of just a single number. For

example, one could register “114.com”. And then one could create subdomains within it

such as “42.12.205.114.com”. At first glance this appears to be an IP address specification,

so someone might think it would resolve to the address 42.12.205.114—when in reality, it is

some other address.

I believe that DNS name registrars have been cracking down on this sort of trickery, so it

may not be as prevalent now as it once was.

Bogus Authentication Information

HTTP URLs theoretically support the inclusion of authentication information, by including

“<user>:<password>@” before the host in the URL. Yet the vast majority of Web sites are

“open” and neither require nor use it. If you specify an authentication string and it is not

needed, it is ignored.

This is one of the most popular techniques at present. One way it is used is by including

“authentication information” that looks like a “benign” host, to make the user think the URL

is for that host. For example, if I wanted to trick you into visiting The PC Guide, I might use

this URL to make it look like clicking it would go to CNN:

<http://www.cnn.com@www.PCGuide.com>

This is still too obvious, however, so this “method” is often combined with some of the

techniques below.

Deceptive Character Encoding

The use of the percent sign to encode special characters such as spaces and punctuation

can also be abused to obscure the name of a domain. For example, the following is

another way of expressing the DNS name for The PC Guide:

<http://%57%57%57.%50%43%47%55%49%44%45.%43%4F%4D>

Try it. ☺

IP Address Math Trickery

Okay, this is where things get really bizarre. Most of the time, we express an IP address as

a dotted decimal number. Remember, however, that to computers, the IP address is just a

32-bit binary number. Most browsers support a rather shocking number of methods for

expressing these numbers. This is unfortunate, because this flexibility is really not needed

and almost never used for legitimate purposes. It can lead to some really bizarre URLs that

are unrecognizable, or that look like regular IP addresses but are not.

Here are some examples, all of which are the same as the IP address form of The PC

Guide (<http://209.68.14.80>):

☯ IP Address in Dotted Octal: A leading zero signifies an IP address where each byte

is in octal:

<http://0321.0104.016.0120>

☯ IP Address in Dotted Hexadecimal: A leading zero followed by an “x” signifies an IP

address where each byte is in hexadecimal:

<http://0xD1.0x44.0x0E.0x50>.

☯ IP Address As Single Number: We can even take the entire 32-bit number and

express it as a single number and that will work too. In decimal:

<http://3510898256/>

In octal:

<http://032121007120/>

And, in hexadecimal:

<http://0xd1440e50/>.

Making Matters Worse: Combining Deceptive Tricks

As if these tricks weren't bad enough taken individually, we can have some real fun by

combining them! For example, start with the regular PC Guide URL:

<http://www.PCGuide.com>

And convert it to IP:

<http://209.68.14.80>

Then add some bogus authentication gibberish:

<http://www.cnn.com@209.68.14.80>

And convert the real URL into a single number so it looks like a document on the CNN web

site:

<http://www.cnn.com@3510898256>

Alternately, we can use the octal form, and even include lots of extra leading zeroes just for

fun:

<http://www.cnn.com@0000000000000321.00000000104.00000000000016.00000120>

Believe it or not, this is just the tip of the iceberg. In some browsers, even the IP address

numbers can be expressed using “percent sign” ASCII encoding!

While quite irritating, I must give these people points for creativity at least—some of the

tricks are quite ingenious. At the same time, their inventiveness is potentially hazardous.

While these false URLs are usually more a waste of time than anything harmful, there are

sometimes good reasons a person would go to great lengths to hide the identity of a

resource. Deceptive URLs are just one more danger that network administrators must deal

with today.

Key Concept: The syntax of Internet URLs includes many elements that provide

great flexibility in how URLs can be constructed. Unfortunately, these capabilities of

expression are now often abused by people who create intentionally obfuscated

URLs to trick users into accessing their Web sites and other resources. Some of these can

be potentially hazardous, which means that care is required before clicking unknown links

or accessing strange URLs.

Uniform Resource Names (URNs)

“HTTP 404 - NOT FOUND”

Have you ever tried to access a Web site or other Internet resource, only to see those

dreaded words appear? You probably have, and in seeing them, you have experienced

first-hand one of the most common problems with Uniform Resource Locators (URLs).

URLs specify a resource using two key pieces of information: the resource's location, and a

method by which the resource may be accessed or retrieved. This focus on the means of

access for the resource makes URLs very practical, in that the URL usually contains all the

data we need to use the resource. This is why URLs are so widely used today. However,

this “access orientation” also means that URLs have a number of serious limitations.

The Problem With URLs

The main difficulty with URLs is that since they describe a resource based on its location,

they tie the resource and its location together inextricably. While this may not seem to be a

big deal, it is actually a fairly serious matter in a number of ways, because a resource and

its location are not the same thing. It is only because most Internet resources rarely change

location that we don't notice this issue more often with URLs.

Suppose that your name is Joe Xavier Zachariah and you live at 44 Glendale Crescent in

Sydney, Australia. If someone asked you who you were, would you say “Joe Xavier

Zachariah”, or “the man living at 44 Glendale Crescent in Sydney, Australia”? Almost

certainly, the former; but a URL would be like describing yourself as a “resource” using the

latter description.

Since we realize that Mr. Zachariah is obviously not always going to be at 44 Glendale

Crescent, we know that describing him using just a location is not sufficient. The same thing

occurs with Internet resources when they are identified using only location.

However, the problem with Internet resources and URLs goes beyond just the matter of

movement. Consider a situation where a particular resource is very popular; we might want

to duplicate the same resource in multiple locations. Using URLs, we would need a different

identifier for each copy of the resource, even though each copy is the same. Again, the

problem is that we are not identifying the resource itself but rather the place where it can be

found.

Overview of URNs

In recognition of this issue, an alternative identification mechanism for Internet resources

was developed, called Uniform Resource Names (URNs). The basic standard describing

URNs is RFC 1737, Functional Requirements for Uniform Resource Names

, which was

published in 1994. In 1997, RFC 2141 was published, which specifies the syntax of URNs.

As you can probably tell from that term, a URN is intended to label a resource based on its

actual identity, rather than where it can be found. So, where a URL is like Joe Zachariah's

address, a URN would be his name. Or, as I gave as an example in the overview of URIs, a

URN would be identifying a book based on its ISBN number rather than specifying what

bookshelf it is on in a building.

To be useful in identifying a particular resource, it is necessary that a URN be globally

unique, and that's not always as simple as it may at first appear. Consider human names,

for example. Even though there is probably only one Charles Marlin Kozierok in the entire

world, if your name is John Paul Smith or José Garcia, you likely share that name with

thousands of others. This means using common names may not be sufficient for identifying

human “resources” and some other method might need to be devised.

URN Namespaces and Syntax

There are many types of resources that URNs are intended to identify on the Internet, each

of which may require a different form of naming. To allow URNs to represent many kinds of

resources, numerous URN namespaces are defined. A namespace is referenced using a

unique string that tells the person or computer interpreting the URN what type of resource

the URN identifies. The namespace also ensures the uniqueness of URNs, when a

particular identifier might exist in more than one context. For example, both North American

telephone numbers and ISBN numbers consist of ten digits, so a particular number such as

“4167819249” could represent both a telephone number and a book number; the

namespace identifier tells us what the number means when it is encountered in a URN.

The general syntax of a URN is:

URN:<namespace-ID>:<resource-identifier>

For example, a book with the ISBN number 0-679-73669-7 could be represented as:

URN:isbn:0-679-73669-7

This string identifies that particular book uniquely, wherever it might happen to be in the

world. Many other namespaces have also been defined to specify the URNs for other types

of resources, such as documents on the Internet.

Key Concept: Where Uniform Resource Locators (URLs) specify a resource based

on an access method and location, Uniform Resource Names (URNs) identify a

resource by name. A URN consists of a namespace identifier, which indicates what

type of name it contains, and a resource identifier that specifies the individual resource

within the context of that namespace.

URN Resolution and Implementation Difficulties

URNs are a more “natural” way of identifying resources, which gives them intuitive appeal.

Despite this, URNs are still not widely used, even though they have been in development

for over a decade. The main reason for this is somewhat ironic: it is in fact because of the

fact that URNs are independent of location. The very characteristic that provides URNs with

identification advantages over URLs also makes URNs much harder to use practically,

which has led to long delays in workable URN systems.

To understand the problem, consider the example string “URN:isbn:0-679-73669-7”. This

uniquely identifies a particular book, and will always refer to it no matter where the book

may be, unlike a URL. The problem is that while the URL-equivalent tells us how to actually

find this book, the URN does not. The same thing goes for our human example before:

identifying Joe Xavier Zachariah by his name is more “sensible” than identifying him as “the

man living at 44 Glendale Crescent in Sydney, Australia”, but at least with the latter, we

know where Joe is!

In order for URNs to be useful on an internetwork, they require an additional mechanism for

translating a simple URN identification string into a particular location and/or access

method. In other words, we need to be able to change a URN into the equivalent of a URL,

so that the resource can be found. This requirement is analogous to the problem of

resolving Internet DNS domain names into IP addresses, and the same term is used to

describe it: URN resolution.

Ideally, we want to be able to use some sort of technique where we specify the name “Joe

Xavier Zachariah” and we are told where Joe is so we can find him. Or, we provide the

string “URN:isbn:0-679-73669-7” and are provided with a list of libraries or other places

where the book can be found. The power of URNs can also be exploited in such a system,

by having the resolution system specify the location of a copy of the resource that is closest

(in terms of network distance, cost or other measurements) to the entity making the request.

However, setting up URN resolution mechanisms is a non-trivial task. The matter of URN

resolution has been the subject of much of the work on URNs over the last decade. RFC

2483, URI Resolution Services Necessary for URN Resolution

, was published in 1999 and

discusses some of the important issues in URN resolution. In October 2002, a series of

RFCs, 3401 to 3405, defined a new system called the Dynamic Delegation Discovery

System (DDDS) that was designed not just to resolve URNs, but to handle the entire class

of resolution problems where an identifier is given and the output is information about

where to get more information about that identifier. RFC 3406 was published at the same

time, providing more information about URN namespaces.