Why should I use urlencode?

Question

I am writing a web application and learning how to urlencode html links...

All the urlencode questions here (see tag below) are "How to...?" questions.

My question is not "How?" but "Why?".

Even the wikipedia article only addresses the mechanics of it:
http://en.wikipedia.org/wiki/Urlencode but not why I should use urlencode in my application at all.

What are the security implications of using (or rather not using) urlencode?

How can a failure to use urlencode be exploited?

What kind of bugs or failures can crop up with unencoded urls?

I'm asking because even without urlencode, a link to my application dev web site like the following works as expected: http://myapp/my%20test/ée/ràé

Why should I use urlencode?

Or another way to put it:

When should I use urlencode? In what kind of situations?

Answer 1

The main reason is it essentially escapes characters to be included in the URL of your webpage.

Suppose a user inputs a user form field as "&joe" and we would like to redirect to a page which contains that name as part of the URL, using URL encoding, it would then be, for example:

localhost/index.php?name=%26joe //note how the ampersand is escaped

If you didnt use urlencoding, you would end up with:

localhost/index.php?name=&joe

and that ampersand would cause all sorts of unpredictability

Answer 2

Update: There is an even better explanation (imo) further above:

A URI is represented as a sequence of characters, not as a sequence of octets. That is because URI might be "transported" by means that are not through a computer network, e.g., printed on paper, read over the radio, etc.

and

For original character sequences that contain non-ASCII characters, however, the situation is more difficult. Internet protocols that transmit octet sequences intended to represent character sequences are expected to provide some way of identifying the charset used, if there might be more than one [RFC2277]. However, there is currently no provision within the generic URI syntax to accomplish this identification. An individual URI scheme may require a single charset, define a default charset, or provide a way to indicate the charset used.

---

Because it is stated in the RFC:

2.4. Escape Sequences

Data must be escaped if it does not have a representation using an unreserved character; this includes data that does not correspond to a printable character of the US-ASCII coded character set, or that corresponds to any US-ASCII character that is disallowed, as explained below.

and

2.4.2. When to Escape and Unescape

A URI is always in an "escaped" form, since escaping or unescaping a completed URI might change its semantics. Normally, the only time escape encodings can safely be made is when the URI is being created from its component parts; each component may have its own set of characters that are reserved, so only the mechanism responsible for generating or interpreting that component can determine whether or not escaping a character will change its semantics. Likewise, a URI must be separated into its components before the escaped characters within those components can be safely decoded.

In some cases, data that could be represented by an unreserved character may appear escaped; for example, some of the unreserved "mark" characters are automatically escaped by some systems. If the given URI scheme defines a canonicalization algorithm, then unreserved characters may be unescaped according to that algorithm. For example, "%7e" is sometimes used instead of "~" in an http URL path, but the two are equivalent for an http URL.

Because the percent "%" character always has the reserved purpose of being the escape indicator, it must be escaped as "%25" in order to be used as data within a URI. Implementers should be careful not to escape or unescape the same string more than once, since unescaping an already unescaped string might lead to misinterpreting a percent data character as another escaped character, or vice versa in the case of escaping an already escaped string.

Answer 3

There're RFCs that define format for URLs, and browser/web server developers rely on this as a standard for interpreting data. If you don't comply, the results may be unpredictable.

HTTP URL has its specification, and it states that practically all non-latin characters need to be encoded.

Answer 4

There are two reasons why you should use URL encoding:

• When you need to pass characters that are invalid for URL, such as „ < > # % \ | ^ [ ] ` spaces. For instance, whitespace is not a valid URL character, since it would be ambiguous to spot the full URL in texts if they would contain whitespaces.
• When you need to pass characters that are reserved for URL, such as ! # $ % & ' ( ) * + , / : ; = ? @ [ ]. For instance, ? is reserved to mark start of query parameters, and if we do not encode ? in the path or inside query parameter, it might break the syntax.

Answer 5

Two reasons I could think of:

• It really depends on how you parse your query server side. E.g. passing parameters using HTTP's GET request will have problems if there are characters like & inside some parameter.
• It allows you to handle non-ansi characters the way you'd like to (you dictate the encoding). Otherwise the browser might pass them in some random encoding (don't think it's really defined in any standard; correct me if I'm wrong).

Answer 6

How will you distinguish if your two of path are like this

http://myapp/my%20test/

and

http://myapp/my test/

Note space & %20 is part of URL.

Answer 7

URL Encoding is the process of converting string into valid URL format. Valid URL format means that the URL contains only what is termed "alpha | digit | safe | extra | escape" characters.

URL encoding is normally performed to convert data passed via html forms, because such data may contain special character, such as "/", ".", "#", and so on, which could either: a) have special meanings; or b) is not a valid character for an URL; or c) could be altered during transfer. For instance, the "#" character needs to be encoded because it has a special meaning of that of an html anchor. The character also needs to be encoded because is not allowed on a valid URL format. Also, some characters, such as "~" might not transport properly across the internet.

Answer 8

It is specified in the web standard RFC 1738.

only alphanumerics, the special characters "$-_.+!*'(),", and reserved characters used for their reserved purposes may be used unencoded within a URL.