\chapter{CGI Programming\label{chap:cgi}}

\section{Introduction}

Up to this point we have been writing Perl scripts that are to be executed on the command line. 
As I pinpointed early in this tutorial, the ability to write CGI programs with 
Perl is the prime motive behind learning Perl for many people. In the following sections we will 
first look at what CGI is, and understand how it allows webmasters to create dynamic content. 
Towards the end, I will introduce security issues associated with CGI scripting. \\

%\p{%
%Command-line Perl scripts have been playing a rather important role %
%on Unix-variant systems. Similar to shell scripts, they both act as the ``glue'' %
%between different components so that these components, though as separate pieces, %
%would be able to collaborate effectively. On Unix, each program is usually very %
%primitive and only carry out a particular task. In order to carry out a more sophisticated %
%task, usually several components are involved and the output of one component would be %
%piped to another component for further processing. It is the shell scripts or Perl %
%script that are responsible for tying these components into a single chain. Therefore, %
%standalone Perl scripts are not uncommon on Unix systems. However, this purpose rarely %
%holds for Windows or Mac systems, so standalone Perl scripts for execution on the command %
%line is scarce. %
%}

\section{Static Content and Dynamic Content}

\subsection{The Hypertext Markup Language}

The earliest Web servers only serve \term{static content}. Content is served in the form of files
that are prepared and made available by the server administrator. 
Static content that are being served by Web servers are primarily in the form of Web pages, or in 
a more technical parlance, HTML files. A document written in the \term{Hypertext Markup Language}, 
or HTML, is actually a plain text document with extra markup added that indicates the logical structure 
of the document. For example, consider the following HTML document: \\

%{\ttfamily
\begin{verbatimtab}[4]
<html>
	<head>
		<title>A Sample HTML Page</title>
	</head>
	<body>
		<p>This is a paragraph.</p>
		<img src="logo.gif" width="80" height="60" alt="">
		<a href="http://www.cbkihong.com">This is a hyperlink</a>
	</body>
</html>
\end{verbatimtab}
%}

This is a very simple HTML document. \texttt{<p>...</p>} denotes the text in between is a paragraph. 
It is then followed by an image of dimensions $80\times60$. At last, we insert a hyperlink that, 
when clicked by a user through a browser window displaying this document, will cause the browser 
to load the resource at the URL ``http://www.cbkihong.com'' into the browser window. This is not 
an HTML tutorial, and if you are not familiar with HTML you should learn it first before proceeding 
with this chapter. But what I would like to demonstrate here is that a markup language serves to differentiate 
different elements present in a document. In this example, the browser, having received and parsed the 
HTML document, finds out that the document consists of a paragraph, an image and a link. Then the browser knows 
how to \term{render} the markup and therefore create the objects specified in the browser window. Because 
a hyperlink has different properties and actions from an image display, the browser needs a way to figure 
out the kinds of objects present in the document --- and that's what HTML is for. \\


\subsection{The World Wide Web}

HTML documents are special as they contain \term{hyperlinks}. Hyperlinks allow readers to jump from one 
document to another document with a \term{Uniform Resource Identifier} (URI). In order for a document to
be accessible on the World Wide Web, it has to be assigned an address. The URI (a more widely used term is
Uniform Resource Locator, URL) is the address in this context. For authors, hyperlinks not 
only make referencing internal or external destinations more convenient, they also bind these separate 
documents together in the form of linkages. Therefore, with a single URI to an HTML document a reader not only can have access 
to the document identified by the URI, but also resources linked to that HTML document. Such linkages 
bind all the linked resources on the Internet into a virtual network, and this is the \term{World Wide Web} 
we are using every day. \\

The World Wide Web utilizes the \term{client-server model}. First we need to establish what a \term{server} and a 
\term{client} is. In everyday language, a server usually refers to a mainframe or other powerful computational 
devices, in contrast to personal computers. However, in Computer Science parlance, server and clients are 
identified by their roles. A server refers to any entities that provides services to clients. In the 
client-server model, a client first initiates a request and address it to the server. Upon receipt of the request 
from the client, the server carries out any necessary actions to fulfill the request, and then return the 
results as a response to the client. Therefore, typical interactions between a client and server in the 
client-server model can be visualized as in Figure~\ref{fig:http}. \\

\image{!hbt}{!}{!}{images/http.eps}{A simplified HTTP client-server interaction}{fig:http}

Note that in the client-server model, server and client do not necessarily refer to any physical devices. 
In previous chapters, you have learned how to construct modules that represent objects, or in the procedural 
approach, represents a set of functions under the same namespace. This situation can fit into the client-server 
model too. The modules can be thought of as providing services to users of these modules. In this case, the modules 
act as servers and a program that uses these modules acts as the client. \\

Have you ever thought about what a Web server is? In fact there is nothing mysterious or complicated. 
It is merely a system with a suitable \term{Web server daemon} installed. A Web server daemon is a small 
program that is executed in the background that handles HTTP requests and responses. A Web browser is actually 
one of the forms of a \term{user agent}. When a user enters a URI in the browser and hit the ``Go'' button, 
the browser, the user agent in this case, sends a request to the Web server concerned encapsulating the command 
of getting the resource located at the requested URI. The daemon captures this request, retrieves the 
specified resource if any and returns the content of which to the client as a response. The browser receives 
the resource. If it is an HTML document, it parses and renders it so that the document is eventually displayed 
in the browser window. Rendering refers to the process of converting HTML into the graphical objects displayed 
in the browser window. The previous figure shown actually illustrates the interaction between the client requesting
the HTML document and the server processing the request. The connections in the figure
are numbered such that you can more easily refer to the explanation below to understand what each message flow 
is for. \\

The interaction between the client and the server is one of the main concerns of this chapter. 
The client and the server may run on vastly different system architectures. For instance, the Web server may be 
running on Solaris while the client on Microsoft Windows 2000. However, a common \term{protocol} defines the 
common language of communication between the two parties so that platform-independent interaction is made possible. 
On the Internet, a single protocol is defined for the World Wide Web, which is the \term{Hypertext Transfer Protocol}, 
with the more widely known abbreviation of HTTP. Enacted by the \term{World Wide Web Consortium}, or W3C, HTTP is an 
open standard that can be freely implemented on any platforms.  \\

Let us briefly outline how documents can be made accessible on the World Wide Web. The administrator of the Web server sets 
aside a directory (or folder) on the server. All the documents that are to be made accessible on the World Wide Web are placed 
in this directory (and subdirectories if any). When the URI is received by the Web server, this address is mapped 
to a location in this directory representing the resource to be retrieved for the client. \\

(1) represents the initial HTTP request to the Web server. Before that, the Web browser has to accomplish
several preliminary tasks. This include transformations of the human-oriented domain name and hostname in the
URI to the IP address necessary for the Internet routing system to deliver the request to the intended Web
server. After the IP address of the Web server is identified, the HTTP request is encapsulated in a \term{packet}
and delivered to the Web server. A packet is the container of the message. Note that an HTTP request is not sent as is. It
is put inside the packet as the payload, and the packet header contains all the necessary information needed to
deliver the packet to the destination. This is analogous to a letter being placed into an envelope before it is
posted. The HTTP request is similar to the letter, and the envelope with the sender's/recipient's addresses is
similar to the packet and its header in this analogy. \\

Having received (1), the Web server will retrieve the document specified. From the file extension, the
Web server recognizes the resource as an HTML document and, therefore, returns the content of the HTML document
and marks it as of type ``text/html''. This is exactly (2). \\

The client receives the packet containing the returned response. Note that a Web server may return content
of types other than HTML. For example, it may be a PDF document or simply some audio clips. Therefore, a means
have to be in place that allows the Web browser to identify the type of the returned content. That explains why
the returned content in (2) have to be marked of type ``text/html''. Upon knowing this is an HTML document, the
browser would parse it and draws the Web page in accordance with the HTML received. Recall that an HTML document
may contain external references (images, audio/video clips, external style sheets or Javascript, Java applets etc.)
that have to be fetched as well in order to display the Web page properly. In the HTML document shown earlier, 
the image ``logo.gif'' is the only external reference that has to be fetched. Therefore, a request for this 
resource is represented by (3). The Web server, on receipt of (3), would return the resource and mark it as of
type ``image/gif'' (4). Note that if the HTML document has multiple external references, additional connections
has to be made by the user agent to request such resources. However, usually the user agent will not fetch such
external resources one by one. Consider a Web page consisting of 30 images. It would be too time consuming to 
request each resource sequentially as the network can be slow. Usually, the browser will send multiple requests
at a time to the Web server by opening multiple \term{threads}. The Web server is also likely to be multithreaded
to allow it to handle multiple incoming requests concurrently. \\


A Web server, in this way, serves only static content. With the same URI, anybody would be accessing exactly the same resource at any instance. %
Also, visitors will not see any differences across visits unless the files have been physically modified. Presentation of static content is generally adequate for many %
Web sites. However, in order for the World Wide Web to become an interactive media, the ability to serve dynamic content is desired. Dynamic content is usually %
achieved by writing scripts, especially server scripts. These scripts can generate the output in real time to clients based on input from the clients and %
data stored in the server database. Therefore, it is possible that every visitor to a Web site serving dynamic content %
may see different layouts and content customized according to their preferences. That is what that makes the World Wide Web a powerful and interesting %
media compared with conventional media. \\

\section{What is CGI?}

In general Computer Science terms, an \term{interface} provides a well-defined way of interaction between a system 
and external entities. Recall that in the object-oriented programming paradigm, each class exposes itself to the
outside through an interface consisting of methods and properties, and users of the classes do not need to (and 
should not have to) know the details of the implementation and, instead, access the objects through their 
interfaces. \\

In Figure~\ref{fig:http} we saw the interaction between the Web server and the client. As the resource is 
static (an HTML document on the filesystem), the Web server can return the specified resources if they exist.
However, if the resource specified is an executable script, the script will then need to be executed before
returning the generated response to the client. \\

\specialbox{NOTES}{
Note that not all Web servers have CGI script execution enabled. In fact, the administrator of the Web server
needs to explicitly enable execution of CGI scripts, set up a CGI script \term{handler} and associate file extensions of
CGI scripts permitted (e.g. .pl and .cgi) with it such that the scripts will be executed instead of being
fetched and displayed in the client's browser window! Execution of CGI scripts pose certain levels of security 
risks, and are especially dangerous if either the scripts contain a lot of security vulnerabilities or the Web 
server is improperly configured. Therefore, many Web hosting companies do not allow execution of CGI scripts in 
free accounts. Some Web hosting companies set up separate filesystems for hosting CGI scripts to prevent damages
due to CGI script attacks or malfunctioned CGI scripts from affecting the entire filesystem. Later in this chapter 
I will introduce some techniques to write more secure Perl programs to be deployed as CGI scripts. \\
}

The \term{Common Gateway Interface} (CGI) specifies the mechanism through which the Web server should pass data pertaining  
to the HTTP request to the server script, thus allowing the server script to capture data from the client. \\

Compared with other protocol specifications, the \link{http://hoohoo.ncsa.uiuc.edu/cgi}{specification} for CGI is 
intriguingly simple and short that looks more like a tutorial rather than a specification\footnote{Some efforts of
transforming the CGI specification into a formal specification with well-defined grammar is ongoing. Visit
\link{http://cgi-spec.golux.com}{http://cgi-spec.golux.com} for details.}. As you will see, the
principle behind the CGI is very easy to understand. Simply speaking, when the Web server receives a request for
an executable CGI program, the program is executed. If the program is an interpreted one, it needs to invoke an
appropriate interpreter to execute it (which is the perl executable for the purpose of this tutorial). The program
writes to the standard output, and the content of which is then returned to the client. \\

In order for a CGI script to have access to certain information that are only known by the Web server, such 
as the remote IP address of the client and the languages the client supports (which is important to certain
sites which serve content with multiple language versions) that are only available in HTTP request headers,
the CGI stipulates the Web server to make available such information by setting additional environment variables
before the Web server executes the CGI program. Figure~\ref{fig:cgi} illustrates this point. \\

\image{!hbt}{0.9\textwidth}{!}{images/cgi.eps}{CGI script execution}{fig:cgi}

The figure shows the process of a client requesting the CGI script ``index.pl''. The command displayed is
the actual HTTP request being sent by the browser to the Web server, the syntax of which complies with the
HTTP specification. \texttt{GET} is the action that indicates the method used to send the request. We usually
use \texttt{GET} or \texttt{POST} to send HTML form data to the Web server. Following the action is the
path to the resource requested. The browser removes the domain name or IP address from the URI, as it is not 
used by the Web server to get the specified resource. At the end, the HTTP version is specified, and in this
diagram, HTTP/1.0. Notice the string after the question mark in the path. These are passed on the URI, and 
the Web server extracts the text after the question mark (if any) until whitespace is encountered. These are
parameters and are made available by the script by setting the \texttt{QUERY\_STRING} environment variable
as indicated in the diagram. The diagram shows several important environment variables that are usually set 
and their corresponding values. Note that some other environment variables are set as well, but the diagram
would be too large to fit in and are thus omitted. \\

As the script is being executed, it writes to the standard output as the CGI response. Upon completion 
of execution, the Web server collects the CGI response and returns them to the client, inserting any HTTP response
headers necessary at the top to comply with the HTTP specifications. In the diagram, it is assumed the content 
returned is an HTML document, and the type of which is indicated on the first line of the response. This would become
part of the HTTP return header. HTTP headers are read and recognized by the browser but hidden from users.
The HTTP specification stipulates a blank line between the header and the content. Therefore, you should put
a blank line after the last line of the HTTP header, and no blank lines should be present before that. \\

There is a common misconception by many people, especially for those who are not familiar with the CGI mechanism,
to think Perl is CGI, or vice versa, which is highly erroneous. As I have tried to explain previously, CGI itself
is the mechanism that allows executable server programs to access information pertaining to the HTTP request 
through standardized means such as environment variables, instead of being a programming language. Therefore,
it is a fallacy to use the terms ``Perl'' and ``CGI'' interchangeably. In fact, any programming languages may support the CGI
mechanism. CGI programs are not confined to interpreted languages like Perl or Python etc., compiled languages
like C/C++ may also be used to develop CGI programs that are used on a Web server. Therefore, the term ``CGI script''
is not adequately specific, although most CGI programs are written in interpreted languages, preferably Perl.
Therefore, it is discreet to refer to an executable CGI-enabled Perl script as a ``CGI Perl script''. However,
for the purpose of this chapter, I would use the term ``CGI script'' to refer to a ``CGI Perl script'', for simplicity. \\

\section{Your First CGI Program}

Having understood all the necessary concepts you need to know for CGI programming, it's time to get your feet 
wet by building your first CGI script in order for you to understand how a typical CGI script is constructed. \\
 
In this section we are going to build an HTML form that contains a textbox for the visitor to input his/her 
name, and a ``Submit'' button. After the user has pressed the submit button on the form, a CGI script will be 
invoked that prints a phrase of greeting based on the time of the server. Therefore, both the form and the script 
needs to be written. \\
 
\head{EXAMPLE}{}
\vspace{\baselineskip} \\ HTML Form (greeting.html)
\lstinputlisting[numbers=none,backgroundcolor=\color{CodeBackground}]{src/cgi/greeting.html}
\vspace{\baselineskip} Greeting Script (greeting.pl)
\lstinputlisting[backgroundcolor=\color{CodeBackground}]{src/cgi/greeting.pl} \addvspace{\baselineskip}

In order to execute the CGI script, you will need a Web server account that allows execution of CGI scripts. 
There are a couple of free Web hosts on the Internet that lets you deploy self-written scripts. At the time
when this Tutorial goes to press, \link{http://beigetower.org}{Beige Tower} seems to be attractive, but it is possible for you to find better bargains 
elsewhere. Such free CGI hosts are excellent places for you to become familiar with CGI Perl programming before 
you are sophisticated enough to set up your own Web server for development or get more powerful hosting package with 
paid hosts. If you would like to set up your own Web server for testing, please flip to Appendix-\ref{chap:webserver}
for installation and configuration instructions. \\
 
Upload both files to your Web server account. Put the script in the same directory as the form. Please note 
that some Web servers set aside a ``cgi-bin'' directory inside your account where CGI scripts are only allowed 
to be executed inside. For some accounts, CGI scripts may be placed and executed anywhere in your account. This 
is subject to the server configuration and you should consult the system administrator for details. Anyway, in 
short, put both files in the same directory where CGI script execution is allowed. \\
 
Next, you will need to give your files the correct access permissions if your account is on Unix-variant systems. 
Changing access permissions is commonly known as ``chmod''. Most probably you would be using FTP (File Transfer Protocol) 
clients to upload your files to your Web server account. You will need to use an FTP client that supports chmod, 
like WS-FTP, leechFTP, SmartFTP on MS Windows systems. If you are using Linux with X-Windows installed, most 
probably you may want to check out gftp or kbear. If you have telnet/SSH access to your account or you have direct
access to your Web server filesystem, you may chmod on the command line too, but I'm not going into details here
as they are very basic skills Unix users should have already been familiar with. \\
 
The chmod values to give to each file and their corresponding verbose representation is: 

\begin{verbatim}
greeting.html: 644 (rw-r--r--)
greeting.pl: 755 (rwxr-xr-x)
\end{verbatim}

Now you are ready to test your script. Enter the URI of greeting.html in the address field of the Web browser 
and press Enter. Please check with your system administrator or relevant instructions from the hosting service 
on the URI to use. The form should be loaded. Enter your name and click ``Submit'', and a message will be 
displayed if there aren't any errors. \\

\image{!hbt}{0.8\textwidth}{!}{images/greeting.eps}{The ``greeting'' script in action}{fig:greeting}

Both the HTML form and the CGI script are indeed very simple. Compared with earlier scripts executed on the
command line, several elements are new. First, we have used the \texttt{CGI} module to fetch the HTML form data in the 
form of name-value pairs to the CGI script. Next, on line 13 we \texttt{print} a line containing the content type
information. The ``real'' server response in HTML format is between line 25 and 34. \\

On line 4, we \mono{use} the package \mono{CGI::Carp}. By importing the ``\mono{fatalsToBrowser}'' symbol,
this package automatically generates an error page with the error string and debugging information such as 
the filename and number of the line on which the error occurs whenever a runtime error occurs. Simple CGI programs
usually do not perform extensive exception handling and simply use this package to generate the error page.
However, as I mentioned in the previous chapter that these error messages are seldom useful to CGI script users
who are not the developers, sophisticated scripts usually have their own exception handling routines builtin
such as what I presented in the previous chapter. \\

The \texttt{CGI} module is the preferred way for a Perl 5 CGI script to handle CGI-related operations. There
are two major operations a CGI script needs to handle in particular. It needs to check if there are any 
incoming data being passed to the script. Such data are usually passed to the script as parameters on the URI 
(the GET method) or as form data (the POST method). The script is then executed, and results from execution
have to be returned to the client through the Web server. The HTTP user client (presumably the Web browser) 
captures the response and renders it in the proper way as previously mentioned. In this example, incoming parameters
are fetched by the CGI module and returned as a hash reference (\texttt{\$params}) by the \texttt{Vars()} object
method. This is usually the most convenient way to get all the incoming parameters in a single operation. The
module actually parses the CGI environment variable \texttt{QUERY\_STRING}, which can be obtained by 
\texttt{\$ENV\{'QUERY\_STRING'\}}.
Many Perl programmers (and some Perl 5 book authors alike) tend to write their own code of getting HTML
form data. An example is quoted below: \\

\begin{lstlisting}[backgroundcolor=\color{CodeBackground}]
if ($ENV{'REQUEST_METHOD'} eq 'POST') {
	read(STDIN, $buffer, $ENV{'CONTENT_LENGTH'});
} else {
	$buffer = $ENV{'QUERY_STRING'};
}

@pairs = split(/&/, $buffer);

foreach $pair (@pairs) {
	($name, $value) = split(/=/, $pair);
	$value =~ tr/+/ /;
	$value =~ s/%([a-fA-F0-9][a-fA-F0-9])/pack("C", hex($1))/eg;
	$FORM{$name} = $value;
}
\end{lstlisting}

The problem is evident. It requires too much typing, and you are actually reinventing the wheel. Because
\texttt{CGI} is now a standard module bundled in every Perl distribution, there is hardly a reason for not 
using it. Also, the code snippet above is not tolerant against malformed URIs or alternative URI formats. For example,
some scripts use the new convention of separating key-value pairs by \texttt{;} instead of the widely-accepted \texttt{\&}. Unless
you are very familiar with such alternative formats, many of which are not well documented in standards or 
specifications, your code will fail with such URIs. On the other hand, the \texttt{CGI} module was carefully developed 
and has been under constant scrutiny by the Perl community to recognize as many alternative formats as possible. As a result,
it is more reliable and well-maintained, not to count its easiness of use. It is instructive to understand how we
can get the form data from the environment variables though because in this way you would understand more about the
CGI specification, yet you are not recommended to get the form data manually in your production code. \\

Recall that earlier I mentioned the Web server would include the content type in the return response. Line 12 of greeting.pl is
doing exactly that. For static content, the Web server knows the type of content being returned. However, this does
not hold for CGI scripts as it is the CGI scripts that decide on the content to return. Therefore, the script should
supply this piece of information by the \mono{Content-Type} header field. 
\texttt{Content-Type} is one of the most frequently used declaration in HTTP headers. HTTP headers have to be \texttt{print}ed
before the real content, and an empty line should exist between the header lines and the content. Web browsers look
for the header-content boundary in this way so that it can hide the headers properly from users. As no more headers
are specified after the content type line in this example, the extra empty line is produced by two consecutive \texttt{\bs n}. \\

A CGI script does not have to generate all necessary HTTP headers because the Web server should generate them 
automatically. An exception is \mono{Content-Type}, for the reason that I have just explained. Apart from defined 
HTTP headers your scripts may generate other custom headers you prefer. However, because CGI scripts are usually
rendered in browsers, unrecognized headers are generally ignored, anyway. \\ % The standard output of a CGI script is collected
%by the Web server and the header fields generated are parsed. The CGI specification defines a \mono{Location} HTTP
%header, which can be used by a CGI script to perform URI redirection. When the Web server parses the CGI script
%headers and find the \mono{Location} header, it will automatically set the HTTP status code to 302 to inform the
%user agent to redirect to the URI indicated in the \mono{Location} header. The user agent should retrieve the 
%requested document from this URI instead. \\

% "nph-" filename prefix for non-parsed header scripts

\section{GET vs. POST}

In the previous example, we have used a form to transmit user-specified data to the server CGI script. Note that we
have used the GET method to pass the form data, as characterized by the \texttt{method} property of the \texttt{form}
element. However, another method POST is also available. Here we shall discuss what they are and how they differ from each other. \\

To start with, let's investigate how the form data are transmitted in each case. Both methods involve construction of 
a query string of the following format: 

\begin{center}
\texttt{\textit{name1}=\textit{value1}\&\textit{name2}=\textit{value2}\&} ...
\end{center}

Recall the HTML form listed in greeting.html. $<$form$>$ ... $<$/form$>$ encloses the form. Inside the form, we can find a text 
entry control, whose \texttt{name} attribute is ``name''. Each control in the form should have the \texttt{name} atttribute
set. When data is transmitted to the server script, as there may be several pieces of data sent in the same form, every
piece of data has to be labeled with a name so that the server script can differentiate the values. The name is set as the
\texttt{name} attribute in the corresponding form control. In this example, as there is only one control, there is only
one name-value pair in the form. Expressed in the above format, that is: 

\begin{center}
\texttt{name=Bernard}
\end{center}

What if the name or value contains the characters \texttt{\&} or \texttt{=}? In fact, non-alphanumeric characters in the 
name or the value is encoded, in a manner compatible to RFC2396 for a reason that is to be explained shortly. This
document documents the format of Uniform Resource Identifiers, of which Uniform Resource Locators form a subset. The URI 
encoding scheme stipulates all characters except alphanumeric characters and a few unreserved characters (section 2.3)
should be encoded, and this is achieved by representing the character by the hexadecimal representation of its ASCII value
preceded by the character ``\%''. Examples: 

\begin{verbatimtab}[4]
Bernard%20Chan			# Bernard Chan
100%25					# 100%
A%26B%20Associates		# A&B Associates
\end{verbatimtab}

However, as the space character frequently occurs in data to be encoded, there exists an alternative representation of
the space character by the character ``+''. Therefore, if you are parsing the form data manually yourself (which you are
discouraged to do so as explained earlier) you should ensure ``\%20'' and ``+'' are both treated as the space character.
This also explains why it is desirable to use CGI.pm, as to decode properly is not a trivial affair in itself. By using
CGI.pm, you don't even have to care such details --- it is handled for you automatically. \\

Having examined the encoding mechanism used on form data, it's time to look at how form data is sent using methods GET
and POST. \\

In the GET method, the query string is appended to the end of the script URI, separated by the ``?'' character. The form
data is part of the HTTP command. That explains why the URI encoding scheme is used for the construction of the query string. 
An example of the exact HTTP command: 

\begin{center}
\texttt{GET /temp/greeting.pl?name=Bernard+Chan HTTP/1.1}
\end{center}

followed by a series of request HTTP headers. These headers together with the values are made available to the CGI script by 
setting environment variables as well. The environment variable name is the corresponding header field name with the prefix
``HTTP\_'' and all hyphens replaced by underscores. For example, the server script can retrieve the value of User-Agent in 
the HTTP header by querying the environment variable HTTP\_USER\_AGENT. \\

In the CGI specification, it is explicitly mentioned that the value of the \texttt{QUERY\_STRING} environment variable be set to
the query string, that is \texttt{name=Bernard+Chan} in this example and the CGI application should parse this variable to
retrieve the form data. Note that by using the GET method, beccause the query string is embedded as part of the URI, it would
also be displayed in the URI in the HTTP response. \\

On the other hand, by using the POST method the query string is not embedded in the URI. The exact mechanism is a little bit more
complicated. An example of the HTTP command is:

\begin{center}
\texttt{POST /temp/greeting.pl HTTP/1.1}
\end{center}

again, followed by a series of HTTP request headers. However, this time we have an additional line at the end:

\begin{center}
\texttt{Content-Type: application/x-www-form-urlencoded}
\end{center}

This indicates an HTML form is going to be sent to the server script. The browser then sends the following line: 

\begin{center}
\texttt{Content-Length: 17}
\end{center}

This indicates that the forthcoming form data in the form of a query string is of length 17 bytes. The server 
script can get the content length by querying the environment variable \texttt{CONTENT\_LENGTH}. The query string
is then sent to the server script. In the CGI mechanism, the query string is input to the server script via the 
standard input (STDIN), and the content length serves as an indication of the number of bytes to read. \\

% http://www.w3.org/TR/REC-html40/interact/forms.html#h-17.13.4.1

You should now be able to understand fully the code snippet for parsing form data in section 10.4. However, as
always, use the \mono{CGI} module whenever possible. \\

So you may ask, should you use GET or POST? My suggestion is to use POST exclusively for transmission of
forms. That is, if you have an HTML form that requires your visitors to fill in, use POST. Recall that
in the CGI mechanism form data received from GET are saved in the \mono{QUERY\_STRING} environment variable.
It is known that certain shells may pose a limit to the maximum size of environment variables. Therefore, it 
is possible that very long forms are truncated as a result. POST does not have this problem because form
data, having received by the Web server is directly piped to the standard input from which CGI applications
can read. GET is also discouraged because of security issues associated with it. 
Please read Section~\ref{sect:cgisecurity} for security advice concerning the use of the GET form transmission 
method. \\

However, the GET method carries a unique characteristic that makes using it unavoidable in some situations.
In fact, when you click on a hyperlink in an HTML document, you are actually using the GET method to access 
it. %The query string, if any, is embedded in the hyperlink and you do not have to set up a form to send it.
Because the query string is directly embedded in the URI, when the URI is accessed by your visitor, the query
string is sent to the server script automatically without the need of creating an HTML form. This property is vital 
to CGI applications. Today, some Web sites no longer use static HTML documents to serve its content, but to
generate the pages dynamically using dynamic scripting. Usually, on these sites there is 
a single script which is invoked and the page to view is passed to the script as a query string. The script
finds out which page to display by parsing the query string, and the corresponding page is generated and returned
to the client. For example, a guestbook CGI application may have one script file \mono{guestbook.pl}, and the
various functions are differentiated just by the query string. To read the guestbook you may go through a URI
like \mono{guestbook.pl?page=view}, while to sign it the URI is \mono{guestbook.pl?page=sign}. Therefore, you
may reach different parts of the CGI application simply by varying the query string in the URI. This function
cannot be accomplished with the POST method. \\


\section{File Upload}

An HTML form is quite powerful in the sense that it also allows Web-based upload of files to the server. If you
have used Webmail (Web-based email) services you may already have the experience of affixing attachments when 
composing an email. This makes use of the file upload capabilities of HTML forms. \mono{CGI} provides certain facilities
for you to write Perl CGI programs that accepts uploading of files from HTML forms. \\

\specialbox{NOTES}{
\p{Note that many Web servers are configured to disable uploading of files. This is especially the case for free
Web hosting services. This is partly because to allow file upload exposes the system to certain security risks.
Free Web hosting services usually have a large number of users, and without provisions of remuneration providers
of such Web hosting services do not have adequate incentive and staff to carry out all necessary regular security 
audits and bear the potential costs of intrusions should security attacks occur. 
Please read section~\ref{sect:cgisecurity} for details on the security issues associated with execution of CGI scripts.}
}

Because the details pertaining to file uploading with the HTTP protocol is rather complicated, I will not go into
details the underlying mechanism but to introduce to you the constructs used in this example. \\

\head{EXAMPLE}{File Upload}
\vspace{\baselineskip} \\ HTML Form (upload.html)
\lstinputlisting[numbers=none,backgroundcolor=\color{CodeBackground}]{src/cgi/upload.html}
\vspace{\baselineskip} File upload script (upload.pl)
\lstinputlisting[backgroundcolor=\color{CodeBackground}]{src/cgi/upload.pl} \addvspace{\baselineskip}

% TODO: explanation
On line 7 and 8 we get the filename and the filehandle from the CGI object respectively. In earlier sections
I introduced to you the \mono{Vars()} method to get a reference of the hash containing the form data (field, value)
pairs. However, there is also the method \mono{param()} that you can use to query the value of a particular field.
On line 7 the filename is passed as the value whose field name is ``filename'', as we specified in the HTML form. 
The \mono{upload()} method with the field name passed as parameter returns a filehandle from which you may read
the uploaded content. This method returns \mono{undef} if the filehandle cannot be read. While uploading a file
the browser usually sends along some additional information, such as the content type, in the headers. We can use the
\mono{uploadInfo()} method returns a reference to a hash containing this information. You have to pass the filename
as the parameter because an HTML form may have multiple file upload fields. On line 45 we try to retrieve the
content type using this method. The name of the header is \mono{Content-Type}. \\

On line 27 we called the \mono{cgi\_error()} method to determine whether any CGI errors had occurred. CGI 
errors rarely occur in practice, unless, for example, file upload dies in the middle as a result of the client
stops sending data. The second condition we test is the filehandle being \mono{undef}. This condition rarely
occurs, either. If either of these conditions is satisfied, we should consider the upload a failed one. 
However, please note that the \mono{CGI} module fails to notice if the user keys in a filename which points to 
a non-existent file in the file location entry. \\

Line 31 extracts the filename portion from the filename passed in.
Note that the filename does not take on any definite formats, and is determined by the client browser. Some browsers
send the full absolute path to the script (e.g. on Windows it looks like \mono{C:\bs somedir\bs file.zip}), while
some other browsers submit only the filename. Note that because the path, if one exists, depends on the client 
system, we should treat both \mono{\bs} and \mono{/} as pathname separators. Line 34 enables binary mode, as 
explained in Chapter~\ref{chap:files}. We read the file received in 1024-byte chunks and write them to the
output file. \\

\section{Important HTTP Header Fields and Environment Variables}

In this section, a few other important environment variables and HTTP header fields that are useful and 
are made available through the CGI mechanism are outlined below. \\

\subsection{CGI-Related Environment Variables}

\mono{REMOTE\_ADDR} is set to the IP address of the client sending the HTTP request to the server. This is usually the 
address of the client machine itself. However, presence of intermediate \term{proxy servers} between the client and the server
may result in seeing the address of the proxy server instead of the client machine itself. This is because the
proxy server, on receipt of the client HTTP request, replaces the source address with its own before sending it to
the server. Proxy servers are set for various purposes, such as caching and imposing security control. \mono{REMOTE\_HOST}
is the fully qualified domain name (hostname and domain name) of the machine identified by \mono{REMOTE\_ADDR}. As the
machine concerned may not have a domain name, this variable may be NULL. You may also access this piece of information  
through a \mono{CGI} object by invoking the \verb|remote_host()| method. If \verb|REMOTE_HOST| is not available
the content of \verb|REMOTE_ADDR| is returned instead. \\

Other CGI environment variables are generally less useful and are not covered in this tutorial. You can find out 
more about these environment variables from the CGI specification. \\

\subsection{HTTP Header Fields}

In this section I will introduce several commonly-used HTTP header fields that are useful to script authors. 
%The environment variables discussed below correspond to the respective headers received in HTTP requests. 
%Information encoded in these variables are generally more useful compared with the CGI variables. 
For specifics of any of these fields, please consult the HTTP/1.1 specification (RFC2616). \\

The \verb|Accept-Charset| and \verb|Accept-Language| HTTP header fields are sometimes used by script 
authors to decide on the language version of the content to return to the client, in case the content 
is available in multiple language versions. The corresponding environment variables are
\verb|HTTP_ACCEPT_CHARSET| and \verb|HTTP_ACCEPT_LANGUAGE|. 
Note that language and character set are two separate concepts, although they are related. A character set is
the set of symbols used to encode a piece of text in the computer system. Multiple languages may be represented
by the same character set. In particular, Unicode is designed to be a universal character set that embraces all
characters used by every language used today, although it is not yet in widespread use because many legacy 
systems are not Unicode-capable. Chinese, for example, may be represented by the ``Big5'' or ``GB2312'' character
sets depending on whether the traditional or the simplified form is used. Therefore, the same language may be
represented by different character sets. On the other hand, you cannot judge the language only by examining
the character set because multiple languages may be encoded with the same character set. For example, both French
and English use the same character set ``ISO-8859-1''. Therefore, you generally need to query both header fields
to determine the exact language version requested by the client. A user agent may supply multiple values in each
field, with decreasing order of preference. The choice of language or/and character set is usually a configurable 
option in most browsers. \\

The \verb|User-Agent| field, represented by the environment variable \verb|HTTP_USER_AGENT|, carries information 
about the user agent used that is initiating the request. This field
allows script authors to guess the user agent used and return a version of the content optimized for the user 
agent concerned. This is an important piece of information for scripts in which certain features are only available to some browsers  
but not the others. However, the method of encoding this field is unspecified, and each user agent is free to
encode this field in whatever way it likes. In fact, some browsers, like Opera, allow the user to decide the 
user agent string to send to the server. This allows such browsers to masquerade themselves as other browsers
to the server. Therefore, the value collected from this field is not reliable. If you have access to the log
files of a Web server you will find a lot of variations in the user agent strings collected.  You can also 
retrieve the value of this field by invoking the \verb|user_agent()| method on the CGI object. \\

Some HTTP header fields may be generated by server scripts. 
Apart from \verb|Content-Type|, the following headers are commonly used: \\

The \verb|Location| header can be used to redirect a user agent to another location with respect to a resource
request. For example, a script may generate the header by

\begin{verbatimtab}[4]
print "Location: http://www.cbkihong.com/index.pl\n\n";
\end{verbatimtab}

then the user agent will try to access the resource from the location specified. Be careful with redirection ---
make sure an infinite recursion will not be introduced. I have seen some possibly misconfigured sites with a page 
redirecting user agents to itself, which gets the browser into an infinite loop. This adds unnecessary load to
the Web server having to entertain a swarm of consecutive incoming connections. Some browsers are smart enough to
notice this error after a few tens of attempts and bail out with an error message. Other browsers simply keep on 
making connections to the server incessantly without the user ever knowing it. Note that you don't need to generate
the \verb|Content-Type| header anymore if a \verb|Location| header is present because there is no data to be returned. \\

% TODO:
% Cache-Control, Expires
The last HTTP header I am going to cover in this section deals with \term{caches}. A cache,
if properly used, may reduce the time needed by a client to retrieve a resource. A cache is a storage device
that sits between the client and the server and intercepts all traffic passing through. It works as follows. 
When a resource request to a remote server reaches a cache on its way, the cache first checks its internal 
storage to check if the resource being requested is already inside. If so, the cache returns its copy to the
client without passing the request to the server. Otherwise, the request is passed on and the returned response
is put into the cache so that it can be used to serve subsequent requests for the same resource. Resources are
only allowed in a cache for a short period of time. When the age of a cached resource reaches a preconfigured
interval, the item is said to be \term{expired} and deleted from the cache. This keeps the cache size manageable
and prevents some frequently-modified items from being stale and out-of-date. Because caches
are usually installed in close proximity to client systems, resources returned from caches can reach the client
with significantly reduced delays. This also helps reduce traffic over the Internet backbone and lowers the
probability of network congestion. A cache is usually installed at the ISP or the corporate network of a large 
organization to intercept outgoing traffic transparent to client users. However, there is one more cache that
you may or may not be aware of. All mainstream browsers also maintain an internal cache that is checked before
making an outgoing connection to fetch a requested resource. You can usually disable the cache as a browser
option, yet this is customarily enabled by default and most people do not bother (or are not even aware of its
existence) to disable it. After all, it is useful most of the time. It is only in certain scenarios that it 
causes problems. \\

In the old days when nearly all resources on the Web are static resources like files, caching was very effective
because the resource requested by any client behind the cache must be identical. Therefore, the cache can readily
return the cached resource to every client requesting for it. However, consider a dynamically generated Web page.
Although the layout of the page as requested by every client looks similar, the data on the page may differ (such
as the account information in the case of an online bank). Therefore, dynamically generated pages should by definition
not be cached. However, they are sometimes being erroneously cached. There are a lot of implications about this
situation. If a Web page with sensitive information about user A is used to serve a similar request from user B, 
you should understand how serious the problem can be. Or at least, the resource returned is simply inaccurate or
out-of-date (such as when the server generates a newer version of the page while the cache retains an older version).
To avoid going into this dreadful situation the script author usually places the header

\begin{verbatimtab}[4]
Pragma: no-cache
\end{verbatimtab}

in the generated list of headers. Then, unless a cache purposefully ignores it, the response will not be cached. \\

\section{Server Side Includes}

One of the major reasons why many people consider PHP more convenient than Perl is that you can embed
PHP code inside parsed HTML documents. On servers that support it, there exists a feature that is known as
\term{Server Side Includes} (SSI). SSI refers to a set of directives that are placed in an HTML document,
and evaluated when the document is parsed by the Web server. The directives are replaced by the result of
evaluation. \\ 

The capabilities of Server Side Includes is very limited. Among the several functionalities supported,
the most commonly used feature is to embed the results of a CGI program in an HTML document. The SSI directive
used is the \mono{include} directive, which looks like this: 

\begin{verbatim}
<!--#include virtual="counter.pl" -->
\end{verbatim}

where \mono{counter.pl} is the CGI program to be executed. Note that only a file path is supported, it cannot
be a URI. The following shows an HTML page with this SSI directive and the source of the program \mono{counter.pl}: \\

\head{EXAMPLE}{Counters (Server Side Includes)}
\vspace{\baselineskip} \\ HTML Test Page (testpage.shtml)
\lstinputlisting[numbers=none,backgroundcolor=\color{CodeBackground}]{src/cgi/countertest.shtml}
\vspace{\baselineskip} Counter script (counter.pl)
\lstinputlisting[backgroundcolor=\color{CodeBackground}]{src/cgi/counter.pl} \addvspace{\baselineskip}

Note that \mono{counter.pl} is still executed as a CGI program. Therefore, the program still have to be given an
executable chmod and it should output the content type header. Whenever you load the page \mono{testpage.shtml}
the counter stored in the data file will be updated and you will see at the bottom of the page the number of 
views of the current page. \\

\image{!hbt}{0.8\textwidth}{!}{images/testpage.eps}{Counter Embedded in HTML With SSI}{fig:testpage}

\section{Security Issues\label{sect:cgisecurity}}

CGI and Perl together provides a great deal of flexibility and ease in developing Web-aware scripting solutions.
However, the issue of security is usually overlooked in the script development process either because the developers
are not aware of security issues associated with server-side scripting, or in order to meet completion deadlines all
necessary security audits are unfortunately bypassed. The consequence is that a large number of functional scripts but 
full of security holes are being pushed into the market every day, and these programs are introducing new security
backdoors to systems on which they are being deployed. It is easy to write a program that is functional, yet many
security holes are so subtle that they are difficult to be discovered and thus avoided. In this section, we attempt
to highlight certain varieties of attacks possible and compile a set of crude but useful guidelines that would hopefully
help you write more secure CGI scripts. \\

\subsection{Why Should I Care?}

Perl programs are distributed in complete source code, making it very easy for potential crackers to study your
source code, locate vulnerabilities and plan for potential exploits. On the other hand, natively compiled programs
can only be examined in assembly language and it is notoriously difficult to trace the instruction flow, not to count
locating vulnerabilities precisely from them which is a practically infeasible task. Attacks on compiled programs are usually either
discovered on an \textit{ad lib} manner or are attacked in general ways by, say, passing a very long sequence of characters to overflow
the input buffer (many programs written in C or C++ are vulnerable to this attack). Attacks on interpreted languages
are generally more well crafted and precise because the source code is available to target at specific vulnerabilities. \\

CGI scripts are installed on Web servers and are thus open to visitors anywhere over the Internet. They are available
for attack 24 hours a day, 365 days in a year. For standalone systems or systems in an intranet, attacks are only possible
from a limited subset of users. On the Internet, attacks may originate from anywhere in the World, hundreds of miles away. \\

Many CGI scripts are used in business settings or are directly involved in electronic commerce, such as shopping cart programs.
The ability to maintain a high level of security has always been paramount in business applications.
That is because data handled by such applications very likely include highly-confidential personal data
of the clients. Such kinds of sensitive data include the PIN of clients, their record of transactions and
personal information like address or social security number etc. Failure to do so not only invites embarrassment
to the financial institution(s) concerned, such institutions may also be held legally liable for improper
handling of personal data. Occasional cases have been heard in that the credit card information of celebrities
were being captured by crackers through some illegal means and subsequently such data were posted to the 
World Wide Web, putting the subjects concerned in profound embarrassment. People are unlikely to have 
confidence in a company being unable to keep such data from being improperly manipulated. This directly leads to
loss in revenue. \\

\subsection{Some Forms of Attack Explained}

\subsubsection{HTML Form Tampering}

In 2000, Internet Security Systems, Inc. released a security alert on 11 shopping cart applications that were found to be
vulnerable to this attack (please visit \link{http://www.iss.net/issEn/delivery/xforce/alertdetail.jsp?id=advise42}{http://www.iss.net/issEn/delivery/xforce/alertdetail.jsp?id=advise42} for the full text).
This vulnerability vividly demonstrates the ignorance of the script developers concerned in security issues of CGI programming. \\

Shopping cart scripts are set up by merchants on Web servers to track down items users have selected to purchase,
usually while browsing online product catalogues. When a user browses the product catalogue and locates an item
of interest, he or she clicks on a button or icon to save the product information, usually the product identifier and
other items necessary to identify the user such as the username, to the server database. When he or she has finished
adding all the items intended to purchase, he or she checks out and the shopping cart is displayed. A shopping cart
is an abstraction of the list of items a user intends to purchase, and is private to the user concerned, in a sense
similar to a typical shopping experience in modern supermarkets (dropping items into a shopping cart and check out at the cashier). Shopping cart
scripts in general implement this by querying the server database with the username and extracts all items to purchase
specific to the user. The results are displayed in the form of an HTML form for the user to confirm the order and optionally
enter additional information required to process the order, such as delivery address, contact phone number, payment
card details etc. (but such kinds of information are generally saved as part of the user profile at user registration so
that such details no longer need to be specified every time he or she purchases) An HTML form that is vulnerable to this
attack may look like the following: \\

\begin{lstlisting}[backgroundcolor=\color{CodeBackground}]
<html>
	...
	<body>
		...
		<form action="checkout.pl" method="put">
			<input type="hidden" name="username" value="bernardchan"></input>
			<input type="hidden" name="total" value="560"></input>

			<!-- A list of items to purchase, omitted here -->

			<p>After you have confirmed your order, click on the button below.</p>
			<center><input type="submit" value="Submit Order"></input></center>
		</form>
	</body>
</html>
\end{lstlisting}

Line 6 and 7 shows two hidden fields in the HTML form. Unlike the visible form controls, such as the text messages
and the submit button, hidden fields are not rendered in a browser window. They are typically used to hold additional 
data which are not bound to any visible controls but required by the server script (the shopping cart script) to process
the form input. Notice the \texttt{name} and \texttt{value} attributes of the hidden fields. The name-value pairs of
hidden fields are also sent to the server script along with that of other visible controls when the form is submitted.
In this example, the shopping cart script adds the two hidden fields dynamically in generating the HTML form to hold
the username and total amount so that on submit the shopping cart software would arrange the total amount indicated to
be charged on the customer's credit card account. \\

And problem comes. Shopping carts with this vulnerability do not verify the ordering information on submit and blindly
assumes the value of the hidden fields are exactly as generated by the script itself. Therefore, a user may give himeself
or herself a discount by replacing the total amount with a smaller one. Why is this possible? As the form is generated and
sent to the user, the browser renders it in the browser window. However, a person with little knowledge will know one can
browse the HTML source and even save it as a disk file. What he or she needs to do is to use a text editor to modify the
value, save it and load the modified version in the browser. A click on the submit button is all it takes to complete this
attack. Because the script does not perform the check on the total, the attack is successful. This attack may only be
discovered one day, possibly in year-end auditing, that the payment amount and the order do not match, but that would
be too late --- the subject may have already closed the account and hid up that you could no longer find him or her anymore. \\

Why did the script developers commit this error in the first place? Possibly the culprit is ``convenience'', or ``laziness''.
It takes quite many steps to generate the shopping cart HTML form, and the script developers could make their lives easier 
by putting all necessary details to process the order on the form instead of having the checkout script calculate the total
amount again, because the calculated total needs to be displayed on the HTML form anyway. By means of ``hidden'' fields they
probably thought no one would bother to read the HTML source and discover this security hole. Most people probably would
not bother to, but it still leaves a backdoor for abuse. As an e-commerce application I believe having a backdoor like this 
is totally unacceptable, however pretty or sophisticated the application can be. \\

\paragraph{Conclusion} Do not trust anything sent over the network. Always carry out all verifications possible at the
best of your knowledge before committing anything. \\

\subsubsection{Privacy Issues}
% SSL
While this is not directly related to Perl CGI programming in general, you should be aware that the Hypertext 
Transfer Protocol (HTTP) that is used for accessing the World Wide Web conveys all its messages in plaintext.
Therefore, when you submit a form with your password, address or credit card number etc. filled in these fields 
are all transmitted in plaintext through the Internet. The Internet is a gigantic interconnected network of
computers. When you send a message to a remote host, for example, to browse a certain Web page on a remote
Web server, the Internet routing system has to find a path between you and the remote Web server. The
scale of the Internet is so large that usually you have to go through many intermediate hosts on your way to 
the remote host. Any of these hosts is able to read the content of the messages, or even to modify it. Under
certain circumstances, other malicious hosts may also be able to \term{eavesdrop} the traffic through these
hosts. Therefore, there is no confidentiality at all. \\
% many hops, prying eyes, network sniffing

If you Web site has to collect privacy-sensitive information from your users by means of forms, the Web server
concerned should be configured to serve these forms using the \term{Secure Socket Layer} (SSL) protocol or \term{Transport
Layer Security} (TLS), a close variant of SSL. The Internet uses a layered architecture. SSL acts between HTTP
on top and TCP/IP, the Internet delivery system at the bottom. Therefore, messages between a Web browser and
the Web server are encrypted in transit and thus malicious hosts, even intermediate hosts on the path are
unable to decrypt the messages except the communicating parties. For example, when your browser sends a form
with your personal information which is to be sent over SSL, the HTTP message encapsulating the form data
are passed to the SSL layer which encrypts it and then passed to the TCP/IP subsystem to send it to the remote
Web server. The Web server on the remote end reverses these steps to receive the message from the TCP/IP 
subsystem, decrypts it and then recover the original HTTP message. Configuration of SSL/TLS is performed at
the Web server and no modification to your CGI programs is needed. \\

% GET
Earlier in this chapter I mentioned the two form submission methods, namely GET and POST. In the GET method,
the form data encoded as a query string are also carried on the URI. This poses a number of security issues
if such form data contain privacy-sensitive information. First, many browsers now cache recently visited URIs
so that users can easily revisit them without having to manually bookmark them or otherwise save them. This
causes potential privacy violations on those single-user systems (and thus no password is asked to log on the
system) or misconfigured multi-user systems. For instance, consider a typical patronage to a cybercaf\'e in
your neighbourhood. You logged on a PC inside and conducted an online transaction to buy something from an online store. Suppose the
GET method was used and thus form data including your password in plaintext were carried on the URI, and was 
thus cached by the browser. Then you logged off and left the caf\'e. Then another person logged on the PC, and
when he or she started the browser, a URI to the online store with your password embedded appeared in the
browsing history. Does this sound scary to you? System configurations of most public workstations generally should
have this issue fixed already. However, this is not a guarantee. Another situation where this URI may be
divulged is due to a header field in HTTP that is called ``Referer''. When a user agent, e.g. browser, sends an HTTP request
message to the remote Web server to retrieve a certain resource, the user agent may include in the header field
``Referer'' the referring URI, that is, the URI of the document from which the URI of this resource was obtained. 
For example, when you are viewing a Web page on a certain site like \mono{http://www.somesite.com/links.html} and
you click on one of the hyperlinks there to \mono{http://www.anothersite.com}, for example, the URI that refers
to \mono{links.html} may be included in the HTTP header which is sent to \mono{anothersite.com}. Therefore, if 
the referrer URI refers to a script URI with privacy-sensitive information embedded in it they will also be carried
along with the next URI access. \\

You, as a CGI script and Web developer, may do your part to protect your customers by using the POST 
form transmission method instead of the GET method whenever privacy-sensitive form data are involved
because form data are not carried in the URI. That does not eliminate the need for encrypted tunnels such as
SSL to keep out of prying eyes over the network, however. \\

\subsubsection{\mono{eval()} and Related Attacks}

As we have seen in the previous chapter, \mono{eval()} and backticks are very dangerous and frequently taken advantage by
attackers to execute on the Web server whatever commands they specify. As noted, enable taint mode to check for potential
insecure input data that may be passed to potentially dangerous operations. Construct precise patterns to ensure all
input data match the patterns intended. Do not use these operations unless that is absolutely necessary. Also bear
in mind that taint mode is not the end. There are a lot of traps taint mode cannot guard against. Because all of these
points have been duly introduced in the previous chapter, I am not repeating it here again. \\


\subsection{Safe CGI Scripting Guidelines}

Today, new CGI scripts emerge on the market every
day. However, how many of them are really secure? A script that has a pretty interface does not imply it must be well
tested for security bugs. Because most users of CGI scripts are not programmers,
they cannot protect themselves but rely on you, the script developers, to safeguard your programs against abuse.
Therefore, you have an irrefutable obligation to ensure your programs are as secure as possible. \\	% undeniable?

Below I put together a set of guidelines that you can check against your programs to evaluate if they are secure.
Some of the points have been discussed earlier in the tutorial but being repeated here as a summary. These guidelines are not exhaustive, however.
Despite the section heading, these guidelines pertain to any kinds of programs receiving input from outside of the program.
Discreet security personnel are well-educated about these guidelines. So should you. \\

% http://tech.irt.org/articles/js099/
\begin{itemize}

\item Refrain from \texttt{eval()}, backticks and \mono{system()} whenever possible. If you do use it, check its content. 
Ensure the content is exactly in the form expected and enable taint mode. 

\item For incoming data, it is generally more secure to define a pattern that describes what is allowed than what is
disallowed. For example, to check if an email address is syntactically valid you create a pattern which describes the syntax of a valid email
address, for example \mono{/\^{}\bs w+@(\bs w+\bs .?)+\$/}. With this deny-everything-by-default arrangement, an email 
address that  matches this pattern must be a syntactically valid one. 
This is preferable to a pattern which enumerates the invalid characters and test whether the string
contains these invalid characters, because given the large number of possible characters you may not be able to enumerate them.

\item Do not be more permissive than necessary. That is, do not set a file permission value to more than what you need to
get your program running. More permissions imply higher risk.

\item Implement \term{rings of security} where applicable. A single line of defense is generally weak because if this
line of defense is broken, intrusion will be successful. In security analysis, we try to locate all possible \term{attack paths}
in the program. An attack path describes a sequence of steps an intruder may take to launch a successful attack. To implement rings
of security implies we should try to establish multiple lines of defense on these paths --- even if a line of defense
can be broken into, there is still another barrier that deters attackers from proceeding. The more barriers to surmount,
the less likely an intrusion attempt will be successful and, very likely, the attacker will give up and turn to another
more vulnerable system or program instead. \\

\end{itemize}

\section*{Summary}

\begin{summary}
\item A document written in HTML is a plain text document with extra markup added that indicates the logical structure of the document.
\item The World Wide Web is a content presentation and distribution platform built on top of HTTP user agents and Web servers. 
\item HTTP is the protocol used for World Wide Web access. A protocol defines a common language of communication between parties. 
\item A Web server serves static content by retrieving resources from the filesystem in the form of files and return them to the client side.
\item A Web server may also serve dynamic content by executing some programs on the server and return the output generated to the client side.
\item A Web browser is just an HTTP user agent which initiates HTTP requests, and captures responses from Web servers and renders them to produce the graphical display as Web pages.
\item CGI, the Common Gateway Interface, defines the method of interaction between the Web server and an executable server program.
\item CGI stipulates that Web servers should pass certain information pertaining to the HTTP request received from the client side to server executables by means of environment variables. Generated output are collected by the Web server and returned to the client side, supplementing any missing HTTP headers as necessary.
\item The \mono{CGI} module may be used for a wide variety of CGI-related operations, including form-based file uploading.
\item The \mono{Vars()} object method of \mono{CGI} returns a reference to a hash containing incoming form data.
\item A Perl CGI program is required to generate the \mono{Content-Type} HTTP header indicating the kind of data returned to the client side.
\item In the GET form transmission mechanism, form data are appended to the URI as parameters. In the POST mechanism, on the other hand, form data are sent independent of the URI and CGI programs may obtain them by reading from the standard input.
\item The GET mechanism is not recommended because of limitations imposed by the shell and potential privacy breaches through carrying privacy-sensitive data over to third-party servers by means of the \mono{Referer} HTTP header.
\item In both mechanisms, form data are transformed into a query string with special characters encoded before being sent.
\item Server Side Includes is a mechanism that you may use to embed the generated output of a CGI program in a server-parsed HTML document. Usually, server-parsed HTML documents take on the extension \mono{.shtml} instead of \mono{.html}.
\item Server CGI scripting may easily expose a server to security breaches if CGI scripts are not written in a careful manner.
\item Do not trust anything sent over the network, including form data received from the client. Carry out all verifications at the best of your knowledge before accepting them.
\item HTTP messages are sent in plain text. Security-aware Web applications should serve their content over encrypted tunnels such as SSL or TLS to protect their users.
\item Refrain from runtime evaluation and system shell access for CGI programs if possible. Otherwise, enable taint mode and verify input data to ensure they are safe.
\end{summary}

\section*{Questions}

\begin{enumerate}
\item {
	We have seen how a shopping cart application with the form tampering vulnerability could be abused to give adversaries
	discounts on items paid for. In the make-up HTML form example shown, identify another attack that may be performed and devise a
	mechanism to defend this attack. Assume on submit that the value received through the ``total'' field is charged on the
	user's account whose name is ``username'' immediately without performing any verifications. \\
}
\end{enumerate}
