HTTP server for unit tests

  • Revision slug: Httpd.js/HTTP_server_for_unit_tests
  • Revision title: HTTP server for unit tests
  • Revision id: 85547
  • Created:
  • Creator: Waldo
  • Is current revision? No
  • Comment SJS is the new CGI

Revision Content

This page describes the JavaScript implementation of an HTTP server located in {{ Source("netwerk/test/httpserver/") }}.

Functionality

Here are some of the things you can do with the server:

  • map a directory of files onto an HTTP path on the server, for an arbitrary number of such directories (including nested directories)
  • define custom error handlers for HTTP error codes
  • serve a given file for requests for a specific path, optionally with custom headers and status
  • define custom "CGI" handlers for specific paths using a JavaScript-based API to create the response (headers and actual content)
  • run multiple servers at once on different ports (8080, 8081, 8082, etc.)

This functionality should be more than enough for you to use it with any test which requires storage on an HTTP server but doesn't require a particular host.

Where you can use it

The server is written primarily for use from xpcshell-based tests, and it can be used as an inline script or as an XPCOM component (assuming you've compiled the server's IDL file and deployed the XPT correctly). The Mochitest framework also uses it to serve its tests, and {{ Source("layout/tools/reftest/README.txt", "reftests") }} can optionally use it when their behavior is dependent upon specific HTTP header values.

Ways you might use it

  • application update testing
  • cross-"server" security tests
  • cross-domain security tests, in combination with the right proxy settings (e.g. using Proxy AutoConfig)
  • tests where the behavior is dependent on the values of HTTP headers (e.g., Content-Type)
  • anything which requires use of files not stored locally
  •  ???

Functionality

The best and first place you should look for documentation is {{ Source("netwerk/test/httpserver/nsIHttpServer.idl") }}. It's extremely comprehensive and detailed, and it should be enough to figure out how to make the server do what you want. I also suggest taking a look at the less-comprehensive server {{ Source("netwerk/test/httpserver/README", "README") }}, although the IDL should usually be sufficient.

Running the server

To use the server in xpcshell-based tests, include the following line at the top of your test file:

do_import_script("netwerk/test/httpserver/httpd.js");

Once you've done that, you can create a new server as follows:

var server = new nsHttpServer();

server.registerDirectory("/", nsILocalFileForBasePath);

const SERVER_PORT = 8080; // or some suitably large number
server.start(SERVER_PORT);

// and when the tests are done, most likely from a callback...
server.stop();

Note that because the server only runs when the xpcshell event loop is spun, you'll have to use do_test_pending(); and doing your testing via callbacks, making sure to call do_test_finished(); when the tests finish. Doing so is a little hairy, but since such a test is unlikely to be synchronous anyway it shouldn't be too hard to handle server asynchronicity.

Note: You must make sure to stop the server (the last line above) before your test completes. Failure to do so will result in the "XPConnect is being called on a scope without a Components property" assertion, which will cause your test to fail in debug builds, and you'll make people running tests grumbly because you've broken the tests.

Debugging errors

The server's default error pages don't give much information, partly because the error-dispatch mechanism doesn't accommodate doing so and partly because exposing errors in a real server could make it easier to exploit them. If you don't know why the server is acting a particular way, edit {{ Source("netwerk/test/httpserver/httpd.js", "httpd.js") }} and change the value of DEBUG to true. This will cause the server to print information about the processing of requests (and errors encountered doing so) to the console, and it's usually not difficult to determine why problems exist from that output. DEBUG is false by default because the information printed with it set to true unnecessarily obscures tinderbox output.

Header modification for files

The server supports modifying the headers of the files (not request handlers) it serves. To modify the headers for a file, create a sibling file with the first file's name followed by ^headers^. Here's an example of how such a file might look:

HTTP 404 I want a cool HTTP description!
Content-Type: text/plain

The status line is optional; all other lines specify HTTP headers in the standard HTTP format. Any line ending style is accepted, and the file may optionally end in a single blank line, to play nice with Unix text tools like diff and cvs.

Hidden files

Any file which ends with a single ^ is inaccessible when querying the web server; if you try to access such a file you'll get a 404 page instead. If for some reason you need to serve a file ending with a ^, just tack another ^ onto the end of the file name and the file will then become available at the single-^ location.

At the moment this feature is basically a way to smuggle header modification for files into the file system without making those files accessible to clients; it remains to be seen whether and how hidden-file capabilities will otherwise be used.

SJS: server-side scripts

Support for server-side scripts is provided through the SJS mechanism. Essentially an SJS is a file with a particular extension, chosen by the creator of the server, which contains a function with the name handleRequest which is called to determine the response the server will generate. That function acts exactly like the handle function on the nsIHttpRequestHandler interface. First, tell the server what extension you're using:

const SJS_EXTENSION = "cgi";
server.registerContentType(SJS_EXTENSION, "sjs");

Now just create an SJS with the extension cgi and write whatever you want. For example:

function handleRequest(request, response)
{
  response.setStatusLine(request.httpVersion, 200, "OK");
  response.write("Hello world!  This request was dynamically " +
                 "generated at " + new Date().toUTCString());
}

Further examples may be found in the Mozilla source tree in existing tests.

Examples

Shorter examples (for tests which only do one test):

  • {{ Source("netwerk/test/unit/test_bug331825.js") }}
  • {{ Source("netwerk/test/unit/test_httpcancel.js") }}
  • {{ Source("netwerk/test/unit/test_cookie_header.js") }}

Longer tests (where you'd need to do multiple async server requests):

  • {{ Source("netwerk/test/httpserver/test/test_setstatusline.js") }}
  • {{ Source("netwerk/test/unit/test_content_sniffer.js") }}
  • {{ Source("netwerk/test/unit/test_authentication.js") }}
  • {{ Source("netwerk/test/unit/test_event_sink.js") }}
  • {{ Source("netwerk/test/httpserver/test/") }}

Examples of modifying HTTP headers in files may be found at {{ Source("netwerk/test/httpserver/test/data/cern_meta/") }}.

Future directions

The server, while very functional, is not yet complete. There are a number of things to fix and features to add, among them support for sending requests as their data is generated (instead of batching it and sending all at once when the response is fully created -- no bug yet), POST method support ({{ Bug("366371") }}), and better conformance to the MUSTs and SHOULDs of HTTP/1.1. If you have suggestions for functionality or find bugs, file them in {{ Enter-bug("Core", "Testing") }} and CC jwalden+bmo.

Revision Source

<p>This page describes the JavaScript implementation of an HTTP server located in {{ Source("netwerk/test/httpserver/") }}.
</p>
<h3 name="Functionality"> Functionality </h3>
<p>Here are some of the things you can do with the server:
</p>
<ul><li> map a directory of files onto an HTTP path on the server, for an arbitrary number of such directories (including nested directories)
</li><li> define custom error handlers for HTTP error codes
</li><li> serve a given file for requests for a specific path, optionally with custom headers and status
</li><li> define custom "CGI" handlers for specific paths using a JavaScript-based API to create the response (headers and actual content)
</li><li> run multiple servers at once on different ports (8080, 8081, 8082, etc.)
</li></ul>
<p>This functionality should be more than enough for you to use it with any test which requires storage on an HTTP server but doesn't require a particular host.
</p>
<h3 name="Where_you_can_use_it"> Where you can use it </h3>
<p>The server is written primarily for use from <a href="en/Writing_xpcshell-based_unit_tests">xpcshell-based tests</a>, and it can be used as an inline script or as an XPCOM component (assuming you've compiled the server's IDL file and deployed the XPT correctly). The <a href="en/Mochitest">Mochitest</a> framework also uses it to serve its tests, and {{ Source("layout/tools/reftest/README.txt", "reftests") }} can optionally use it when their behavior is dependent upon specific HTTP header values.
</p>
<h3 name="Ways_you_might_use_it"> Ways you might use it </h3>
<ul><li> application update testing
</li><li> cross-"server" security tests
</li><li> cross-domain security tests, in combination with the right proxy settings (e.g. using <a href="en/Proxy_AutoConfig">Proxy AutoConfig</a>)
</li><li> tests where the behavior is dependent on the values of HTTP headers (e.g., Content-Type)
</li><li> anything which requires use of files not stored locally
</li><li> ???
</li></ul>
<h3 name="Functionality_2"> Functionality </h3>
<p>The best and first place you should look for documentation is {{ Source("netwerk/test/httpserver/nsIHttpServer.idl") }}. It's extremely comprehensive and detailed, and it should be enough to figure out how to make the server do what you want. I also suggest taking a look at the less-comprehensive server {{ Source("netwerk/test/httpserver/README", "README") }}, although the IDL should usually be sufficient.
</p>
<h4 name="Running_the_server"> Running the server </h4>
<p>To use the server in <a href="en/Writing_xpcshell-based_unit_tests">xpcshell-based tests</a>, include the following line at the top of your test file:
</p>
<pre class="eval">do_import_script("netwerk/test/httpserver/httpd.js");
</pre>
<p>Once you've done that, you can create a new server as follows:
</p>
<pre class="eval">var server = new nsHttpServer();

server.registerDirectory("/", nsILocalFileForBasePath);

const SERVER_PORT = 8080; // or some suitably large number
server.start(SERVER_PORT);

// and when the tests are done, most likely from a callback...
server.stop();
</pre>
<p>Note that because the server only runs when the xpcshell event loop is spun, you'll have to use <code>do_test_pending();</code> and doing your testing via callbacks, making sure to call <code>do_test_finished();</code> when the tests finish. Doing so is a little hairy, but since such a test is unlikely to be synchronous anyway it shouldn't be too hard to handle server asynchronicity.
</p>
<div class="note">
<p>Note: You <b>must</b> make sure to stop the server (the last line above) before your test completes. Failure to do so will result in the "XPConnect is being called on a scope without a Components property" assertion, which will cause your test to fail in debug builds, and you'll make people running tests grumbly because you've broken the tests.
</p>
</div>
<h4 name="Debugging_errors"> Debugging errors </h4>
<p>The server's default error pages don't give much information, partly because the error-dispatch mechanism doesn't accommodate doing so and partly because exposing errors in a real server could make it easier to exploit them. If you don't know why the server is acting a particular way, edit {{ Source("netwerk/test/httpserver/httpd.js", "httpd.js") }} and change the value of <code>DEBUG</code> to <code>true</code>. This will cause the server to print information about the processing of requests (and errors encountered doing so) to the console, and it's usually not difficult to determine why problems exist from that output. <code>DEBUG</code> is <code>false</code> by default because the information printed with it set to <code>true</code> unnecessarily obscures tinderbox output.
</p>
<h4 name="Header_modification_for_files"> Header modification for files </h4>
<p>The server supports modifying the headers of the files (not request handlers) it serves. To modify the headers for a file, create a sibling file with the first file's name followed by <code>^headers^</code>. Here's an example of how such a file might look:
</p>
<pre class="eval">HTTP 404 I want a cool HTTP description!
Content-Type: text/plain
</pre>
<p>The status line is optional; all other lines specify HTTP headers in the standard HTTP format. Any line ending style is accepted, and the file may optionally end in a single blank line, to play nice with Unix text tools like diff and cvs.
</p>
<h4 name="Hidden_files"> Hidden files </h4>
<p>Any file which ends with a single <code>^</code> is inaccessible when querying the web server; if you try to access such a file you'll get a 404 page instead. If for some reason you need to serve a file ending with a <code>^</code>, just tack another <code>^</code> onto the end of the file name and the file will then become available at the single-<code>^</code> location.
</p><p>At the moment this feature is basically a way to smuggle header modification for files into the file system without making those files accessible to clients; it remains to be seen whether and how hidden-file capabilities will otherwise be used.
</p>
<h4 name="SJS:_server-side_scripts"> SJS: server-side scripts </h4>
<p>Support for server-side scripts is provided through the SJS mechanism. Essentially an SJS is a file with a particular extension, chosen by the creator of the server, which contains a function with the name <code>handleRequest</code> which is called to determine the response the server will generate. That function acts exactly like the <code>handle</code> function on the <code>nsIHttpRequestHandler</code> interface. First, tell the server what extension you're using:
</p>
<pre class="eval">const SJS_EXTENSION = "cgi";
server.registerContentType(SJS_EXTENSION, "sjs");
</pre>
<p>Now just create an SJS with the extension <code>cgi</code> and write whatever you want. For example:
</p>
<pre class="eval">function handleRequest(request, response)
{
  response.setStatusLine(request.httpVersion, 200, "OK");
  response.write("Hello world!  This request was dynamically " +
                 "generated at " + new Date().toUTCString());
}
</pre>
<p>Further examples may be found <a class="external" href="http://mxr.mozilla.org/mozilla/find?text=&amp;kind=text&amp;string=sjs%24">in the Mozilla source tree</a> in existing tests.
</p>
<h3 name="Examples"> Examples </h3>
<p>Shorter examples (for tests which only do one test):
</p>
<ul><li> {{ Source("netwerk/test/unit/test_bug331825.js") }}
</li><li> {{ Source("netwerk/test/unit/test_httpcancel.js") }}
</li><li> {{ Source("netwerk/test/unit/test_cookie_header.js") }}
</li></ul>
<p>Longer tests (where you'd need to do multiple async server requests):
</p>
<ul><li> {{ Source("netwerk/test/httpserver/test/test_setstatusline.js") }}
</li><li> {{ Source("netwerk/test/unit/test_content_sniffer.js") }}
</li><li> {{ Source("netwerk/test/unit/test_authentication.js") }}
</li><li> {{ Source("netwerk/test/unit/test_event_sink.js") }}
</li><li> {{ Source("netwerk/test/httpserver/test/") }}
</li></ul>
<p>Examples of modifying HTTP headers in files may be found at {{ Source("netwerk/test/httpserver/test/data/cern_meta/") }}.
</p>
<h3 name="Future_directions"> Future directions </h3>
<p>The server, while very functional, is not yet complete. There are a number of things to fix and features to add, among them support for sending requests as their data is generated (instead of batching it and sending all at once when the response is fully created -- no bug yet), POST method support ({{ Bug("366371") }}), and better conformance to the MUSTs and SHOULDs of HTTP/1.1. If you have suggestions for functionality or find bugs, file them in {{ Enter-bug("Core", "Testing") }} and CC jwalden+bmo.
</p>
Revert to this revision