Mochitest

  • Revision slug: Mochitest
  • Revision title: Mochitest
  • Revision id: 9860
  • Created:
  • Creator: Waldo
  • Is current revision? No
  • Comment 136 words added, 72 words removed

Revision Content

Mochitest is an automated testing framework built on top of the MochiKit JavaScript libraries. It's just one of the automated regression testing frameworks used by Mozilla. Tests report success or failure to the test harness using JavaScript function calls.

Mochitest's unique strength is that it runs tests written as webpages in a full browser environment where the tests have chrome (elevated) privileges. This allows JavaScript in the tests to do much, much more than it would otherwise be able to do. In addition to the capabilities a script would normally have (e.g. DOM manipulation), scripts can access XPCOM components and services, and even access the browser itself. This allows a script to, say, simulate user input to the browser's user interface, before examining the browser to verify that the input had the intended results.

Mochitest's use of JavaScript function calls to communicate test success or failure can make it unsuitable for certain types of test. Only things that can in some way be tested using JavaScript (with chrome privileges!) can be tested with this framework. Given some creativity, that's actually much more than you might first think, but it's not possible to write Mochitest tests to directly test a non-scripted C++ component, for example.  (Use a compiled-code test to do that.)

Running tests

The Mozilla build machines run Mochitest as part of the build process, so we get to know pretty quickly if someone commits a change to the source code that breaks something. However, you should still run Mochitest yourself before you commit any risky new code. You don't want to be the one who wastes everyone's time by breaking the tree if you can help it. :-)

Running the whole test suite

To run Mochitest, first build Mozilla with your changes; then run Mochitest's runtests.py script without passing it any command line arguments:

cd $(OBJDIR)/_tests/testing/mochitest
python runtests.py

This will open your build with a document containing a "Run Tests" link at the top. To run the tests simply click this link and watch the results being generated. Test pass/fail is reported for each test as it runs and is recorded on the page.

Image:Mochitest.png

Note: you should keep focus on the browser window while the test are being run, as some may fail otherwise (like the one for {{ Bug("330705") }} for example). Linux users can save themselves this inconvenience by using a dummy X server (see Diverting X output below).

Running select tests

To run a single test (perhaps a new test you just added) or a subset of the entire Mochitest suite, use the --test-path option to specify the test or the subdirectory of tests that you want to run. For example, to run only the test {{ Source("content/base/test/test_CrossSiteXHR.html", "test_CrossSiteXHR.html") }} in the Mozilla source tree, you would run this command:

python runtests.py --test-path=content/base/test/test_CrossSiteXHR.html

To run all the tests in {{ Source("content/svg/") }}, this command would work:

python runtests.py --test-path=content/svg/

Note that the path specified by the --test-path is the path to the test or directory within the Mozilla source tree. If the path is a directory, then the tests in that directory and all of its subdirectories will be loaded.

Logging results

The output from a test run can be sent to the console and/or a file (by default the results are only displayed in the browser). There are several levels of detail to choose from. The levels are DEBUG, INFO, WARNING, ERROR and FATAL, where DEBUG produces the highest detail (everything), and FATAL produces the least (a message will only be logged for an event that causes the test run to abort prematurely).

To log to a file use --log-file=FILE. By default the file logging level is INFO but you can change this using --file-level=LEVEL.

To turn on logging to the console use --console-level=LEVEL.

For example, to log test run output to the file ~/mochitest.log at DEBUG level detail you would use:

python runtests.py --log-file=~/mochitest.log --file-level=DEBUG

Diverting X output

The tests must run in a focused window, which effectively prevents any other user activity on the engaged computer. Linux users can reclaim their boxes by telling the suite to use a hidden virtual desktop. If Xvfb is or can be installed, the following command launches the tests without blocking the active session:

nice xvfb-run python runtests.py --log-file=~/mochitest.log --file-level=DEBUG --autorun --close-when-done --console-level=DEBUG

Other possible configurations have also been discussed in {{ Bug("434365") }}.

Other 'runtests' options

The 'runtests' script recognizes several other options - use the --help option to get a list. Note that there is separate documentation for the --chrome, --browser-chrome and --a11y options.

Writing tests

A Mochitest test is simply an HTML, XHTML or XUL file that contains some JavaScript to test for some condition(s).

You can use Mochitest maker to run most tests without having to build Mozilla.

Try to avoid Mochitest

Yes, really. For many things Mochitest is overkill. In general you should always try to use one of the lighter-weight testing frameworks. For example, if you only want to test a single XPCOM component then you should use xpcshell. On the other hand there are some things that Mochitest cannot do, or isn't designed to do. For example, for visual output tests you should try to use the reftest framework. For more information on the different types of automated testing frameworks see Mozilla automated testing.

Test templates

You can avoid typing out boilerplate by using the {{ Source("testing/mochitest/gen_template.pl", "gen_template") }} perl script to generate a test template. This script takes two optional arguments:

  1. -b : a bug number
  2. -type : template type. {html|xhtml|xul}. defaults to html.

For example:

cd mozilla/testing/mochitest/
perl gen_template.pl -b=123456 > path/to/test_bug123456.html
perl gen_template.pl -b=123456 --type=xul > path/to/test_bug123456.xul

Note that Mochitest requires the file name of all tests to begin with the string "test_". See the section below for help on deciding where your tests should go in the tree.

The elements with id 'content' and 'display' in the generated file can be used by your script if you need elements to mess around with. TODO: uh, would someone care to expand on that?

Test functions

Each test must contain some JavaScript that will run and tell Mochitest whether the test has passed or failed. [http://mxr.mozilla.org/mozilla-centr...chiKit/Test.js MochiKit/Test.js] provides a number of functions for the test to use that communicate the pass/fail to Mochitest. These include:

  • ok(expressionThatShouldBeTrue, "Error message") -- tests a value for truthiness
  • is(thingA, thingB, "Error message") -- compares two values (using ==, not ===)
  • isnot(thingA, thingB, "Error message") -- opposite of is()

See the {{ Source("testing/mochitest/README.txt", "README") }} for an example of their use.

If you want to include a test for something that currently fails, then instead of commenting it out, you should use one of the "todo" equivalents so Tinderbox can notice if it suddenly starts passing:

  • todo(falseButShouldBeTrue, "Error message")
  • todo_is(thingA, thingB, "Error message")
  • todo_isnot(thingA, thingB, "Error message")

Helper functions

Right now all of Mochikit is available (this will change in {{ Bug("367393") }}); {{ Bug("367569") }} added sendChar, sendKey, and sendString helpers. These are available in {{ Source("testing/mochitest/tests/SimpleTest/EventUtils.js") }}.

Adding tests to the tree

Once you've written a new test you need to add it to the Mozilla source tree and tell the build system about it so that the Mozilla tinderboxes will run it automatically.

Choosing a location

New Mochitest tests should go somewhere close to the code they are testing, hopefully in the same module, so that ownership of the test cases is clear. For example, if you create a new test for some HTML feature, you probably want to put the test in {{ Source("content/html/content/test") }} or {{ Source("content/html/document/test") }}. If a test directory does not exist near the code you are testing you can add a new test directory as the patch in {{ Bug("368531") }} demonstrates.

Makefile changes

To tell the build system about your new test you need to add the name of your test file to _TEST_FILES in the test directory's Makefile.in.

If your test spans multiple files, only name the main one "test_...". This is the one that will show up in the list of testcases to run. The other files should have some other name, but must still be added to the _TEST_FILES in Makefile.in.

Keep in mind that if you're adding chrome tests, you'll need to change the Makefile to install the tests in _tests/testing/mochitest/chrome rather than _tests/testing/mochitest/tests.

Building and running new tests

Before committing a new tests you should check that the Makefile.in changes are correct and that your tests pass as you expect them to. To check your test, first export it to the Mochitest directory by running the command:

make

in the object directory corresponding to the test file's location in the source tree. Now open Mochitest as described above, but this time, instead of clicking on the "Run Tests" link, search for your test and click on it.

SSL and https enabled tests


Mochitests must be run from http://localhost:8888 to succeed; however, some tests may require use of additional protocols, hosts, or ports to test cross-origin functionality.  The Mochitest harness addresses this need by mirroring all content of the original server onto a variety of other servers through the magic of proxy autoconfig and SSL tunneling.  The full list of schemes, hosts, and ports on which tests are served, all of which serve exactly the same content as http://localhost:8888, is specified in {{ Source("build/pgo/server-locations.txt") }}.  Note, however, that not all origins described there are equivalent: some specify particular SSL certificates for testing purposes, while some allow pages on that server to request elevated privileges; read the file for full details.

How Mochitest SSL server works?


Mochitest (better say, the Profile Guided Optimization) is using FindProxyForURL JS function as a proxy configuration function to interactively provide the listed hosts. It also, when accessing an https host in the list, redirects transparently to ssltunnel server program that is configured to act as an HTTPS proxy and forwarding tunnel. That is, it receives the 'CONNECT host' request from the browser, checks the host is on the list, if so, answers with '200 CONNECTED' response and switches to pure SSL tunnel. Ssltunnel then acts as the remote host server. Accepts SSL session, identifies it self with the proper server certificate for the host and from now on just forwards all encrypted communication between the browser and the ssltunnel in unencrypted form to the common testing server listening on localhost:8888, transparently.

How are SSL certificates managed?


For default SSL Mochitest capability there is a pair of a persistent testing certification authority certificate and an on-demand generated server certificate. This schema provides simulation of a common model of a trusted certificate issuer and a trusted server in the Internet.

The certification authority (CA) certificate is a pre-generated self-signed certificate, valid for 10 years. This CA certificate resides in build/pgo/certs/pgoca.p12 (PKCS12 module) and pgoca.ca (the issuer certificate itself) and both files are permanently stored to the repository. The CA certificate is at run-time added as trusted to the certificate database of the testing profile what ensures no need to manually add an exception when you wish to use the common list of https hosts.

The default server certificate resides in the certificate database stored permanently to the repository, at build/pgo/certs directory of the source tree. The certificate is based on the list in server-locations.txt and is signed by the certification authority certificate described above. Each https host is added to the server certificate's subject as an alternative domain name (SubjectAltName) what makes the host accessible.

In general, any file in build/pgo/certs with extension '.ca' is considered a certification authority certificate in ASCII BASE64 format and is added as trusted to the testing profile. The default CA is such a file.

How can I customize the list of https hosts or create my own server or CA certificates?


To simply add a new https host for which the default server certificate will be used just add it to the list in server-locations.txt. The content is self explanatory. To reflect these changes run:

    obj-dir/build/pgo$ make genservercerts

To add a new https host that has no server certificate assigned, even the default one, just add an option 'nocert' after the host name, like this:
# This is a host without any certificate
https://nocert.example.com:443         privileged,nocert


Note: this host has already been present in the list of hosts.

To add a new host with a new server certificate, different from the default one, you have to generate or provide the certificate and bound it with the host. Just decide if you want to sign it by the prepared CA certificate or create a new CA certificate by your self to sign the new server certificate or create a self-signed server certificate or add your own certificate and key you already have.

Note: to generate new certificates it is most appropriate to use certutil and pk12util binaries built as part of a test-enabled build. These utils reside in obj-dir/dist/bin.

Note: you will change the certificate database while creating a new certificate. Thus, don't forget to commit the new certificate database with the new test.

 

I. Use the prepared CA to create a new server certificate.

Note: in this case keep the command to generate your server certificate for case when the prepared CA get renewed. You must then generate the server certificate again.

To generate your new certificate: (please replace "My" names with something meaningful)

    build/pgo/certs$ certutil -S -n "MyNewTestingServerCert" -s "CN=my-new-testing-host.com" -c "pgo temporary ca" -t "P,," -k rsa -g 1024 -m <LARGE RANDOM NUMBER> -v 120 -d .

      follow on-screen instructions, "pgo temporary ca" is nickname of the prepared CA present in the database
      IMPORTANT NOTE: make sure you allocate a new unique serial number (the '-m' option). If there were two or more certificates with the same
serial number signed by the pgo CA the ssltunnel server would not start! It is best to assign a large random number.

II. Create my own CA and server certificate:

1. create a new CA certificate: (please replace "My" names with something meaningful)

      build/pgo/certs$ certutil -S -s "MyNewCertificationAuthority" -s "CN=My Certificate Authority" -t "C,," -x -m 1 -v 120 -n "my certificate authority" -2 -d .
      type 'Y', confirm with enter
      type '0' (zero), confirm with enter
      type 'N', confirm with enter

2. generate the server certificate as described in previous section, just make sure to change the '-c "pgo temporary ca"' to '-c "my certificate authority"' to reflect you want to sign the server certificate by your new CA certificate

3. export the CA certificate:

    build/pgo/certs$ certutil -L -d . -n "my certificate authority" -a -o mycertauth.ca
      add this file to the repository, it must be checked in with the new test

      optionally, if you wish to save the CA cert with its private key, do:
    build/pgo/certs$ pk12util -o myca.p12 -n "my certificate authority" -d .
      provide any password you wish

      to delete your CA certificate from the database (recommended), do:
    build/pgo/certs$ certutil -D -n "my certificate authority" -d .

III. Create self signed server certificate:

to generate a self signed server certificate: (please replace "My" names with something meaningful)

    build/pgo/certs$ certutil -S -n "MySelfSignedServerCertificate" -s "CN=my-new-testing-host.com" -x -t "P,," -m 1 -v 120 -d .

IV. Add my own certificate and private key I already have

1. import the existing server certificate and its private key from PKCS12 module:

    build/pgo/certs$ pk12util -i my-own-server-cert.p12 -d .
      enter correct password of the PKCS12 module, nickname of the cert will be used to refer it in server-locations.txt, see bellow

2. optionally, if there is also a CA certificate the server certificate is signed with and you need that CA be trusted, copy this CA to build/pgo/certs and give it a '.ca' extension, the file should be an X.509 certificate in ASCII BASE64 formatting, make sure to commit this file as well


Now you must let Mochitest know about your new server certificate. Edit the server-locations.txt as follows:
    # My testing host and its bond to certificate nickname in the certificate database
    https://my-new-testing-host.com:443   privileged,cert=MyNewTestingServerCert

Run make at obj-dir/testing/mochitest to reflect the changes. If the build fails it is possible you created a certificate with a same serial number as another certificate signed by the same certification authority or two server certificates has a same nickname.

How to re-generate the temporary certification authority (CA) certifcate?


Use genpgocert.py script like this: obj-dir/_profile/pgo$ genpgocert.py --gen-ca. This will re-create a CA with a different public and private key pair valid from that moment for next 10 years. You must make testing/mochitest after this operation to reflect the changes. Don't forget to commit changed pgoca.ca and pgoca.p12 files to the repository. Because the CA's public key changes during this operation keep a command generating your server certificates based on this CA to regenerate them easily. Signatures of your server certificates will become invalid after re-generation of the CA.

Note: I will try to enhance the script to allow preservation of the key pair.

FAQ

How do I find an error in the log?

Search for the string "ERROR FAIL" to find unexpected failures. You can also search for "SimpleTest FINISHED" to see the final test summary. This is particularly useful when viewing full Tinderbox logs, since the Mochitest output isn't necessarily at the end of the combined log.

What if my tests have failures in them?

You still have to test that. Mochitest provides a todo() function that is identical to ok(), but is expected to fail. We've also added todo_is() and todo_isnot() to match is() and isnot().

What if my tests aren't done when onload fires?

Call SimpleTest.waitForExplicitFinish() before onload fires. Then, when you're done, call SimpleTest.finish().

What if I need to change a preference to run my test?

netscape.security.PrivilegeManager.enablePrivilege("UniversalXPConnect");
var prefService = Components.classes["@mozilla.org/preferences-service;1"]
                            .getService(Components.interfaces.nsIPrefService);
var domBranch = prefService.getBranch("dom.");
var oldVal = domBranch.getIntPref("max_script_run_time");
domBranch.setIntPref("max_script_run_time", 0);

// do what you need

domBranch.setIntPref("max_script_run_time", oldVal);

Can tests be run under a chrome URL?

Yes, use python runtests.py --chrome. Keep in mind that the xpcshell test harness should be your first choice for XPCOM testing. Only use mochitest if you need events, browser features, networking, etc.

How can I get around the error "Permission denied to get property XPCComponents.classes"?

Adding the following line to your test file (and each event handler) will allow full XPCOM usage.

netscape.security.PrivilegeManager.enablePrivilege('UniversalXPConnect');

This approach is obviously inconvenient. That's why we're working on the build hacking necessary to copy tests into a chrome directory for testing.

How do I change the HTTP headers or status sent with a file used in a Mochitest?

Create a text file next to the file whose headers you want to modify. The name of the text file should be the name of the file whose headers you're modifying followed by ^headers^. For example, if you have a file foo.jpg, the text file should be named foo.jpg^headers^. (Don't try to actually use the headers file in any other way in the test, because the HTTP server's hidden-file functionality prevents any file ending in exactly one ^ from being served.) Edit the file to contain the headers and/or status you want to set, like so:

HTTP 404 Not Found
Content-Type: text/html
Random-Header-of-Doom: 17

The first line sets the HTTP status and (optionally a) description associated with the file. This line is optional; you don't need it if you're fine with the normal response status and description. Any other lines in the file describe additional headers which you want to add or overwrite (most typically the Content-Type header, for the latter case) on the response. The format follows the conventions of HTTP, except that you don't need to have HTTP line endings and you can't use a header more than once (the last line for a particular header wins). The file may end with at most one blank line to match Unix text file conventions, but the trailing newline isn't strictly necessary.

How do I test issues which only show up when tests are run across domains?

The Mochitest harness runs one web server to serve tests, but through the magic of proxy autoconfig, all test files are available on a variety of different domains and ports. Tests running on any of these servers (with two exceptions for testing privilege escalation functionality) automatically have the ability to request elevated privileges such as UniversalXPConnect. The full list of domains and ports on which tests are served, all of which serve exactly the same content as http://localhost:8888, is specified in {{ Source("build/pgo/server-locations.txt") }}.

Unfortunately, there is currently no support for running tests over non-HTTP protocols such as FTP or HTTPS in ways that are useful for cross-domain testing. This limitation will probably be rectified in the future.

How do I write tests that check header values, method types, etc. of HTTP requests?

To write such a test, you simply need to write an SJS (server-side JavaScript) for it. An SJS is simply a JavaScript file with the extension sjs which is loaded in a sandbox; the global property handleRequest defined by the script is then executed with request and response objects, and the script populates the response based on the information in the request.

Here's an example of a simple SJS:

function handleRequest(request, response)
{
  // avoid confusing cache behaviors
  response.setHeader("Cache-Control", "no-cache", false);

  response.setHeader("Content-Type", "text/plain", false);
  response.write("Hello world!");
}

The exact properties of the request and response parameters are defined in the nsIHttpRequestMetadata and nsIHttpResponse interfaces in {{ Source("netwerk/test/httpserver/nsIHttpServer.idl", "nsIHttpServer.idl") }}. Note carefully: the browser is free to cache responses generated by your script, so if you ever want an SJS to return different data for multiple requests to the same URL, you should add a Cache-Control: no-cache header to the response to prevent the test from accidentally failing if it's manually run multiple times in the same Mochitest session.

A simple example of an SJS used in reftests is {{ Source("modules/libpr0n/test/reftest/generic/check-header.sjs", "check-header.sjs") }}.

{{ languages( { "ja": "ja/Mochitest" } ) }}

Revision Source

<p>Mochitest is an automated testing framework built on top of the <a class="external" href="http://mochikit.com/">MochiKit</a> JavaScript libraries. It's just one of the automated regression testing frameworks used by Mozilla. Tests report success or failure to the test harness using JavaScript function calls.</p>
<p>Mochitest's unique strength is that it runs tests written as webpages in a full browser environment where the tests have chrome (elevated) privileges. This allows JavaScript in the tests to do much, much more than it would otherwise be able to do. In addition to the capabilities a script would normally have (e.g. DOM manipulation), scripts can access XPCOM components and services, and even access the browser itself. This allows a script to, say, simulate user input to the browser's user interface, before examining the browser to verify that the input had the intended results.</p>
<p>Mochitest's use of JavaScript function calls to communicate test success or failure can make it unsuitable for certain types of test. Only things that can in some way be tested using JavaScript (with chrome privileges!) can be tested with this framework. Given some creativity, that's actually much more than you might first think, but it's not possible to write Mochitest tests to directly test a non-scripted C++ component, for example.  (Use a <a class="internal" href="/En/Compiled-code%20automated%20tests" title="En/Compiled-code automated tests">compiled-code test</a> to do that.)</p>
<h3 name="Running_tests">Running tests</h3>
<p>The Mozilla build machines run Mochitest as part of the build process, so we get to know pretty quickly if someone commits a change to the source code that breaks something. However, you should still run Mochitest yourself before you commit any risky new code. You don't want to be the one who wastes everyone's time by breaking the tree if you can help it. :-)</p>
<h4 name="Running_the_whole_test_suite">Running the whole test suite</h4>
<p>To run Mochitest, first <a href="/en/Build_Documentation" title="en/Build_Documentation">build Mozilla</a> with your changes; then run Mochitest's <code>runtests.py</code> script without passing it any command line arguments:</p>
<pre class="eval">cd $(OBJDIR)/_tests/testing/mochitest
python runtests.py
</pre>
<p>This will open your build with a document containing a "Run Tests" link at the top. To run the tests simply click this link and watch the results being generated. Test pass/fail is reported for each test as it runs and is recorded on the page.</p>
<p><img alt="Image:Mochitest.png" class="internal" src="/@api/deki/files/269/=Mochitest.png"></p>
<p><strong>Note:</strong> you should keep focus on the browser window while the test are being run, as some may fail otherwise (like the one for {{ Bug("330705") }} for example). Linux users can save themselves this inconvenience by using a dummy X server (see <a href="#Diverting_X_output">Diverting X output</a> below).</p><h4 name="Running_select_tests">Running select tests</h4>
<p>To run a single test (perhaps a new test you just added) or a subset of the entire Mochitest suite, use the <code>--test-path</code> option to specify the test or the subdirectory of tests that you want to run. For example, to run only the test {{ Source("content/base/test/test_CrossSiteXHR.html", "test_CrossSiteXHR.html") }} in the Mozilla source tree, you would run this command:</p>
<pre class="eval">python runtests.py --test-path=content/base/test/test_CrossSiteXHR.html
</pre>
<p>To run all the tests in {{ Source("content/svg/") }}, this command would work:</p>
<pre class="eval">python runtests.py --test-path=content/svg/
</pre>
<p>Note that the path specified by the <code>--test-path</code> is the path to the test or directory within the Mozilla source tree. If the path is a directory, then the tests in that directory and all of its subdirectories will be loaded.</p>
<h4 name="Logging_results">Logging results</h4>
<p>The output from a test run can be sent to the console and/or a file (by default the results are only displayed in the browser). There are several levels of detail to choose from. The levels are DEBUG, INFO, WARNING, ERROR and FATAL, where DEBUG produces the highest detail (everything), and FATAL produces the least (a message will only be logged for an event that causes the test run to abort prematurely).</p>
<p>To log to a file use --log-file=FILE. By default the file logging level is INFO but you can change this using --file-level=LEVEL.</p>
<p>To turn on logging to the console use --console-level=LEVEL.</p>
<p>For example, to log test run output to the file <code>~/mochitest.log</code> at DEBUG level detail you would use:</p>
<pre class="eval">python runtests.py --log-file=~/mochitest.log --file-level=DEBUG
</pre>
<h4 name="Diverting_X_output">Diverting X output</h4>
<p>The tests must run in a focused window, which effectively prevents any other user activity on the engaged computer. Linux users can reclaim their boxes by telling the suite to use a hidden virtual desktop. If <a class="external" href="http://en.wikipedia.org/wiki/Xvfb">Xvfb</a> is or can be installed, the following command launches the tests without blocking the active session:</p>
<pre class="eval">nice xvfb-run python runtests.py --log-file=~/mochitest.log --file-level=DEBUG --autorun --close-when-done --console-level=DEBUG
</pre>
<p>Other possible configurations have also been discussed in {{ Bug("434365") }}.</p>
<h4 name="Other_.27runtests.27_options">Other <code>'runtests'</code> options</h4>
<p>The 'runtests' script recognizes several other options - use the --help option to get a list. Note that there is separate documentation for the <a href="/en/Chrome_tests" title="en/Chrome_tests">--chrome</a>, <a href="/en/Browser_chrome_tests" title="en/Browser_chrome_tests">--browser-chrome</a> and <a href="/en/Accessibility" title="en/Accessibility">--a11y</a> options.</p>
<h3 name="Writing_tests">Writing tests</h3>
<p>A Mochitest test is simply an HTML, XHTML or XUL file that contains some JavaScript to test for some condition(s).</p>
<p>You can use <a class="external" href="http://ted.mielczarek.org/code/mozilla/mochitest-maker/">Mochitest maker</a> to run most tests without having to build Mozilla.</p>
<h4 name="Try_to_avoid_Mochitest">Try to avoid Mochitest</h4>
<p>Yes, really. For many things Mochitest is overkill. In general you should always try to use one of the lighter-weight testing frameworks. For example, if you only want to test a single XPCOM component then you should use <a href="/en/Writing_xpcshell-based_unit_tests" title="en/Writing_xpcshell-based_unit_tests">xpcshell</a>. On the other hand there are some things that Mochitest cannot do, or isn't designed to do. For example, for visual output tests you should try to use the <a href="/en/Creating_reftest-based_unit_tests" title="en/Creating_reftest-based_unit_tests">reftest</a> framework. For more information on the different types of automated testing frameworks see <a href="/en/Mozilla_automated_testing" title="en/Mozilla_automated_testing">Mozilla automated testing</a>.</p>
<h4 name="Test_templates">Test templates</h4>
<p>You can avoid typing out boilerplate by using the {{ Source("testing/mochitest/gen_template.pl", "gen_template") }} perl script to generate a test template. This script takes two optional arguments:</p>
<ol> <li>-b : a bug number</li> <li>-type : template type. {html|xhtml|xul}. defaults to html.</li>
</ol>
<p>For example:</p>
<pre class="eval">cd mozilla/testing/mochitest/
perl gen_template.pl -b=123456 &gt; path/to/test_bug123456.html
perl gen_template.pl -b=123456 --type=xul &gt; path/to/test_bug123456.xul
</pre>
<p>Note that Mochitest requires the file name of all tests to begin with the string "test_". See the section below for help on deciding where your tests should go in the tree.</p>
<p>The elements with id 'content' and 'display' in the generated file can be used by your script if you need elements to mess around with. TODO: uh, would someone care to expand on that?</p>
<h4 name="Test_functions">Test functions</h4>
<p>Each test must contain some JavaScript that will run and tell Mochitest whether the test has passed or failed. [<a class=" external" href="http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/MochiKit/Test.js" rel="freelink">http://mxr.mozilla.org/mozilla-centr...chiKit/Test.js</a> MochiKit/Test.js] provides a number of functions for the test to use that communicate the pass/fail to Mochitest. These include:</p>
<ul> <li><code>ok(expressionThatShouldBeTrue, "Error message")</code> -- tests a value for truthiness</li> <li><code>is(thingA, thingB, "Error message")</code> -- compares two values (using ==, not ===)</li> <li><code>isnot(thingA, thingB, "Error message")</code> -- opposite of is()</li>
</ul>
<p>See the {{ Source("testing/mochitest/README.txt", "README") }} for an example of their use.</p>
<p>If you want to include a test for something that currently fails, then instead of commenting it out, you should use one of the "todo" equivalents so Tinderbox can notice if it suddenly starts passing:</p>
<ul> <li><code>todo(falseButShouldBeTrue, "Error message")</code></li> <li><code>todo_is(thingA, thingB, "Error message")</code></li> <li><code>todo_isnot(thingA, thingB, "Error message")</code></li>
</ul>
<h4 name="Helper_functions">Helper functions</h4>
<p>Right now all of Mochikit is available (this will change in {{ Bug("367393") }}); {{ Bug("367569") }} added <code>sendChar</code>, <code>sendKey</code>, and <code>sendString</code> helpers. These are available in {{ Source("testing/mochitest/tests/SimpleTest/EventUtils.js") }}.</p>
<h3 name="Adding_tests_to_the_tree">Adding tests to the tree</h3>
<p>Once you've written a new test you need to add it to the Mozilla source tree and tell the build system about it so that the Mozilla tinderboxes will run it automatically.</p>
<h4 name="Choosing_a_location">Choosing a location</h4>
<p>New Mochitest tests should go somewhere close to the code they are testing, hopefully in the same module, so that ownership of the test cases is clear. For example, if you create a new test for some HTML feature, you probably want to put the test in {{ Source("content/html/content/test") }} or {{ Source("content/html/document/test") }}. If a test directory does not exist near the code you are testing you can add a new test directory as the patch in {{ Bug("368531") }} demonstrates.</p>
<h4 name="Makefile_changes">Makefile changes</h4>
<p>To tell the build system about your new test you need to add the name of your test file to <code>_TEST_FILES</code> in the test directory's <code>Makefile.in</code>.</p>
<p>If your test spans multiple files, only name the main one "test_...". This is the one that will show up in the list of testcases to run. The other files should have some other name, but must still be added to the <code>_TEST_FILES</code> in <code>Makefile.in</code>.</p>
<p>Keep in mind that if you're adding chrome tests, you'll need to change the Makefile to install the tests in <code>_tests/testing/mochitest/<strong>chrome</strong></code> rather than <code>_tests/testing/mochitest/<strong>tests</strong></code>.</p>
<h4 name="Building_and_running_new_tests">Building and running new tests</h4>
<p>Before committing a new tests you should check that the Makefile.in changes are correct and that your tests pass as you expect them to. To check your test, first export it to the Mochitest directory by running the command:</p>
<pre class="eval">make
</pre>
<p>in the object directory corresponding to the test file's location in the source tree. Now open Mochitest as described above, but this time, instead of clicking on the "Run Tests" link, search for your test and click on it.</p>
<h3 name="FAQ">SSL and https enabled tests</h3>
<br>
<p>Mochitests must be run from <code><span class="nowiki">http://localhost:8888</span></code> to succeed; however, some tests may require use of additional protocols, hosts, or ports to test cross-origin functionality.  The Mochitest harness addresses this need by <em>mirroring</em> all content of the original server onto a variety of other servers through the magic of proxy autoconfig and SSL tunneling.  The full list of schemes, hosts, and ports on which tests are served, all of which serve exactly the same content as <code><span class="nowiki">http://localhost:8888</span></code>, is specified in {{ Source("build/pgo/server-locations.txt") }}.  Note, however, that not all origins described there are equivalent: some specify particular SSL certificates for testing purposes, while some allow pages on that server to request elevated privileges; read the file for full details.</p>
<h4>How Mochitest SSL server works?</h4>
<p><br>
Mochitest (better say, the Profile Guided Optimization) is using FindProxyForURL JS function as a proxy configuration function to interactively provide the listed hosts. It also, when accessing an https host in the list, redirects transparently to ssltunnel server program that is configured to act as an HTTPS proxy and forwarding tunnel. That is, it receives the 'CONNECT host' request from the browser, checks the host is on the list, if so, answers with '200 CONNECTED' response and switches to pure SSL tunnel. Ssltunnel then acts as the remote host server. Accepts SSL session, identifies it self with the proper server certificate for the host and from now on just forwards all encrypted communication between the browser and the ssltunnel in unencrypted form to the common testing server listening on localhost:8888, transparently.</p>
<h4>How are SSL certificates managed?</h4>
<p><br>
For default SSL Mochitest capability there is a pair of a persistent testing certification authority certificate and an on-demand generated server certificate. This schema provides simulation of a common model of a trusted certificate issuer and a trusted server in the Internet.<br>
<br>
The certification authority (CA) certificate is a pre-generated self-signed certificate, valid for 10 years. This CA certificate resides in build/pgo/certs/pgoca.p12 (PKCS12 module) and pgoca.ca (the issuer certificate itself) and both files are permanently stored to the repository. The CA certificate is at run-time added as trusted to the certificate database of the testing profile what ensures no need to manually add an exception when you wish to use the common list of https hosts.<br>
<br>
The default server certificate resides in the certificate database stored permanently to the repository, at build/pgo/certs directory of the source tree. The certificate is based on the list in server-locations.txt and is signed by the certification authority certificate described above. Each https host is added to the server certificate's subject as an alternative domain name (SubjectAltName) what makes the host accessible.</p>
<p>In general, any file in build/pgo/certs with extension '.ca' is considered a certification authority certificate in ASCII BASE64 format and is added as trusted to the testing profile. The default CA is such a file.</p>
<h4>How can I customize the list of https hosts or create my own server or CA certificates?</h4>
<p><br>
To <span style="color: rgb(51, 102, 255);">simply add a new https host</span> for which the default server certificate will be used just add it to the list in server-locations.txt. The content is self explanatory. To reflect these changes run:</p>
<p><code>    obj-dir/build/pgo$ make genservercerts</code></p>
<p>To <span style="color: rgb(51, 102, 255);">add a new https host that has no server certificate assigned</span>, even the default one, just add an option 'nocert' after the host name, like this:<br>
<code># This is a host without any certificate<br>
<a class=" link-https" href="https://nocert.example.com:443" rel="freelink">https://nocert.example.com:443</a>         privileged,nocert</code><br>
<br>
<span style="color: rgb(153, 51, 0);">Note:</span> this host has already been present in the list of hosts.<br>
<br>
To <span style="color: rgb(51, 102, 255);">add a new host with a new server certificate, different from the default one</span>, you have to generate or provide the certificate and bound it with the host. Just decide if you want to sign it by the prepared CA certificate or create a new CA certificate by your self to sign the new server certificate or create a self-signed server certificate or add your own certificate and key you already have.<br>
<br>
<span style="color: rgb(153, 51, 0);">Note:</span> to generate new certificates it is most appropriate to use <span style="color: rgb(153, 51, 0);"><strong>certutil </strong></span>and <span style="color: rgb(153, 51, 0);"><strong>pk12util </strong></span>binaries built as part of a test-enabled build. These utils reside in obj-dir/dist/bin.</p>
<p><span style="color: rgb(153, 51, 0);">Note:</span> you will change the certificate database while creating a new certificate. Thus, don't forget to commit the new certificate database with the new test.</p>
<p> </p>
<p><strong><em>I. Use the prepared CA to create a new server certificate.</em></strong></p>
<p><span style="color: rgb(153, 51, 0);">Note:</span> in this case keep the command to generate your server certificate for case when the prepared CA get renewed. You must then generate the server certificate again.</p>
<p>To generate your new certificate: (please replace "My" names with something meaningful)</p>
<p><code>    build/pgo/certs$ certutil -S -n "MyNewTestingServerCert" -s "CN=my-new-testing-host.com" -c "pgo temporary ca" -t "P,," -k rsa -g 1024 -m &lt;LARGE RANDOM NUMBER&gt; -v 120 -d .</code></p>
<p>      follow on-screen instructions, "pgo temporary ca" is nickname of the prepared CA present in the database<br>
    <span style="color: rgb(255, 0, 0);">  IMPORTANT NOTE:</span> make sure you allocate a new unique serial number (the '-m' option). If there were two or more certificates with the same<br>
serial number signed by the pgo CA the ssltunnel server would not start! It is best to assign a large random number.<br>
<br>
<strong><em>II. Create my own CA and server certificate:</em></strong></p>
<p>1. create a new CA certificate: (please replace "My" names with something meaningful)</p>
<p>      <code>build/pgo/certs</code><code>$ certutil -S -s "MyNewCertificationAuthority" -s "CN=My Certificate Authority" -t "C,," -x -m 1 -v 120 -n "my certificate authority" -2 -d .<br>
</code>       type 'Y', confirm with enter<br>
      type '0' (zero), confirm with enter<br>
      type 'N', confirm with enter<br>
<br>
2. generate the server certificate as described in previous section, just make sure to change the '-c "pgo temporary ca"' to '-c "my certificate authority"' to reflect you want to sign the server certificate by your new CA certificate</p>
<p>3. export the CA certificate:</p>
<p><code>    </code><code>build/pgo/certs</code><code>$ certutil -L -d . -n "my certificate authority" -a -o mycertauth.ca<br>
</code>       add this file to the repository, it must be checked in with the new test<br>
<br>
      optionally, if you wish to save the CA cert with its private key, do:<br>
<code>    </code><code>build/pgo/certs</code><code>$ pk12util -o myca.p12 -n "my certificate authority" -d .<br>
</code>      provide any password you wish</p>
<p>      to delete your CA certificate from the database (recommended), do:<br>
<code>    </code><code>build/pgo/certs</code><code>$ certutil -D -n "my certificate authority" -d .<br>
</code><br>
<strong><em>III. Create self signed server certificate:</em></strong></p>
<p>to generate a self signed server certificate: (please replace "My" names with something meaningful)</p>
<p><code>    build/pgo/certs</code><code>$ certutil -S -n "MySelfSignedServerCertificate" -s "CN=my-new-testing-host.com" -x -t "P,," -m 1 -v 120 -d .<br>
</code> <br>
<strong><em>IV. Add my own certificate and private key I already have</em></strong></p>
<p>1. import the existing server certificate and its private key from PKCS12 module:</p>
<p><code>    </code><code>build/pgo/certs</code><code>$ pk12util -i my-own-server-cert.p12 -d .<br>
</code>       enter correct password of the PKCS12 module, nickname of the cert will be used to refer it in server-locations.txt, see bellow<br>
<br>
2. optionally, if there is also a CA certificate the server certificate is signed with and you need that CA be trusted, copy this CA to build/pgo/certs and give it a '.ca' extension, the file should be an X.509 certificate in ASCII BASE64 formatting, make sure to commit this file as well<br>
<br>
<br>
Now <span style="color: rgb(51, 102, 255);">you must let Mochitest know about your new server certificate</span>. Edit the server-locations.txt as follows:<br>
<code>    # My testing host and its bond to certificate nickname in the certificate database<br>
    <a class=" link-https" href="https://my-new-testing-host.com:443" rel="freelink">https://my-new-testing-host.com:443</a>   privileged,cert=MyNewTestingServerCert<br>
</code><br>
Run <code>make</code> at <code>obj-dir/testing/mochitest</code> to reflect the changes. If the build fails it is possible you created a certificate with a same serial number as another certificate signed by the same certification authority or two server certificates has a same nickname.</p>
<h4>How to re-generate the temporary certification authority (CA) certifcate?</h4>
<p><br>
Use genpgocert.py script like this: <code> obj-dir/_profile/pgo$ genpgocert.py --gen-ca. </code> This will re-create a CA with a different public and private key pair valid from that moment for next 10 years. You must make testing/mochitest after this operation to reflect the changes. Don't forget to commit changed pgoca.ca and pgoca.p12 files to the repository. Because the CA's public key changes during this operation keep a command generating your server certificates based on this CA to regenerate them easily. Signatures of your server certificates will become invalid after re-generation of the CA.<br>
<br>
Note: I will try to enhance the script to allow preservation of the key pair.</p><h3 name="FAQ">FAQ</h3>
<h4 name="How_do_I_find_an_error_in_the_log.3F">How do I find an error in the log?</h4>
<p>Search for the string "ERROR FAIL" to find unexpected failures. You can also search for "SimpleTest FINISHED" to see the final test summary. This is particularly useful when viewing full Tinderbox logs, since the Mochitest output isn't necessarily at the end of the combined log.</p>
<h4 name="What_if_my_tests_have_failures_in_them.3F">What if my tests have failures in them?</h4>
<p>You still have to test that. Mochitest provides a <code>todo()</code> function that is identical to <code>ok()</code>, but is expected to fail. We've also added <code>todo_is()</code> and <code>todo_isnot()</code> to match <code>is()</code> and <code>isnot()</code>.</p>
<h4 name="What_if_my_tests_aren.27t_done_when_onload_fires.3F">What if my tests aren't done when onload fires?</h4>
<p>Call <code>SimpleTest.waitForExplicitFinish()</code> before onload fires. Then, when you're done, call <code>SimpleTest.finish()</code>.</p>
<h4 name="What_if_I_need_to_change_a_preference_to_run_my_test.3F">What if I need to change a preference to run my test?</h4>
<pre class="eval">netscape.security.PrivilegeManager.enablePrivilege("UniversalXPConnect");
var prefService = Components.classes["@mozilla.org/preferences-service;1"]
                            .getService(Components.interfaces.nsIPrefService);
var domBranch = prefService.getBranch("dom.");
var oldVal = domBranch.getIntPref("max_script_run_time");
domBranch.setIntPref("max_script_run_time", 0);

// do what you need

domBranch.setIntPref("max_script_run_time", oldVal);
</pre>
<h4 name="Can_tests_be_run_under_a_chrome_URL.3F">Can tests be run under a chrome URL?</h4>
<p>Yes, use <code>python runtests.py --chrome</code>. Keep in mind that the <a href="/en/Writing_xpcshell-based_unit_tests" title="en/Writing_xpcshell-based_unit_tests">xpcshell test harness</a> should be your first choice for XPCOM testing. Only use mochitest if you need events, browser features, networking, etc.</p>
<h4 name="How_can_I_get_around_the_error_.22Permission_denied_to_get_property_XPCComponents.classes.22.3F">How can I get around the error "Permission denied to get property XPCComponents.classes"?</h4>
<p>Adding the following line to your test file (and each event handler) will allow full XPCOM usage.</p>
<p><code> netscape.security.PrivilegeManager.enablePrivilege('UniversalXPConnect'); </code></p>
<p>This approach is obviously inconvenient. That's why we're working on the build hacking necessary to copy tests into a chrome directory for testing.</p>
<h4 name="How_do_I_change_the_HTTP_headers_or_status_sent_with_a_file_used_in_a_Mochitest.3F">How do I change the HTTP headers or status sent with a file used in a Mochitest?</h4>
<p>Create a text file next to the file whose headers you want to modify. The name of the text file should be the name of the file whose headers you're modifying followed by <code>^headers^</code>. For example, if you have a file <code>foo.jpg</code>, the text file should be named <code>foo.jpg^headers^</code>. (Don't try to actually use the headers file in any other way in the test, because the HTTP server's hidden-file functionality prevents any file ending in exactly one <code>^</code> from being served.) Edit the file to contain the headers and/or status you want to set, like so:</p>
<pre class="eval">HTTP 404 Not Found
Content-Type: text/html
Random-Header-of-Doom: 17
</pre>
<p>The first line sets the HTTP status and (optionally a) description associated with the file. This line is optional; you don't need it if you're fine with the normal response status and description. Any other lines in the file describe additional headers which you want to add or overwrite (most typically the Content-Type header, for the latter case) on the response. The format follows the conventions of HTTP, except that you don't need to have HTTP line endings and you can't use a header more than once (the last line for a particular header wins). The file may end with at most one blank line to match Unix text file conventions, but the trailing newline isn't strictly necessary.</p>
<h4 name="How_do_I_test_issues_which_only_show_up_when_tests_are_run_across_domains.3F">How do I test issues which only show up when tests are run across domains?</h4>
<p>The Mochitest harness runs one web server to serve tests, but through the magic of proxy autoconfig, all test files are available on a variety of different domains and ports. Tests running on any of these servers (with two exceptions for testing privilege escalation functionality) automatically have the ability to request elevated privileges such as UniversalXPConnect. The full list of domains and ports on which tests are served, all of which serve exactly the same content as <code><span class="nowiki">http://localhost:8888</span></code>, is specified in {{ Source("build/pgo/server-locations.txt") }}.</p>
<p>Unfortunately, there is currently no support for running tests over non-HTTP protocols such as FTP or HTTPS in ways that are useful for cross-domain testing. This limitation will probably be rectified in the future.</p>
<h4 name="How_do_I_write_tests_that_check_header_values.2C_method_types.2C_etc._of_HTTP_requests.3F">How do I write tests that check header values, method types, etc. of HTTP requests?</h4>
<p>To write such a test, you simply need to write an SJS (server-side JavaScript) for it. An SJS is simply a JavaScript file with the extension <code>sjs</code> which is loaded in a sandbox; the global property <code>handleRequest</code> defined by the script is then executed with request and response objects, and the script populates the response based on the information in the request.</p>
<p>Here's an example of a simple SJS:</p>
<pre class="eval">function handleRequest(request, response)
{
  // avoid confusing cache behaviors
  response.setHeader("Cache-Control", "no-cache", false);

  response.setHeader("Content-Type", "text/plain", false);
  response.write("Hello world!");
}
</pre>
<p>The exact properties of the request and response parameters are defined in the <code>nsIHttpRequestMetadata</code> and <code>nsIHttpResponse</code> interfaces in <code>{{ Source("netwerk/test/httpserver/nsIHttpServer.idl", "nsIHttpServer.idl") }}</code>. Note carefully: the browser is free to cache responses generated by your script, so if you ever want an SJS to return different data for multiple requests to the same URL, you should add a <code>Cache-Control: no-cache</code> header to the response to prevent the test from accidentally failing if it's manually run multiple times in the same Mochitest session.</p>
<p>A simple example of an SJS used in reftests is <code>{{ Source("modules/libpr0n/test/reftest/generic/check-header.sjs", "check-header.sjs") }}</code>.</p>
<p>{{ languages( { "ja": "ja/Mochitest" } ) }}</p>
Revert to this revision