Mochitest

  • Revision slug: Mochitest
  • Revision title: Mochitest
  • Revision id: 9787
  • Created:
  • Creator: DBaron
  • Is current revision? No
  • Comment /* How do I run just one test? */ add question

Revision Content

Introduction

Mochitest is an automated testing framework built on top of the MochiKit JavaScript libraries. It's just one of the automated regression testing facilities Mozilla developers have at their disposal. Tests report success or failure to the test harness using JavaScript function calls.

Mochitest's unique strength is that it runs tests written as webpages in a full browser environment where the tests have chrome (elevated) privileges. This allows JavaScript in the tests to do much, much more than it would otherwise be able to do. In addition to the capabilities a script would normally have (e.g. DOM manipulation), scripts can access XPCOM components and services, and even access the browser itself. This allows a script to, say, simulate user input to the browser's user interface, before examining the browser to verify that the input had the intended results.

Mochitest's use of JavaScript function calls to communicate test success or failure can make it unsuitable for certain types of test. Only things that can in some way be tested using JavaScript (with chrome privileges!) can be tested with this framework. Given some creativity, that's actually much more than you might first think, but it's not possible to write Mochitest tests to directly test a non-scripted C++ component, for example.

Try not to use Mochitest

Yes, really. For many things Mochitest is overkill. In general you should always try to use one of the lighterweight testing frameworks. For example, if you only want to test a single XPCOM component then you should use xpcshell. On the other hand there are some things that Mochitest cannot do, or isn't designed to do. For example, for visual output tests you should try to use the reftest framework. For more information on the different types of automated testing frameworks see Mozilla automated testing.

Running Mochitest

The Mozilla tinderboxes now run Mochitest as part of the build process, so anyone committing changes to the source will know pretty quickly if they've broken something. Nevertheless, it is a good idea to run Mochitest before committing new code. This is especially true if the code changes are large or would be difficult to back out.

To run Mochitest you need to build Mozilla with your changes, change directory to $(OBJDIR)/_tests/testing/mochitest, and run the command:

> perl runtests.pl

This will open your new build with a document containing a "Run Tests" link at the top. To run the tests simply click this link and watch the results being generated. If any tests have failed this will be reported.

Writing new Mochitest tests

A basic document on Writing MochiTest-based unit tests has already been started. (Maybe the basic stuff from that document should be moved here and that document should give the matter more detailed treatment?)

Adding new Mochitest tests to the tree

Once you've written a new test you need to add it to the Mozilla source tree and tell the build system about it so that the Mozilla tinderboxes will run it automatically.

New Mochitest tests should go somewhere close to the code they are testing. For example, if you create a new test for some HTML feature, you probably want to put the test in the directory mozilla/content/html/content/test or mozilla/content/document/content/test. If a test directory does not exist near the code you are testing you can add a new test directory as the patch in bug 368531 demonstrates.

To tell the build system about your new test you need to add the name of your test file to _TEST_FILES in the test directory's Makefile.in.

Before committing your new test and the Makefile.in changes, do run Mochitest in an up to date trunk build to check that you will not unexpectedly turn the tree orange.

Testing new tests

Before committing a new test you should check that it does actually pass as you expect it to. To check your test, first export it to the Mochitest directory by running the command:

> make

in the object directory corresponding to the test file's location in the source tree. Now open Mochitest as described above, but this time, instead of clicking on the "Run Tests" link, search for your test and click on it.

FAQ

How do I get started?

Start by reading the introductory material provided by the article Mochitest, as well as by {{wiki.template('Named-source', [ "testing/mochitest/README.txt", "reading the README" ])}}.

  • <tt>mozilla/testing/mochitest</tt> should be checked out as part of the normal checkout on trunk.
  • mochitest is built by default (unless <tt>--disable-mochitest</tt> is specified in mozconfig) and installed into <tt>($OBJ_DIR)/_tests/testing/mochitest</tt>
  • To run tests
    • <tt>cd ($OBJ_DIR)/_tests/testing/mochitest</tt> (Note that the working directory matters for runtests.pl!)
    • <tt>./runtests.pl --autorun</tt>
  • The README has information on how to create a new mochitest testcase.

How do I run just one test?

Follow the above instructions, but skip the <tt>--autorun</tt> argument, and then click on the test you want to run.

How do the comparison functions work?

ok() tests the truth of the first argument.

ok(true==1, "this passes");
ok(true===1, "this fails");

is() compares the first argument to the second, with a loose comparison method.

is(true, 1, "1 equals true"); 

Here is a typical test failure:

var foo = "abc";
is(foo, "xyz", "foo should hold the last letters of the alphabet");

Mochitest will log the failure to all log listeners (console, file writer, screen, etc).

 > 001: FAIL | Expected "xyz" got "abc" | foo should hold the last letters of the alphabet

What other helper functions are available?

Right now all of Mochikit is available (this will change in {{template.Bug(367393)}}); {{template.Bug(367569)}} added sendChar, sendKey, and sendString helpers. These are available in /tests/SimpleTest/EventUtils.js.

How do I avoid typing boilerplate?

Use the quaint {{wiki.template('Named-source', [ "testing/mochitest/gen_template.pl", "gen_template" ])}} perl script.

Here's enough to be dangerous:

~/firefox/mozilla> cd testing/mochitest/
~/firefox/mozilla/testing/mochitest> perl gen_template.pl -b=123456 > tests/test_bug123456.html

The elements with id 'content' and 'display' in the generated file can be used by your script if you need elements to mess around with.

Where do the tests go?

They go somewhere near the code they're testing, hopefully in the same module, so that ownership of the test cases is clear.

What do the results look like?

The test runner page shows a table with green and red rows for each test page.

Image:testrunner table.png

Where do the results get logged?

There are several possibilities. Right now, calling the test page (http://localhost:8888/tests/index.html) with parameters can make it log to a file and/or to the console, with varying logging levels (INFO/DEBUG/ERROR). See the top of {{wiki.template('Named-source', [ "testing/mochitest/runtests.pl.in", "runtests.pl.in" ])}} for details.

How do I find an error in the log?

Search for the string "ERROR FAIL" to find unexpected failures. You can also search for "SimpleTest FINISHED" to see the final test summary. This is particularly useful when viewing full Tinderbox logs, since the Mochitest output isn't necessarily at the end of the combined log.

What if my tests have failures in them?

You still have to test that. Mochitest provides a todo() function that is identical to ok(), but is expected to fail.

Can tests be run under a chrome URL?

Not yet, but soon. Keep in mind that the xpcshell test harness should be your first choice for XPCOM testing. Only use mochitest if you need events, browser features, networking, etc.


How can I get around the error "Permission denied to get property XPCComponents.classes"?

Adding the following line to your test file (and each event handler) will allow full XPCOM usage.

netscape.security.PrivilegeManager.enablePrivilege('UniversalXPConnect');

This approach is obviously inconvenient. That's why we're working on the build hacking necessary to copy tests into a chrome directory for testing.

{{ wiki.languages( { "ja": "ja/Mochitest" } ) }}

Revision Source

<p>
</p>
<h3 name="Introduction"> Introduction </h3>
<p>Mochitest is an automated testing framework built on top of the <a class="external" href="http://mochikit.com/">MochiKit</a> JavaScript libraries. It's just one of the automated regression testing facilities Mozilla developers have at their disposal. Tests report success or failure to the test harness using JavaScript function calls.
</p><p>Mochitest's unique strength is that it runs tests written as webpages in a full browser environment where the tests have chrome (elevated) privileges. This allows JavaScript in the tests to do much, much more than it would otherwise be able to do. In addition to the capabilities a script would normally have (e.g. DOM manipulation), scripts can access XPCOM components and services, and even access the browser itself. This allows a script to, say, simulate user input to the browser's user interface, before examining the browser to verify that the input had the intended results.
</p><p>Mochitest's use of JavaScript function calls to communicate test success or failure can make it unsuitable for certain types of test. Only things that can in some way be tested using JavaScript (with chrome privileges!) can be tested with this framework. Given some creativity, that's actually much more than you might first think, but it's not possible to write Mochitest tests to directly test a non-scripted C++ component, for example.
</p>
<h3 name="Try_not_to_use_Mochitest"> Try not to use Mochitest </h3>
<p>Yes, really. For many things Mochitest is overkill. In general you should always try to use one of the lighterweight testing frameworks. For example, if you only want to test a single XPCOM component then you should use <a href="en/Writing_xpcshell-based_unit_tests">xpcshell</a>. On the other hand there are some things that Mochitest cannot do, or isn't designed to do. For example, for visual output tests you should try to use the <a href="en/Creating_reftest-based_unit_tests">reftest</a> framework. For more information on the different types of automated testing frameworks see <a href="en/Mozilla_automated_testing">Mozilla automated testing</a>.
</p>
<h3 name="Running_Mochitest"> Running Mochitest </h3>
<p>The Mozilla tinderboxes now run Mochitest as part of the build process, so anyone committing changes to the source will know pretty quickly if they've broken something. Nevertheless, it is a good idea to run Mochitest before committing new code. This is especially true if the code changes are large or would be difficult to back out.
</p><p>To run Mochitest you need to <a href="en/Build_Documentation">build Mozilla</a> with your changes, change directory to $(OBJDIR)/_tests/testing/mochitest, and run the command:
</p>
<code><pre>&gt; perl runtests.pl
</pre></code>
<p>This will open your new build with a document containing a "Run Tests" link at the top. To run the tests simply click this link and watch the results being generated. If any tests have failed this will be reported.
</p>
<h3 name="Writing_new_Mochitest_tests"> Writing new Mochitest tests </h3>
<p>A basic document on <a href="en/Writing_MochiTest-based_unit_tests">Writing MochiTest-based unit tests</a> has already been started. (Maybe the basic stuff from that document should be moved here and that document should give the matter more detailed treatment?)
</p>
<h3 name="Adding_new_Mochitest_tests_to_the_tree"> Adding new Mochitest tests to the tree </h3>
<p>Once you've written a new test you need to add it to the Mozilla source tree and tell the build system about it so that the Mozilla tinderboxes will run it automatically.
</p><p>New Mochitest tests should go somewhere close to the code they are testing. For example, if you create a new test for some HTML feature, you probably want to put the test in the directory mozilla/content/html/content/test or mozilla/content/document/content/test. If a test directory does not exist near the code you are testing you can add a new test directory as the patch in <a class="external" href="https://bugzilla.mozilla.org/show_bug.cgi?id=368531">bug 368531</a> demonstrates.
</p><p>To tell the build system about your new test you need to add the name of your test file to _TEST_FILES in the test directory's Makefile.in.
</p><p>Before committing your new test and the Makefile.in changes, do run Mochitest in an up to date trunk build to check that you will not unexpectedly turn the tree orange.
</p>
<h3 name="Testing_new_tests"> Testing new tests </h3>
<p>Before committing a new test  you should check that it does actually pass as you expect it to. To check your test, first export it to the Mochitest directory by running the command:
</p>
<code><pre>&gt; make
</pre></code>
<p>in the object directory corresponding to the test file's location in the source tree. Now open Mochitest as described above, but this time, instead of clicking on the "Run Tests" link, search for your test and click on it.
</p>
<h3 name="FAQ"> FAQ </h3>
<h4 name="How_do_I_get_started.3F"> How do I get started? </h4>
<p>Start by reading the introductory material provided by the article <a href="en/Mochitest">Mochitest</a>, as well as by {{wiki.template('Named-source', [ "testing/mochitest/README.txt", "reading the README" ])}}.
</p>
<ul><li> <tt>mozilla/testing/mochitest</tt> should be checked out as part of the normal checkout on trunk.
</li><li> mochitest is built by default (unless <tt>--disable-mochitest</tt> is specified in mozconfig) and installed into <tt>($OBJ_DIR)/_tests/testing/mochitest</tt>
</li><li> To run tests
<ul><li> <tt>cd ($OBJ_DIR)/_tests/testing/mochitest</tt> (Note that the working directory matters for runtests.pl!)
</li><li> <tt>./runtests.pl --autorun</tt>
</li></ul>
</li><li> The README has information on how to create a new mochitest testcase.
</li></ul>
<h4 name="How_do_I_run_just_one_test.3F"> How do I run just one test? </h4>
<p>Follow the above instructions, but skip the <tt>--autorun</tt> argument, and then click on the test you want to run.
</p>
<h4 name="How_do_the_comparison_functions_work.3F"> How do the comparison functions work? </h4>
<p><code>ok()</code> tests the truth of the first argument. 
</p>
<pre class="eval">ok(true==1, "this passes");
ok(true===1, "this fails");
</pre>
<p><code>is()</code> compares the first argument to the second, with a loose comparison method.
</p>
<pre class="eval">is(true, 1, "1 equals true"); 
</pre>
<p>Here is a typical test failure:
</p>
<pre class="eval">var foo = "abc";
is(foo, "xyz", "foo should hold the last letters of the alphabet");
</pre>
<p>Mochitest will log the failure to all log listeners (console, file writer, screen, etc).
</p>
<pre class="eval"> &gt; 001: FAIL | Expected "xyz" got "abc" | foo should hold the last letters of the alphabet
</pre>
<h4 name="What_other_helper_functions_are_available.3F"> What other helper functions are available? </h4>
<p>Right now all of Mochikit is available (this will change in {{template.Bug(367393)}}); {{template.Bug(367569)}} added <code>sendChar</code>, <code>sendKey</code>, and <code>sendString</code> helpers. These are available in /tests/SimpleTest/EventUtils.js.
</p>
<h4 name="How_do_I_avoid_typing_boilerplate.3F"> How do I avoid typing boilerplate? </h4>
<p>Use the quaint {{wiki.template('Named-source', [ "testing/mochitest/gen_template.pl", "gen_template" ])}} perl script.
</p><p>Here's enough to be dangerous:
</p>
<pre class="eval">~/firefox/mozilla&gt; cd testing/mochitest/
~/firefox/mozilla/testing/mochitest&gt; perl gen_template.pl -b=123456 &gt; tests/test_bug123456.html
</pre>
<p>The elements with id 'content' and 'display' in the generated file can be used by your script if you need elements to mess around with.
</p>
<h4 name="Where_do_the_tests_go.3F"> Where do the tests go?  </h4>
<p>They go somewhere near the code they're testing, hopefully in the same module, so that ownership of the test cases is clear.
</p>
<h4 name="What_do_the_results_look_like.3F"> What do the results look like? </h4>
<p>The test runner page shows a table with green and red rows for each test page.
</p>
<table border="1"><tbody><tr><td><img alt="Image:testrunner table.png" src="File:en/Media_Gallery/Testrunner_table.png"></td></tr></tbody></table>
<h4 name="Where_do_the_results_get_logged.3F"> Where do the results get logged? </h4>
<p>There are several possibilities. Right now, calling the test page (<span class="plain">http://localhost:8888/tests/index.html</span>) with parameters can make it log to a file and/or to the console, with varying logging levels (INFO/DEBUG/ERROR). See the top of {{wiki.template('Named-source', [ "testing/mochitest/runtests.pl.in", "runtests.pl.in" ])}} for details.
</p>
<h4 name="How_do_I_find_an_error_in_the_log.3F"> How do I find an error in the log? </h4>
<p>Search for the string "ERROR FAIL" to find unexpected failures. You can also search for "SimpleTest FINISHED" to see the final test summary. This is particularly useful when viewing full Tinderbox logs, since the Mochitest output isn't necessarily at the end of the combined log.
</p>
<h4 name="What_if_my_tests_have_failures_in_them.3F"> What if my tests have failures in them? </h4>
<p>You still have to test that. Mochitest provides a <code>todo()</code> function that is identical to <code>ok()</code>, but is expected to fail.
</p>
<h4 name="Can_tests_be_run_under_a_chrome_URL.3F"> Can tests be run under a chrome URL? </h4>
<p>Not yet, but soon. Keep in mind that the <a href="en/Writing_xpcshell-based_unit_tests">xpcshell test harness</a> should be your first choice for XPCOM testing. Only use mochitest if you need events, browser features, networking, etc.
</p><p><br>
</p>
<h4 name="How_can_I_get_around_the_error_.22Permission_denied_to_get_property_XPCComponents.classes.22.3F"> How can I get around the error "Permission denied to get property XPCComponents.classes"? </h4>
<p>Adding the following line to your test file (and each event handler) will allow full XPCOM usage.
</p><p><code> netscape.security.PrivilegeManager.enablePrivilege('UniversalXPConnect'); </code>
</p><p>This approach is obviously inconvenient. That's why we're working on the build hacking necessary to copy tests into a chrome directory for testing.
</p>
<div class="noinclude">
</div>
{{ wiki.languages( { "ja": "ja/Mochitest" } ) }}
Revert to this revision