Creating JavaScript tests

  • Revision slug: SpiderMonkey/Creating_JavaScript_tests
  • Revision title: Creating JavaScript tests
  • Revision id: 75958
  • Created:
  • Creator: Jorend
  • Is current revision? No
  • Comment 585 words added, 1930 words removed

Revision Content

There are two large SpiderMonkey test suites: js/src/tests and js/src/jit-test. See Running Automated JavaScript Tests for details.

Most new tests could go in either suite. The main differences are: (1) jstests run in both the shell and the browser, whereas jit-tests run only in the shell; (2) to add a jit-test you just have to add the file, whereas jstests require some boilerplate; (3) jstests automatically load js/src/tests/shell.js before they run, which creates a ton of functions.

To add a new jit-test, make a new file in js/src/jit-test/tests/basic or one of the other subdirectories of jit-test/tests.

To add a new jstest, put the new code in one of these three directories:

  • js/src/tests/ecma_5 - New tests for behavior required by Edition 5 of the ECMAScript standard belong in the appropriate subdirectory here.
  • js/src/tests/js1_8_5/extensions - New tests that cover SpiderMonkey-specific extensions can go here.
  • js/src/tests/js1_8_5/regress - All other new regression tests can go here.

Other js/src/tests subdirectories exist, but most of them contain older tests.

Creating the test case file

Just look at the existing files and follow what they do.

jstests have two special requirements:

  • For each jstest you add, you must add a line to the jstests.list file in the same directory.
  • The call to reportCompare in every jstest is required by the test harness. Except in old tests or super strange new tests, it should be the last line of the test.

All tests can use the assertEq function.

assertEq(v1, v2[, message])

Check that v1 and v2 are the same value. If they're not, throw an exception (which will cause the test to fail).

Handling shell or browser specific features

jstests run both in the browser and in the JavaScript shell.

If your test needs to use browser-specific features, either:

  • make the test silently pass if those features aren't present; or
  • write a mochitest instead (preferred); or
  • in the jstests.list file, mark the test with skip-if(xulRuntime.shell), so that it only runs in the browser.

If your test needs to use shell-specific features, like gc(), either

  • make the test silently pass if those features aren't present; or
  • make it a jit-test (so that it never runs in the browser); or
  • make it a jstest, and in the jstests.list file, mark the test with skip-if(xulRuntime.shell), so that it only runs in the shell.

It is easy to make a test silently pass; anyone who has written JS code for the Web has written this kind of if-statement:

if (typeof gc === 'function') {
    var arr = [];
    arr[10000] = 'item';
    gc();
    assertEq(arr[10000], 'item', 'gc must not wipe out sparse array elements');
} else {
    print('Test skipped: no gc function');
}
reportCompare(0, 0, 'ok');

Choosing the comparison function

reportCompare

reportCompare(expected, actual, description) is used to test if an actual value ( a value computed by the test) is equal to the expected value. If necessary, convert your values to strings before passing them to reportCompare. For example, if you were testing addition of 1 and 2, you might write:

expected = 3;
actual   = 1 + 2;
reportCompare(expected, actual, '3==1+2');

reportMatch

reportMatch(expectedRegExp, actual, description) is used to test if an actual value is matched by an expected regular expression. This comparison is used in circumstances where the actual value may vary within a set pattern and also to allow tests to be used both in the C implementation of the JavaScript engine (SpiderMonkey) and the Java implementation of the JavaScript engine (Rhino) which differ in their error messages or when an error message has changed between branches. For example, a test which recurses to death can report Internal Error: too much recursion on the 1.8 branch while reporting InternalError: script stack space quota is exhausted on the 1.9 branch. To handle this you might write:

actual   = 'No Error';
expected = /InternalError: (script stack space quota is exhausted|too much recursion)/;
try {
  f = function() { f(); }
}
catch(ex) {
  actual = ex + '';
  print('Caught exception ' + ex);
}
reportMatch(expected, actual, 'recursion to death');

compareSource

compareSource(expected, actual, description) is used to test if the decompilation of a JavaScript object (conversion to source code) matches an expected value. Note that tests which use compareSource should be located in the decompilation sub-suite of a suite. For example, to test the decompilation of a simple function you could write:

var f  = (function () { return 1; });
expect = 'function () { return 1; }';
actual = f + '';
compareSource(expect, actual, 'decompile simple function');

Handling abnormal test terminations

Some tests can terminate abnormally even though the test has technically passed. Earlier we discussed the deprecated approach of using the -n naming scheme to identify tests whose PASSED, FAILED status is flipped by the post test processing code in jsDriver.pl and post-process-logs.pl. A different approach is to use the expectExitCode(exitcode) function which outputs a string

--- NOTE: IN THIS TESTCASE, WE EXPECT EXIT CODE <exitcode> ---

that tells the post-processing scripts jsDriver.pl or post-process-logs.pl that the test passes if the shell or browser terminates with that exit code. Multiple calls to expectExitCode will tell the post-processing scripts that the test actually passed if any of the exit codes are found when the test terminates.

This approach has limited use however. In the JavaScript shell, an uncaught exception or out of memory error will terminate the shell with an exit code of 3. However an uncaught error or exception will not cause the browser to terminate with a non-zero exit code. To make the situation even more complex, newer C++ compilers will abort the browser with a typical exit code of 5 by throwing a C++ exception when an out of memory error occurs. Simply testing the exit code does not allow you to distinguish the variety of causes a particular abnormal exit may have.

In addition, some tests pass if they do not crash however they may not terminate unless killed by the test driver.

A modification will soon be made to the JavaScript tests to allow an arbitrary string to be output which will be used to post process the test logs to better determine if a test has passed regardless of its exit code.

Performance testing

It is not possible to test all performance related issues using the JavaScript tests. In particular, it is not possible to test absolute timing values for a test. This is due to the varied hardware and platforms upon which the JavaScript tests will be executed. It may be the case that a particular bug improves an operation from 200ms to 50ms on your machine, however it will not be true in general. Tests which measure absolute times of tests belong in other test frameworks such as Talos or Dromaeo.

It is possible to test ratios of times to some extent although it can be tricky to write a test which will not be affected by the host machines performance.

It is possible to test the polynomial time dependency of a test using the BigO function.

Testing polynomial time behavior

To test the polynomial time dependency, follow these steps:

  1. Create a test file as described above.
  2. Add a global variable var data = {X: [], Y: []} which you will use to record the sizes and times for executing a test function.
  3. Create a test function which takes a size argument and which times performing the operations of that size. Each size and time interval should be stored in the data object. Note that in order to reduce the possibility that a garbage collection will affect the timing of your test, you should call the gc() function after completing the timing of each size.
  4. Create a loop which will call your test function for a range of sizes.
  5. Calculate the order of the timing data you have collected by calling BigO(data).
  6. Perform a reportCompare comparison of the calculated order against the expected order.

For example, to test if the "Big O" time dependency of adding a character to a string is less than quadratic you might do something like:

var data = {X: [], Y:[]};

for (var size = 1000; size < 10000; size += 1000)
{
  appendchar(size);
}

var order = BigO(data);

reportCompare(true', order < 2, 'append character BigO < 2');

function appendchar(size)
{
  var i;
  var s = '';

  var start = new Date();
  for (i = 0; i < size; i++)
  {
    s += 'c';
  }
  var stop  = new Date();
  gc();
  
  data.X.push(size);
  data.Y.push(stop - start);
}

Note: The range of sizes and the increment between tests of different sizes can have an important effect on the validity of the test. You should strive to keep the minimum size above a certain value so that the minimum times are not too close to zero.

Testing your test

Run your new test locally before checking it in (or posting it for review). Nobody likes patches that include failing tests!

It's also good sanity check to run each new test against an unpatched shell or browser. The test should fail, if it's working properly.

Checking in completed tests

Handling non-security sensitive tests

Tests are usually reviewed and pushed just like any other code change. Just include the test in your patch. Don't forget to hg add the new test files.

It is OK under certain circumstances to push new tests to certain repositories without a code review. Don't do this unless you know what you're doing. Ask a SpiderMonkey peer for details.

Handling security sensitive tests

Security senstive tests should be not be checked into CVS or mercurial until they have been made public. Instead, ask a SpiderMonkey peer how to proceed.

Revision Source

<p>There are two large SpiderMonkey test suites: js/src/tests and js/src/jit-test. See <a href="/en/SpiderMonkey/Running_Automated_JavaScript_Tests" title="en/SpiderMonkey/Running Automated JavaScript Tests">Running Automated JavaScript Tests</a> for details.</p>
<p>Most new tests could go in either suite. The main differences are: (1) jstests run in both the shell and the browser, whereas jit-tests run only in the shell; (2) to add a jit-test you just have to add the file, whereas jstests require some boilerplate; (3) jstests automatically load js/src/tests/shell.js before they run, which creates a ton of functions.</p>
<p><strong>To add a new jit-test,</strong> make a new file in js/src/jit-test/tests/basic or one of the other subdirectories of jit-test/tests.</p>
<p><strong>To add a new jstest,</strong> put the new code in one of these three directories:</p>
<ul> <li>js/src/tests/ecma_5 - New tests for behavior required by Edition 5 of the ECMAScript standard belong in the appropriate subdirectory here.</li> <li>js/src/tests/js1_8_5/extensions - New tests that cover SpiderMonkey-specific extensions can go here.</li> <li>js/src/tests/js1_8_5/regress - All other new regression tests can go here.</li>
</ul>
<p>Other js/src/tests subdirectories exist, but most of them contain older tests.</p>
<h2 name="Creating_the_test_case_file">Creating the test case file</h2>
<p>Just look at the existing files and follow what they do.</p>
<p>jstests have two special requirements:</p>
<ul> <li>For each jstest you add, you must add a line to the <strong>jstests.list</strong> file in the same directory.</li> <li>The call to reportCompare in every jstest is required by the test harness. Except in old tests or <em>super</em> strange new tests, it should be the last line of the test.</li>
</ul>
<p>All tests can use the assertEq function.</p>
<p><strong>assertEq(v1, v2[, message])<br>
</strong></p>
<p style="margin-left: 40px;">Check that v1 and v2 are the same value. If they're not, throw an exception (which will cause the test to fail).</p>
<h3 name="Handling_shell_or_browser_specific_features">Handling shell or browser specific features</h3>
<p>jstests run both in the browser and in the JavaScript shell.</p>
<p>If your test needs to use browser-specific features, either:</p>
<ul> <li>make the test silently pass if those features aren't present; or</li> <li>write a mochitest instead (preferred); or</li> <li>in the jstests.list file, mark the test with <code>skip-if(xulRuntime.shell)</code>, so that it only runs in the browser.</li>
</ul>
<p>If your test needs to use shell-specific features, like gc(), either</p>
<ul> <li>make the test silently pass if those features aren't present; or</li> <li>make it a jit-test (so that it never runs in the browser); or</li> <li>make it a jstest, and in the jstests.list file, mark the test with <code>skip-if(xulRuntime.shell)</code>, so that it only runs in the shell.</li>
</ul>
<p>It is easy to make a test silently pass; anyone who has written JS code for the Web has written this kind of if-statement:</p>
<pre>if (typeof gc === 'function') {
    var arr = [];
    arr[10000] = 'item';
    gc();
    assertEq(arr[10000], 'item', 'gc must not wipe out sparse array elements');
} else {
    print('Test skipped: no gc function');
}
reportCompare(0, 0, 'ok');
</pre>
<h3 name="Choosing_the_comparison_function">Choosing the comparison function</h3>
<h4 name="reportCompare">reportCompare</h4>
<p><code>reportCompare(expected, actual, description)</code> is used to test if an actual value ( a value computed by the test) is equal to the expected value. If necessary, convert your values to strings before passing them to reportCompare. For example, if you were testing addition of 1 and 2, you might write:</p>
<pre>expected = 3;
actual   = 1 + 2;
reportCompare(expected, actual, '3==1+2');
</pre>
<h4 name="reportMatch">reportMatch</h4>
<p><code>reportMatch(expectedRegExp, actual, description)</code> is used to test if an actual value is matched by an expected regular expression. This comparison is used in circumstances where the actual value may vary within a set pattern and also to allow tests to be used both in the C implementation of the JavaScript engine (SpiderMonkey) and the Java implementation of the JavaScript engine (Rhino) which differ in their error messages or when an error message has changed between branches. For example, a test which <em>recurses to death</em> can report <code>Internal Error: too much recursion</code> on the 1.8 branch while reporting <code>InternalError: script stack space quota is exhausted</code> on the 1.9 branch. To handle this you might write:</p>
<pre>actual   = 'No Error';
expected = /InternalError: (script stack space quota is exhausted|too much recursion)/;
try {
  f = function() { f(); }
}
catch(ex) {
  actual = ex + '';
  print('Caught exception ' + ex);
}
reportMatch(expected, actual, 'recursion to death');
</pre>
<h4 name="compareSource">compareSource</h4>
<p><code>compareSource(expected, actual, description)</code> is used to test if the decompilation of a JavaScript object (conversion to source code) matches an expected value. Note that tests which use <code>compareSource</code> should be located in the <code>decompilation</code> sub-suite of a suite. For example, to test the decompilation of a simple function you could write:</p>
<pre>var f  = (function () { return 1; });
expect = 'function () { return 1; }';
actual = f + '';
compareSource(expect, actual, 'decompile simple function');
</pre>
<h4 name="Handling_abnormal_test_terminations">Handling abnormal test terminations</h4>
<p>Some tests can terminate abnormally even though the test has technically passed. Earlier we discussed the deprecated approach of using the <code>-n</code> naming scheme to identify tests whose PASSED, FAILED status is flipped by the post test processing code in <code>jsDriver.pl</code> and <code>post-process-logs.pl</code>. A different approach is to use the <code>expectExitCode(exitcode)</code> function which outputs a string</p>
<pre>--- NOTE: IN THIS TESTCASE, WE EXPECT EXIT CODE &lt;exitcode&gt; ---</pre>
<p>that tells the post-processing scripts <code>jsDriver.pl</code> or <code>post-process-logs.pl</code> that the test passes if the shell or browser terminates with that exit code. Multiple calls to <code>expectExitCode</code> will tell the post-processing scripts that the test actually passed if any of the exit codes are found when the test terminates.</p>
<p>This approach has limited use however. In the JavaScript shell, an uncaught exception or out of memory error will terminate the shell with an exit code of 3. However an uncaught error or exception will not cause the browser to terminate with a non-zero exit code. To make the situation even more complex, newer C++ compilers will abort the browser with a typical exit code of <code>5</code> by throwing a C++ exception when an out of memory error occurs. Simply testing the exit code does not allow you to distinguish the variety of causes a particular abnormal exit may have.</p>
<p>In addition, some tests pass if they do not crash however they may not terminate unless killed by the test driver.</p>
<p>A modification will soon be made to the JavaScript tests to allow an arbitrary string to be output which will be used to post process the test logs to better determine if a test has passed regardless of its exit code.</p>
<h3 name="Performance_testing">Performance testing</h3>
<p>It is not possible to test all performance related issues using the JavaScript tests. In particular, it is not possible to test absolute timing values for a test. This is due to the varied hardware and platforms upon which the JavaScript tests will be executed. It may be the case that a particular bug improves an operation from 200ms to 50ms on <em>your</em> machine, however it will not be true in general. Tests which measure absolute times of tests belong in other test frameworks such as Talos or <a class="link-https" href="https://wiki.mozilla.org/Dromaeo" title="https://wiki.mozilla.org/Dromaeo">Dromaeo</a>.</p>
<p>It is possible to test ratios of times to some extent although it can be tricky to write a test which will not be affected by the host machines performance.</p>
<p>It is possible to test the polynomial time dependency of a test using the <code>BigO</code> function.</p>
<h4 name="Testing_polynomial_time_behavior">Testing polynomial time behavior</h4>
<p>To test the polynomial time dependency, follow these steps:</p>
<ol> <li>Create a test file as described above.</li> <li>Add a global variable <code>var data = {X: [], Y: []}</code> which you will use to record the <em>sizes</em> and times for executing a test function.</li> <li>Create a test function which takes a <em>size</em> argument and which times performing the operations of that size. Each <em>size</em> and time interval should be stored in the <code>data</code> object. Note that in order to reduce the possibility that a garbage collection will affect the timing of your test, you should call the <code>gc()</code> function after completing the timing of each size.</li> <li>Create a loop which will call your test function for a range of sizes.</li> <li>Calculate the <em>order</em> of the timing data you have collected by calling <code>BigO(data)</code>.</li> <li>Perform a <code>reportCompare</code> comparison of the calculated order against the expected order.</li>
</ol>
<p>For example, to test if the "Big O" time dependency of adding a character to a string is less than quadratic you might do something like:</p>
<pre>var data = {X: [], Y:[]};

for (var size = 1000; size &lt; 10000; size += 1000)
{
  appendchar(size);
}

var order = BigO(data);

reportCompare(true', order &lt; 2, 'append character BigO &lt; 2');

function appendchar(size)
{
  var i;
  var s = '';

  var start = new Date();
  for (i = 0; i &lt; size; i++)
  {
    s += 'c';
  }
  var stop  = new Date();
  gc();
  
  data.X.push(size);
  data.Y.push(stop - start);
}

</pre>
<p><strong>Note</strong>: The range of sizes and the increment between tests of different sizes can have an important effect on the validity of the test. You should strive to keep the minimum size above a certain value so that the minimum times are not too close to zero.</p>
<h2 name="Testing_your_test">Testing your test</h2>
<p>Run your new test locally before checking it in (or posting it for review). Nobody likes patches that include failing tests!</p>
<p>It's also good sanity check to run each new test against an unpatched shell or browser. The test should fail, if it's working properly.</p>
<h2 name="Checking_in_completed_tests">Checking in completed tests</h2>
<h3 name="Handling_non-security_sensitive_tests">Handling non-security sensitive tests</h3>
<p>Tests are usually reviewed and pushed just like any other code change. Just include the test in your patch. Don't forget to <code>hg add</code> the new test files.</p>
<p>It is OK under certain circumstances to push new tests to certain repositories without a code review. Don't do this unless you know what you're doing. Ask a SpiderMonkey peer for details.</p>
<h3 name="Handling_security_sensitive_tests">Handling security sensitive tests</h3>
<p>Security senstive tests should be <strong>not</strong> be checked into CVS or mercurial until they have been made public. Instead, ask a SpiderMonkey peer how to proceed.</p>
Revert to this revision