Running Automated JavaScript Tests

  • Revision slug: SpiderMonkey/Running_Automated_JavaScript_Tests
  • Revision title: Running Automated JavaScript Tests
  • Revision id: 75963
  • Created:
  • Creator: Dmandelin
  • Is current revision? No
  • Comment 118 words added

Revision Content

There are two test suites, the JavaScript Test Suite (aka JSTest) and the trace test suite (aka trace-test). JSTest has about 3000 tests, mainly for the interpreter, but with some tests for the tracing JIT. JSTest can be run for either a browser or a JS shell. trace-test has about 500 tests and is exclusively for testing the tracing JIT in the JS shell.

Running JSTest

Running JSTest on a JS shell

The JSTest shell harness is js/src/tests/jstests.py. Basic usage is:

python jstests.py PATH_TO_JS_SHELL

Developers will usually want to run something like this, where NPROC is the number of processors on your system:

python jstests.py -d -j NPROC PATH_TO_JS_SHELL

The -d option skips tests that are marked as randomly failing; random failures are usually just noise when being run for day-to-day developer testing. The -j option runs the suite in parallel (the default is 2, so if you have a 2-core machine you don't need that option); on a fast 8-core machine, the entire suite runs in under a minute.

Another common use case is to run just one test, or all the tests in just one directory:

python jstests.py PATH_TO_JS_SHELL TEST_PATH_SUBSTRING [ TEST_PATH_SUBSTRING_2 ... ]

This runs all the tests whose paths contain TEST_PATH_SUBSTRING. Thus, you can run one test by giving its filename, or any substring that is a substring of that test filename only. You can run all tests in a directory by giving the directory path. You can run 3 specific tests by giving the 3 names.

Other options allow you to show the test command lines being run, command output and return codes, run tests named in a given file, exclude tests named in a given file, hide the progress bar, change the timeout, run skipped tests, print output in Tinderbox format, run a test in the debugger, or run tests in valgrind. Run with the -h option to see all the options.

Running JSTest in a browser

[TBA]

Running trace-test

The trace-test suite uses a Python harness that is similar to the JSTest shell harness, but with some minor differences. Basic usage is the same:

python trace-test.py PATH_TO_JS_SHELL

Developers will usually want to run like this to skip the slow tests (they can run for minutes):

python trace-test.py --no-slow PATH_TO_JS_SHELL

You can select specific tests to run in the same way as the JSTest shell harness. Most of the options in the JSTest harness are also available for this one, but a few have different names, and -j is omitted. Use -h to see all the options.

One helpful option specific to trace-test is -R filename (or --retest filename). The first time this is run, it runs the suite and writes a list of tests that failed to the given filename. The next time it is run, it will run only the tests in the given filename, and then will write the list of tests that failed when it is done. Thus, you can keep retesting as you fix bugs and only the tests that failed the last time will run, speeding things up a bit.

Creating new tests for trace-test

Creating new tests for trace-test is easy. Just add a new JS file to the tests/basic directory (or any appropriate directory under 'tests'). The test harness will automatically find the test and run it. The test is considered to pass if the exit code of the JS shell is zero (i.e., JS didn't crash and there were no JS errors). Use the assertEq function to verify values in your test.

There are some advanced options for tests. See the README (in js/src/trace-test) for details.

Revision Source

<p>There are two test suites, the JavaScript Test Suite (aka JSTest) and the trace test suite (aka trace-test). JSTest has about 3000 tests, mainly for the interpreter, but with some tests for the tracing JIT. JSTest can be run for either a browser or a JS shell. trace-test has about 500 tests and is exclusively for testing the tracing JIT in the JS shell.</p>
<h3>Running JSTest</h3>
<h4>Running JSTest on a JS shell</h4>
<p>The JSTest shell harness is js/src/tests/jstests.py. Basic usage is:</p>
<p style="margin-left: 40px;"><code>python jstests.py PATH_TO_JS_SHELL</code></p>
<p>Developers will usually want to run something like this, where NPROC is the number of processors on your system:</p>
<p style="margin-left: 40px;"><code>python jstests.py -d -j NPROC PATH_TO_JS_SHELL</code></p>
<p>The -d option skips tests that are marked as randomly failing; random failures are usually just noise when being run for day-to-day developer testing. The -j option runs the suite in parallel (the default is 2, so if you have a 2-core machine you don't need that option); on a fast 8-core machine, the entire suite runs in under a minute.</p>
<p>Another common use case is to run just one test, or all the tests in just one directory:</p>
<p style="margin-left: 40px;"><code>python jstests.py PATH_TO_JS_SHELL TEST_PATH_SUBSTRING [ TEST_PATH_SUBSTRING_2 ... ]</code></p>
<p>This runs all the tests whose paths contain TEST_PATH_SUBSTRING. Thus, you can run one test by giving its filename, or any substring that is a substring of that test filename only. You can run all tests in a directory by giving the directory path. You can run 3 specific tests by giving the 3 names.</p>
<p>Other options allow you to show the test command lines being run, command output and return codes, run tests named in a given file, exclude tests named in a given file, hide the progress bar, change the timeout, run skipped tests, print output in Tinderbox format, run a test in the debugger, or run tests in valgrind. Run with the -h option to see all the options.</p>
<h4>Running JSTest in a browser</h4>
<p>[TBA]</p>
<h3>Running trace-test</h3>
<p>The trace-test suite uses a Python harness that is similar to the JSTest shell harness, but with some minor differences. Basic usage is the same:</p>
<p style="margin-left: 40px;"><code>python trace-test.py PATH_TO_JS_SHELL</code></p>
<p>Developers will usually want to run like this to skip the slow tests (they can run for minutes):</p>
<p style="margin-left: 40px;"><code>python trace-test.py --no-slow PATH_TO_JS_SHELL</code></p>
<p>You can select specific tests to run in the same way as the JSTest shell harness. Most of the options in the JSTest harness are also available for this one, but a few have different names, and -j is omitted. Use -h to see all the options.</p>
<p>One helpful option specific to trace-test is -R filename (or --retest filename). The first time this is run, it runs the suite and writes a list of tests that failed to the given filename. The next time it is run, it will run only the tests in the given filename, and then will write the list of tests that failed when it is done. Thus, you can keep retesting as you fix bugs and only the tests that failed the last time will run, speeding things up a bit.</p>
<h3>Creating new tests for trace-test</h3>
<p>Creating new tests for trace-test is easy. Just add a new JS file to the tests/basic directory (or any appropriate directory under 'tests'). The test harness will automatically find the test and run it. The test is considered to pass if the exit code of the JS shell is zero (i.e., JS didn't crash and there were no JS errors). Use the assertEq function to verify values in your test.</p>
<p>There are some advanced options for tests. See the README (in js/src/trace-test) for details.</p>
Revert to this revision