The majority of the acceptance tests are written in actionscript and located in test/acceptance.
See Running Tamarin Acceptance Tests for instructions on running the acceptance test suite.
The most common tests are written in actionscript and end with .as.
The acceptance test template can be found here.
Testing versioned bug fixes
When testing versioned bug fixes (using Avmplus.System.swfVersion) there are a couple of requirements:
- You must use System.swfVersion when looking up the swfVersion.
- The test file can not be wrapped in a package or define classes as the file will be wrapped in a function when used in the ATS. If you must use classes, define them in a sub-dir with the same name as the test.
- You must create a testname.as.avm_args file with the USES_SWFVERSION directive.
There are a variety of support files available for non-standard actionscript tests.
This file specifies additional arguments to pass to asc when compiling the test:
# ASC args for file # two modes are available: # override| all command line arguments (except builtin.py) are ignored and replaced by these # merge| merge these args in with the current args # specifiy an arg that starts with -no will disable the arg... eg: -no-optimize merge| -AS3
A dir.asc_args file can also be created with the same contents that will apply to the entire directory.
testname.as.asc_args files will take precedence over the dir.asc_args file.
This file specifies additional arguments to pass to the shell when running the test - the user can use the special variable $DIR to refer to the current directory. This example passes in another .abc file as an argument to the file being run:
Another use would be to pass a specific argument to the shell:
This file can have multiple lines with different arguments. When there are multiple lines, the harness will re-run the test for every line passing along the different arguments.
There is one special keyword that can be used, USES_SWFVERSION. When that keyword is encountered, the test harness will run the test against all possible -swfversion versions:
is the equivalent of:
-swfversion 9 -swfversion 10 -swfversion 11 -swfversion 12 ...
Any additional args on the USES_SWFVERSION line will be passed to each run of the vm. (See Bug 587093 for details.)
Specifying multiple .abcs:
If you desire to run a test with multiple .abc files (e.g: avmshell a.abc b.abc testname.abc) there is a special avm_args directive called multiabc that is specified surrounded by two pipe (|) symbols:
|multiabc| $DIR/testname_support/a.abc $DIR/testname_support/b.abc
The extra abc files will usually be placed in a _support dir (see Additional Files below). Any avm arguments can be placed before the |multiabc| directive, or the line can start with the directive. Any test arguments (--) must be placed after the multiabc file listings. Here is a more complicated example that passes both avm args, multiple abcs and test args:
-Ojit |multiabc| $DIR/testname_support/a.abc -- test_args
A dir.avm_args file can also be created with the same contents that will apply to the entire directory.
testname.as.avm_args files will take precedence over the dir.avm_args file
If a test is only valid for certain timezones, the timezones can be specified in this support file.
Below is a sample file that covers the 4 main US timezones:
# Place all timezones that this file applies to. # These values correspond to the value returned by time.tzname tuple. # (See http://docs.python.org/library/time.html for details) # e.g.: Eastern Standard Time/Eastern Daylight Time = ('EST','EDT') # Write timezones as tuples, one to each line. ('EST', 'EDT') ('CST', 'CDT') ('MST', 'MDT') ('PST', 'PDT')
This file is used when the test throws an uncatchable error. The first line must contain the expected error (without the extra debugger information). The exitcode is optional, but must be defined in this file if non-zero.
VerifyError: Error #1021 exitcode: 1
File that specifies the expected exitcode. Should just be an integer with nothing else. (Note that if a .err file is defined, the .exitcode file will be ignored). In this example the .as test runs out of memory and so the expected exitcode is:
File that specifies expected output. This should only be used when it is not possible to generate the test using the standard test framework. When this file is defined, the output from the testcase is matched line by line to the .out file and any difference is considered a failure. May be used in conjunction with a .exitcode file.
For special cases where the .as source is not available (e.g. fuzzed files) it is possible to check in a .abc_ which will let the test harness run compiled files without source. The underscore is necessary otherwise the buildbot system will delete the .abc before starting up a testrun (not an issue when running locally, but makes it easy to differentiate between binary-only abc files and generated abcs.
In some cases a test may require additional files to successfully run. There are two kinds of support directories available.
Files to be included when compiling the test:
If a test requires additional .as files to be included when compiling, a directory with the same name as the test (minus any extension) can be created. The name is case-sensitive.
Given a test named
inheritClass.as, the user can create a directory named
inheritClass and place files
b.as in that directory.
When the harness compiles the
inheritClass.as test, the files in
inheritClass will be included:
java -jar asc.jar -import builtin.abc -in shell.as -in inheritClass/a.as -in inheritClass/b.as inheritClass.as
Additional files required by a test:
If a test requires additional files to run, but are not to be included when compiling, a testname_support directory can be created. The directory name should not include any testname extensions. Any .as files in this directory will be compiled, but will not be run by the harness.
Note that if you have .as files in the _support dir, they will be compiled as if they are regular tests. If you do not wish to have shell.as included when compiling, you must create a dir.asc_args file with an override parameter:
# the following line will override all compile arguments and just compile a .as file with -import builtin.abc override|