Running Tamarin performance tests

The tamarin performance tests can be used to measure performance changes made to tamarin. To run performance tests on the android shell see 'Testing the Android Shell' below. Running the performance tests requires the following steps:

  1. Set the AVM environment variable to the path of the avmshell executable.
  2. Set the BUILTINABC environment variable to the path of the builtin.abc. (in the tamarin-redux/generated directory)
  3. Set the SHELLABC environment variable to the path of the shell_toplevel.abc. (in tamarin-redux/generated directory)
  4. Set the ASC environment variable to the path of the asc.jar compiler. You may download the latest asc.jar from ftp://ftp.mozilla.org/pub/js/tamarin...latest/asc.jar. Also the source to asc.jar may be downloaded and rebuilt from http://opensource.adobe.com/wiki/dis...exsdk/Flex+SDK.
  5. (Optional) You may compare the performance of the avmshell to another avmshell. For example you may download or build the latest checked in avmshell to compare against a local change. To compare a build against another build set the AVM2 environment variable to the other avmshell or use the "-S --avm2" options. Results from the two shells will be shown side to side and the percent speedup will be shown.
$ cd tamarin-redux/test/performance
$ python runtests.py
Executing tests at 2008-07-22 13:56:54.820920
avm: c:/dev/tamarin-tracing/bld/shell/avmshell.exe

test                                                   avm
sunspider/access-binary-trees.as                      82.0
sunspider/access-fannkuch.as                         152.0
sunspider/access-nbody.as                            173.0
sunspider/access-nsieve.as                            65.0
sunspider/bitops-3bit-bits-in-byte.as                 13.0
sunspider/bitops-bits-in-byte.as                      36.0

$ export AVM2=c:/dev/tamarin-tracing2/bld/shell/avmshell.exe
$ python ./runtests.py
Executing tests at 2008-07-22 14:03:51.957381
avm:  c:/dev/tamarin-tracing/bld/shell/avmshell.exe
avm2: c:/dev/tamarin-tracing2/bld/shell/avmshell.exe

test                                                   avm    avm2     %sp

sunspider/access-binary-trees.as                      82.0    80.0     2.5
sunspider/access-fannkuch.as                         153.0   155.0    -1.3
sunspider/access-nbody.as                            176.0   178.0    -1.1
sunspider/access-nsieve.as                            65.0    68.0    -4.4
sunspider/bitops-3bit-bits-in-byte.as                 12.0    13.0    -7.7
sunspider/bitops-bits-in-byte.as                      36.0    36.0     0.0

By default tests sunspider and sunspider-as3 (optimized for as3) and run. Other tests may be specified by selecting ./runtests.py -c [misc,scimark,jsbench]. Also for better accuracy tests may be run multiple times. Specifying ./runtests.py -i 10 will run each tests 10 time and use the fastest result. If more than 2 repetitions are specified the 95% percentile confidence rating is calculated. The confidence rating measures the percentage of the mean 95% of the results were within the range.

Performance Test Options

There are a variety of options available with runtests.py - here is the help text followed by explanations for some options:

$ ./runtests.py -h
usage: runtests.py [options] [tests]
-v --verbose       enable additional output
-E --avm           avmplus command to use
-a --asc           compiler to use
-g --globalabc     DEPRECATED but still works - use builtin.abc (used to be location of global.abc)
-b --builtinabc    location of builtin.abc
-s --shellabc      location of shell_toplevel.abc
-x --exclude       comma separated list of directories to skip
-h --help          display help and exit
-t --notime        do not generate timestamps (cleaner diffs)
-f --forcerebuild  force rebuild all test files
-c --config        sets the config string [default OS-tvm]
-q --quiet         display minimum output during testrun
-l --log           also log all output to given logfile
    --summaryonly   only display final summary
    --rebuildtests  rebuild the tests only - do not run against VM
    --showtimes     shows the time for each test
    --ascargs       args to pass to asc on rebuild of test files
    --vmargs        args to pass to vm
    --timeout       max time to run all tests
    --testtimeout   max time to let a test run, in sec (default -1 = never timeout)
    --html          also create an html output file
    --notimecheck   do not recompile .abc if timestamp is older than .as
    --java          location of java executable (default=java)
    --javaargs      arguments to pass to java
    --random        run tests in random order
    --seed          explicitly specify random seed for --random
-S --avm2          second avmplus command to use
    --avmname       nickname for avm to use as column header
    --avm2name      nickname for avm2 to use as column header
    --detail        display results in 'old-style' format
    --raw           output all raw test values
-i --iterations    number of times to repeat test
-l --log           logs results to a file
-k --socketlog     logs results to a socket server
-r --runtime       name of the runtime VM used, including switch info eg. TTVMi (tamarin-tracing interp)
-m --memory        logs the high water memory mark
    --aotsdk        location of the AOT sdk used to compile tests to standalone executables.
    --aotout        where the resulting binaries should be put (defaults to the location of the as file).
    --aotargs       any extra arguments to pass to compile.py.
    --vmversion     specify vmversion e.g. 502, use this if cannot be calculated from executable
    --vm2version    specify version of avm2
    --vmargs2       args to pass to avm2, if not specified --vmargs will be used
    --nooptimize    do not optimize files when compiling
    --perfm         parse the perfm results from avm
    --csv=          also output to csv file, filename required
    --csvappend     append to csv file instead of overwriting
    --score         compute and print geometric mean of scores
    --index=        index file to use (must end with .py)
    --saveindex=    save results to given index file name

Option Details

  • indexing:
    There are two command line flags that control indexing: --index and —saveindex both require an index filename to be specified after the flag.

    "--index index.py" will load index.py and run all of the tests in that index and report results normalized to the index values.  A subset of the tests in the index can be run by passing what tests you want to run after specifying the index file. 

    "--saveindex indexname.py" will run the specified tests and save the values to the give index file.  Specifying both --index and --saveindex will merge the values specified in --index to the new saved indexfile.
  • cumulative score:
    new option of --score will display a cumulative score for every metric.  Really most useful with an index file (or v8 tests), since running with raw time values will skew the results towards longer running tests
  • csv output:
    "--csv filename" will save run results to a csv file that can be analyzed using excel pivottables.  Use --csvappend if you want to append the results to the csv file

 Support Files

testconfig.txt

This file contains instructions for how the harness should handle performance tests.

metricinfo.py

This file contains a list of metrics that the harness will look for and record.  See the file header for details

 

Testing the Android Shell

The instructions above apply to running tests on an android device as well, but with a few differences. You should set environment variables as described above except for AVM, which should be:

$ export AVM=$TAMARIN_BUILD_TOP/platform/android/android_shell.sh

where $TAMARIN_BUILD_TOP is your main Tamarin repo folder. Note that AVM needs to point to android_shell.sh which is a proxy for running the tests on the phone. The /android-public/android-vars.sh script has commented-out examples of the environment settings for running tests on the android avm shell. To recap the other environment settings:

- Set the BUILTINABC environment variable to the path of the /generated/builtin.abc.

- You may build the .abc files by downloading the latest asc.jar. The latest asc.jar can be downloaded from from ftp://ftp.mozilla.org/pub/js/tamarin...latest/asc.jar. Also the source to asc.jar may be downloaded and rebuilt from http://opensource.adobe.com/wiki/dis...exsdk/Flex+SDK.

- Set the ASC environment variable, as described above.

- Set the SHELLABC environment variable to the path of the /generated/shell_toplevel.abc.

adb

'adb' stands for Android Debug Bridge. adb is used to access your android device and can run both direct and shell-type commands. If you've correctly edited and run the android-vars.sh script mentioned on the Tamarin Build Documentation page you should already be pathed to the adb executable in the public sdk/ndk, but if not it exists at /android-public/android-sdk-mac_86/platform-tools.

adb supports ssh-like commands to an android device. adb automatically connects to a phone if connected by usb. adb shell "ls /" is a good test to see if connected. The command:

$ adb devices

lists the device id of each phone. If this fails, do you have a dev OS installed? Under Settings->Applications->Development, is USB Debugging enabled?

Testsuite Setup

Example commands:

Root the phone

$ adb root

Copy avmshell to /data/app. The shell can be named variously in the build but on the phone the executable must be 'avmshell'

$ adb push avmshell /data/app/avmshell
$ adb shell chmod 777 /data/app/avmshell 

Copy android_runner.sh, if it doesn't already exist on the phone in /data/app

$ adb push tamarin-redux/platform/android/android_runner.sh /data/app/android_runner.s
$ adb shell chmod 777 /data/app/android_runner.sh

Test it out with a simple .abc or no args for usage (should return EXITCODE=0)

$ tamarin-redux/platform/android/android_shell.h hello.abc
hello
EXITCODE=0

Running Performance Tests

To run performance tests on android you don't need to pass --androidthreads or --threads=1 as with the acceptance tests; in fact, a usage error will result. Just type the following commands after making sure you've set the environment variables correctly. 

$ cd /test/performance
$ ./runtests.py

 

Note on using an emulator: the emulator does seem to support hardware floating point but gives a BusError when running avmshell -Darm_vfp. It's unreliable, losing its adb connection frequently. An emulator is ok for testing a few abc files but for any consistency a real device is better.

Document Tags and Contributors

Contributors to this page: Sheppy, jsudduth, Dschaffe, Cpeyer
Last updated by: Sheppy,