Endurance tests

The purpose of our Endurance Tests is to verify Firefox OS and Gaia application stability and detect memory leaks, by performing multiple iterations of common use cases. The Gaia-UI Endurance tests are running nightly, while other types of endurance tests are currently under development (and will be added here when they are brought online). This article explains how these tests work, and how to run them.

The Gaia-UI Endurance tests are a suite of Firefox OS tests built on top of the Gaiatest (Gaia UI Tests) Framework. The tests run on real Firefox OS devices, and use Marionette to drive the device's UI.

As mentioned above, the Gaia-UI Endurance Tests verify Firefox OS device and Gaia application stability by performing several iterations of common use cases. Examples of such use cases include adding a contact to the address book, adding an event to the calendar, sending an SMS message, opening the Firefox browser and browsing to a specific site.

As well as verifying Gaia application functionality under numerous iterations, the endurance tests also perform device-side "checkpoints" after every N number of iterations. This checkpoint currently consists of recording the device’s B2G process RSS memory use value. After the tests are complete, the checkpoint data is posted to the B2G Project on Datazilla. This allows us to detect any sudden changes in B2G memory consumption that may occur during the various test scenarios.

How often are the tests run?

The Gaia-UI Endurance tests are currently run once per day, on a Firefox OS Leo device that is running the B2G-18 V1-Train (1.1) code. The test suite is driven by a Jenkins instance on an Ubuntu 12.04 x64 test machine. Each test in the suite is run with 100 iterations, and memory checkpoints are performed after every 10 iterations. The entire suite takes approximately 16 hours to run.

Where can I see the results?

The B2G memory data that is collected during the endurance test checkpoints is posted on Datazilla after each test suite completes. To view the results, go to the B2G Datazilla Project page, then:

  • On the "Gaia Branch" drop down on the top right, select "v1-train".
  • On the "Device" drop down on the top right, select "d300".
  • In the test list (left side, bottom half) select one of the "endurance_" tests (e.g. "endurance_add_contact".)
  • Ensure that the checkbox beside the listed application (top left of the page) is checked.

Each point on the resulting graph is the average of all of the B2G RSS values collected during the test run. Clicking on one of the single data points on the top graph will result in detailed data for the test run being displayed underneath. The detailed data includes each checkpoint value that was retrieved from the device during the test (one checkpoint after every 10 iterations).

Running the tests

Note: The Gaia-UI Endurance tests and framework source code can be found in the "v1-train" branch of the Mozilla Github Gaia repository.

Let's go through the steps required to set up the Gaia-UI Endurance test environment and run the tests on your local machine and Firefox OS device.

WARNING: Running the Gaia-UI Endurance Tests will result in data being ERASED from the Firefox OS device, including data on the device’s microSD card. Therefore, if there is any data on your Firefox OS device and microSD card that you wish to keep, be sure to back it up first before running the Gaia-UI Endurance Tests. Consider yourself warned!

Prerequisites

  • Ubuntu 12.04 (or better) x64, or Mac OS X (these instructions are confirmed for 10.8 but should work on previous versions of 10, theoretically.)
  • A Firefox OS device ALREADY FLASHED with an ENGINEERING build of Firefox OS B2G-18 V1-Train (1.1.)
  • ADB installed, and the environment variable ADB_PATH pointing to your ADB location.

If you are on Ubuntu, you need to check that it is configured to support the USB connection to the Firefox OS device. To verify this, connect the device to your computer via USB, open a terminal and enter the adb logcat command to see if it connects.

Note: To start with, the Firefox OS device should not be connected to your computer. You will be told when to connect it in the steps below.

Step 1: Clone the Gaia repository

The Gaia-UI Endurance Tests are located in the Mozilla Github Gaia repository. Assuming that you haven’t done so already, the first step is to clone that repo:

git clone https://github.com/mozilla-b2g/gaia.git

You may want to go grab a coffee and come back in five minutes.

Step 2: Run the GaiaTest setup

The Gaia-UI Endurance tests are built upon the GaiaTest framework (which uses Marionette). The next step is to run the setup script to install GaiaTest and all of the required dependencies. You may wish to create a new virtual environment to use with the Gaia-UI Endurance Tests. If you don’t, you may need to use sudo while running the setup command. In your terminal, type:

cd gaia/tests/python/gaia-ui-tests
python setup.py develop

Step 3: Set test vars and acknowledge risks

GaiaTest uses a special file to set certain variables that are required for test configuration. For example, to tell the device which WiFi network it should use. Before running the Gaia-UI Endurance Tests, you must setup the test vars file. Make a copy of the gaia/tests/python/gaia-ui-tests/gaiatest/testvars_template.json file in the same location (call it what you like) and edit it:

  • If the Firefox OS device has a SIM card, add the corresponding phone number in the phone number entry in the carrier section, e.g. "phone_number": "5551234567".
  • Add the same phone number for the "remote_phone_number" entry.
  • In the wifi section, add the SSID for the Wifi network that the Firefox OS device is to connect to during the tests in the ssid line, for example "ssid": "Wifi Network Name".

As mentioned in the warning above, running the Gaia-UI Endurance tests will result in data being erased from the Firefox OS device and microSD card. This is to ensure that the tests start cleanly each time. For example, running a test on a device that already has 10,000 contacts will have very different memory value results vs running the same test on a device with no existing contacts. In order to run the tests, you must acknowledge that you are aware of this data loss risk.

To acknowledge the risks, add the following entry to your testvars file as the first entry in the list: "acknowledged_risks": true.

Note: If the risks are not acknowledged in the testvars file, the tests will not run.

Step 4: Connect the device to WiFi

Next, power on your Firefox OS device. After it boots up, go into the Wifi settings then manually select and connect to the Wifi network that you want the device to use for the tests. Ensure the Wifi settings confirm that the device is connected to Wifi before continuing.

Step 5: Connect to USB and ADB Forward the Device

Attach the Firefox OS device to your computer via USB.

Note: If you’re using an Ubuntu VM, after attaching the device ensure the VM sees the device and connects to it; in the VM select VM > Removable Devices > Your Device > Connect and wait for the device to connect to the VM.

Now tell adb to forward the device port to GaiaTest using the following command:

adb forward tcp:2828 tcp:2828

Note: If you are using the Firefox OS Leo device, you must first tell ADB to be the root user, like so:

adb root
adb forward tcp:2828 tcp:2828

Step 6: Run a Test

Now you’re ready to actually try running a test. To run the add_contact endurance test, with a single iteration, use the following commands:

cd gaia/tests/python/gaia-ui-tests/gaiatest
gaiatest --type=b2g --address=localhost:2828 --testvars=mytestvars.json --iterations=1 --checkpoint=1 --restart tests/endurance/test_endurance_add_contact.py

If you get a “connection refused” error it means the device USB connection didn’t work; just repeat the device connection and try again.

The Firefox OS device b2g process should now restart, then the add_contact endurance test will run with a single iteration. If you watch the Firefox OS device, you’ll see the device UI being manipulated by Marionette. After the test finishes, a memory checkpoint will be performed.

Note: The Gaia-UI Endurance tests now grab the Firefox OS device’s b2g process RSS value for the memory use checkpoint (it used to be the V-SIZE value.)

The test result will be displayed in the terminal window. Note that this result doesn’t include the b2g process memory value; this value is stored in a text file that was created at the time of the checkpoint in the checkpoints directory. To see the resulting b2g process, open this file. This "suite_summary" file will contain the average b2g process memory use (RSS) value, averaged from all of the test checkpoints (in our example there was only one checkpoint anyway).

There are two other files present in the checkpoints folder:

  • The checkpoint_add_contact_(datetime)_summary.log file contains a summary for the test run. This includes the number of iterations, all of the RSS value readings from each checkpoint, the average RSS value, etc.
  • The checkpoint_add_contact_(datetime).log file contains the raw data received from each of the device checkpoints, from which the summary files were produced.

Command Line Options

Here is a description of the GaiaTest command line options that you can use when running the Gaia-UI Endurance Test(s):

  • --type: The type of test being run; always use --type=b2g for this.
  • --address: The address where the device port is forwarded, when you issued the ADB forward command; just use --address=localhost:2828.
  • --testvars: The path and name of the file containing the test vars, as described in Step 3 above.
  • --iterations: The number of times to repeat each test case/test; i.e. 100 iterations of the add_contact test will result in 100 new address book entries being added.
  • --checkpoints: After every N iterations a b2g process memory checkpoint will be grabbed from the device; by default a checkpoint will be performed after all iterations are complete.
  • --restart: The Firefox OS device’s b2g process will be restarted before each new test (not each iteration) begins (i.e. a soft reset.)

As an example, to run the add_contact test and have it perform 100 test case iterations, grabbing a device memory checkpoint after every 10 iterations, and restarting the b2g process before the test begins, you would use the following command line:

gaiatest --type=b2g --address=localhost:2828 --testvars=mytestvars.json --iterations=100 --checkpoint=10 --restart tests/endurance/test_endurance_add_contact.py

If you wish to run the entire Gaia-UI Endurance test suite, instead of specifying a single test on the command line, just point to the manifest file for the test name, like this:

gaiatest --type=b2g --address=localhost:2828 --testvars=mytestvars.json --iterations=1 --checkpoint=1 --restart tests/endurance/manifest.ini

However, note that running the entire test suite, with 100 iterations of each test (and a checkpoint after every 10 iterations) takes approximately 16 hours.

Now you should be all set and have the Gaia-UI Endurance Tests up and running on your local machine/device.

Contributing to the project

If you have any questions about the Firefox OS Endurance tests or are interested in contributing to this important automation development effort, feel free to contact us in the Mozilla #automation IRC channel.

Document Tags and Contributors

Contributors to this page: Sheppy, chrisdavidmills, rwood
Last updated by: Sheppy,