mozilla
Your Search Results

    Testing Gaia code changes

    When you have finished making a change to the Gaia codebase and it seems to work ok, the next step is to go through a testing procedure to make sure your changes really work — and are working ok with the rest of Gaia — before submitting a patch back to the project. This article explains how.

    The testing procedure generally consists of:

    • Standard debugging procedure
    • Running automated tests

    Let's examine both of these areas now.

    Standard debugging

    If your are an experienced web developer then debugging Gaia code should be a familiar process. We already discussed how to run Gaia in your Desktop Firefox installation, and how to make a basic change. For more complex code base additions, you'll want to make a lot more use of the Firefox debugging tools accessible in Desktop Firefox view.

    Note: More instructions for using all these tools can be found in our Tools zone.

    Automated tests

    You should also run the standard automated test suites that code with Gaia before submitting a patch, to make sure that your code changes do not adversely affect any of the essential existing phone functionality. The tests you can run are:

    • unit tests
    • integration tests
    • performance tests
    • UI tests

    We generally ask that you run the tests before submitting a patch; if it is your first contribution then you can submit without tests, but you need to ask for help with running the tests in future. You should update the Gaia repo before running tests, to make sure you have the latest ones.

    Note: You can find more information on how these tests work at the Firefox OS Automated testing page.

    Note: You should consider running each set of tests on a real device if available (some features/hardware aren't supported on an emulator), or on the B2G Desktop emulator or Firefox Nightly if not.

    Unit tests

    Unit tests are tests on individual units of code in a larger application — in the case of Gaia, individual apps. Gaia uses:

    You can run the following command to download, install and host a unittest server (running it takes a good few minutes, so this might be a good time to make a cup of tea):

    DEBUG=1 make
    export FIREFOX=/Applications/FirefoxNightly.app/Contents/MacOS/firefox
    bin/gaia-test

    This will open a local web page with a list of unit tests. To run them:

    • Select the tests you want to run from the list in the page (an asterisk appears beside them).
    • Click the "Execute" button.
    • Scroll to the bottom of the page to see the results.

    With the window open, you can also run the full test suite from a new terminal window:

    make test-agent-test

    Note: This can take quite a long time, as there are a lot of tests to run (possibly an hour or more), therefore you’ll probably just want to run the tests for the app you’ve modified. You can do this by appending APP=<app folder name> onto the command, for example APP=settings.

    Note: You can also read Gaia Unit Tests for more unit test information.

    Integration tests

    Integration testing involves testing different units of code together in a group to see how well they work together, and is the logical next step after unit testing. Gaia Integration tests are driven by a marionette script written in JavaScript and a python-based server. It can communicate with Gecko so it’s possible to control both the browser and Firefox OS device, and get them interacting with each other.

    You can run the following command to trigger the integration tests:

    make test-integration

    Note: As with the unit tests, running the whole integration test suite can be very time consuming, so you can append APP=<app folder name> onto the above command to test a single app, for example APP=calendar.

    Note: for more on integration tests, read Gaia integration tests.

    Performance tests

    Gaia performance tests will trigger B2G Desktop, launch apps several times, and calculate an average average app load time. After running a test, the performance framework also collects the memory usage of both the app and the system process (b2g).

    To run the tests, you need to have B2G Desktop installed, and run the following command

    make test-perf

    Note: As with the other test types, you are able to append APP=<app folder name> to the above command to test a single app, for example APP=settings.

    The overall average will be returned as a mozPerfDurationsAverage value, like so:

    "mozPerfDurationsAverage": 225.5

    This is the average app load time in milliseconds; you should aim for less than 1 second for optimal user experience. The performance tests also returns some detailed memory usage values:

    {
      "app": {
        "name": "Settings",
        "uss": 16.6,
        "pss": 19.5,
        "rss": 32.9,
        "vsize": 73.3
      },
      "system": {
        "name": "b2g",
        "uss": 50.1,
        "pss": 53.2,
        "rss": 67,
        "vsize": 148.1
      }
    },

    The golden rule for test-perf is "lower numbers are better"; the above numbers mean the following:

    • uss: unique set size
    • pss: proportional set size
    • rss: resident set size
    • vsize: virtual set size

    Generally vsize >= rss >= pss >= uss . vsize and rss don’t accurately reflect a process’s usage of pages shared with other processes. So the two numbers you want to watch are the pss and uss.

    uss is the memory that is completely unique to that process. This is the amount of memory that would be freed if the application was terminated right now. It’s a key value for evaluation.

    pss is the proportional size of the processes' shared libraries. This is memory that would not be released if the process was terminated.

    Note: for more on performance tests, read Gaia performance tests.

    UI tests

    See  Gaia UI Tests Introduction.

    Document Tags and Contributors

    Contributors to this page: rbrandao, SteveLee, P.A., chrisdavidmills
    Last updated by: P.A.,
    Hide Sidebar