mozilla

Revision 542971 of Gaia Integration tests

  • Revision slug: Mozilla/Firefox_OS/Platform/Automated_testing/Gaia_integration_tests
  • Revision title: Gaia Integration tests
  • Revision id: 542971
  • Created:
  • Creator: gaye
  • Is current revision? No
  • Comment

Revision Content

This document covers running and writing integration tests for Gaia apps, and provides a detailed explanation of how the integration tests are run.

Running tests

This section looks at setting up your envinronment correctly to run the existing integration test suite. Note that tests live with the rest of the apps code (in apps/my_app/test/marionette for example) and test files always end in _test.js. Shared code for tests lives under shared/test/integration.

Setup

To setup your environment for running tests:

  1. Make sure you have marionette-js-runner installed by using the below command:
    npm install --save-dev marionette-client marionette-js-runner
    
    Gaia uses this to run the integration tests with a custom profile builder for Gaia.
  2. All integration tests run under a node environment. You need node 0.10 or higher for this to work predictably (detailed instructions on installing node on different platforms).

Running all the tests

The following command runs all the integration tests for all the apps:

make test-integration

You will get to watch the tests pop up on your screen and you will probably also get to hear various sounds as they run.  This can be distracting or possibly maddening. To run specific tests you can set the TEST_FILES environment variable. To do so in the bash shell (default on Mac OS X), use a command of the form:

TEST_FILES=<test> make test-integration

For example, you could run the day_view_test.js test in calendar app with the below command.

TEST_FILES=./apps/calendar/test/marionette/day_view_test.js make test-integration

If you would like to run more than one test, you could do the following:

TEST_FILES="./apps/calendar/test/marionette/day_view_test.js ./apps/calendar/test/marionette/today_test.js" make test-integration

Invoking tests for a specific app

To run just the tests for a specific app, you can use the following form:

make test-integration APP=<APP>

For example, you could run all tests for the calendar app with

make test-integration APP=calendar

Running the tests in the background, quietly

You might want to run the tests in the background, meaning that you won't have to see the tests run, worry about them stealing your focus, or risk impacting the running tests. One solution is to use Xvfb:

xvfb-run make test-integration

If you are using PulseAudio and want to keep the tests quiet, then you can specify PULSE_SERVER=":" to force an invalid server so no sound is output:

PULSE_SERVER=":" make test-integration

You can of course combine both:

PULSE_SERVER=":" xvfb-run make test-integration

Running tests without building a profile

if you would like to run tests without building a profile, use make test-integration-test:

PROFILE_FOLDER=profile-test make # generate profile directory in first time
make test-integration-test

Debugging Tests

To view gecko log output from a test, use the following:

VERBOSE=1 make test-integration

Use cases and design patterns

This section explores some use cases and design patterns that can help make the js marionette tests we write more maintainable.

What are common things for ui tests to do?

Marionette tests simulate users interacting with apps.

In the abstract, your marionette tests will test your app by interacting with it as a user does. What do Firefox OS app users do? Here are some examples:

  • launch an app,
  • click inside of an app,
  • read text in an area of an app,
  • wait for the ui to change,
  • type in input fields,
  • fill out forms,
  • switch between apps,
  • close apps, and
  • swipe inside of an app

Amazingly, you can simulate all of these user actions using marionette! You can find detailed, code-level documentation for the basic tasks here. For more advanced interactions, marionette js has a rich collection of plugins including ones for app management, form completion, injecting scripts, reading and writing to device storage, editing settings, and anything else your heart desires!

Marionette is faster than my application!

An extremely common issue is that marionette performs UI actions much faster than the average user. Imagine, for instance, that we had an application with two simple views each with a button to navigate to the other. We might expect for the following snippet of pseudocode to work:

This is an example of what not to do. Most marionette tests that fail intermittently exhibit the following behavior.

view1.navigateToOther();
view2.navigateToOther();
assert.ok(view1.isDisplayed());

This simple test tries to navigate from one view to the other, navigate back, and then make sure that the original view is displayed. This test will fail intermittently since there's a race condition between your application code which renders the two views and your test code which makes the implicit (and incorrect) assumption that the ui is responding synchronously. Instead, you need to program your test to behave like a user. Users don't try to click things or read things before they show up in the UI. Users "poll" the UI by looking at it until the thing they're waiting for shows up. Here's a better version of our test:

client.waitFor(view1.isDisplayed.bind(view1));
view1.navigateToOther();
client.waitFor(view2.isDisplayed.bind(view2));
view2.navigateToOther();
client.waitFor(view1.isDisplayed.bind(view1));

Marionette.Client#waitFor is a utility that blocks your test execution until the parameter function returns true.

How should I structure my ui test libraries?

Most of the test code you will write won't be general purpose enough to warrant abstracting into a general-purpose plugin. For app-specific code, we recommend having a separate file/class/module for each view. We demonstrate this pattern in gaia's calendar test libraries. The views folder has different a class for each calendar app view. The general calendar object has methods to navigate between the different views. In each of the views, we "scope" methods to search for elements in the DOM to the view's root element.

Oftentimes the views will contain getter methods to read an element or data from the view and setter methods to write data to an <input> or <select> tag. A good example is creating a calendar event which involves calling setter methods on the edit event view and saving. It's also idiomatic to abstract multiple ui actions which form a single "logical" action into view methods. For instance, setting a collection of reminders for a calendar event involves tapping a select option for each reminder.

How the test runner works

This section provides a detailed review of how the test runner works, for those that are interested.

Note: All of the various ways the Marionette JavaScript integration tests are run happens via the same make test-integration path.  Only parameters are changed.

What triggers the tests?

On travis:

On TBPL:

  • The build automation runs the following:
    make test-integration NPM_REGISTRY=http://npm-mirror.pub.build.mozilla.org
    REPORTER=mocha-tbpl-reporter TEST_MANIFEST=./shared/test/integration/tbpl-manifest.json
  • The custom NPM_REGISTRY may or may not continue to exist; our node_modules are now sourced from our gaia-node-modules repo.
  • The custom reporter mocha-tbpl-reporter just prints out TEST-START / TEST-PASS / TEST-UNEXPECTED-FAIL / TEST-END that the parsers consuming TBPL output expect, instead of all of those cool checkmarks you get when using the (default) spec reporter we use locally and on Travis.  Note that the time-stamps from the TBPL output are much more realistic than what the spec reporter claims for test durations.  Do not believe the spec reporter: it doesn't include your setup times!
  • The tbpl-manifest.json manifest lists tests specifically blacklisted on TBPL/

Locally:

  • Whatever you want, you got it.

Okay, what does make test-integration actually do?

  • The Makefile invokes bin/gaia-marionette, passing MARIONETTE_RUNNER_HOST, TEST_MANIFEST, and REPORTER as command-line arguments pulled out of the make environment.
  • bin/gaia-marionette is a wrapper around marionette-js-runner's bin/marionette-mocha and generally appears to be a place to add a few more defaults, duplicate a bunch of work already done in the Makefile, and generately be a place to cram stuff if you don't understand Makefiles / want to make things take longer to run by avoiding parallelization.  The good news is that because of all this you can invoke it directly.  The notable bits in here are:
    • It uses "find" to locate all potential tests.  If APP is in the environment, it only looks for tests under a given app.  It will find blacklisted test files, but these are filtered out by marionette-mocha.
    • gaia/shared/test/integration/setup.js is specified as the first script to run when we spin up the mocha runtime.  This currently is just a place to require and add plugins for use across all app tests.
    • gaia/shared/test/integration/profile.js is specified as the base configuration for all marionette integration tests' Gecko profiles.  Adding things to this file will cause all tests to have certain prefs set, settings set, and/or apps installed.  You almost certainly should not add things to this file.
    • gaia/shared/test/integration/profile_builder.js is specified as the profile builder.  It uses the Gaia Makefile, mozilla-profile-builder in conjunction with profile.js above and specific per-test-file settings you specify in your marionette.client({ ... }) call to configure your profile.  We discuss the actual steps, when they actually happen at runtime, below.
  • marionette-js-runner's bin/marionette-mocha applies its own defaults that don't matter to us, applies the manifest rules in lib/applymanifest.js to filter out blacklisted tests and spins up a ParentRunner instance.
  • The ParentRunner instantiates a ChildRunner.  The ParentRunner is boring and its name is somewhat misleading.  Both the ParentRunner and ChildRunner live entirely in the same "runner" process.  There is currently only ever one ChildRunner.

Now things get somewhat complicated, so let's take a second to get an overview of all the processes that can be active and how they work:

  • The runner process. This is the bin/marionette-mocha process with the ParentRunner and ChildRunner.
    • It is a node.js environment.
    • None of your test code runs in this process!
    • It forks the mocha / bin/_mocha node.js script to be the "mocha test" (sub-)process.
      • This is done using the node.js child_process.fork mechanism.  This allows the "runner" process to send messages to the child and receive message events in return.
    • It gets bossed around by the "mocha test" process.  Literally.  The messages get received and converted into lookups on the ChildRunner object with apply() then called and a callback generated that will marshal the result back down to the client.  This means that the runner process can and is told to:
      • Start/stop/restart "hosts".  Hosts are your B2G Desktop/Firefox/B2G devices.
  • The mocha test process.  This is the mocha / bin/_mocha node.js script.
    • It is a node.js environment and this is the stock mocha test runner.
    • This is where your test code runs!
    • The mocha runner gets told about the following files, in the following order:
    • See below for details on the execution model and how a test run actually works.
  • The "host" process(es).  These are B2G Desktop/Firefox/(magic adb control wrapper) instances.
    • These are Gecko processes, potentially a hierarchy of processes now that OOP is in play.
    • They are communicated with via the Marionette protocol.
  • "server" processes: These are test/fake servers spun up by various tests.  These currently all seem to be spun up by the "mocha test" processes directly, but it's conceivable the "runner" process could also spin some up.  Some known examples:
    • A test web-server.  gaia/shared/test/integration/server.js forks off a subprocess that runs gaia/shared/test/integration/server_child.js and communicates via the same child_process message-passing mechanism that the "runner" and "mocha test" processes use.
      • there are duplicates of this implementation in browser and homescreen for some reason, likely historical.
    • The e-mail app has fake IMAP/POP3/SMTP servers that it shares with its back-end implementation.  These live in the mail-fakeservers repo.  The fake-servers actually run in an XPCShell Gecko-style instance that initially communicates via a low-level IPC interface using json-wire-protocol to bootstrap equivalence with the back-end tests' mechanism and then afterwards with a synchronous HTTP REST interface for all the e-mail domain work.  It's due for some cleanup.

What is in my global scope in my test file?

  • Marionette: this comes from lib/runtime.js.  It is a function that's a wrapper around mocha's TDD suite() method.  Source is in lib/runtime/marionette.js.  It has the following notable properties annotated onto it:
    • client(stuffToAddToTheProfile, driver that defaults to the synchronous Marionette API): Bound to HostManager.createHost from lib/runtime/hostmanager.js.
    • plugin(name, module): This is a bound function that invokes HostManager.addPlugin from lib/runtime/hostmanager.js.  It adds the plugin for the duration of your test suite.
  • The TDD interface pokes setup/teardown/suiteSetup/suiteTeardown/suite/test into the global namespace.

What are marionette plugins?

  • Marionette plugins are node.js modules which extend the native abilities of the js marionette client.
  • They conform to the following simple plugin API:
MarionettePlugin = {
  /**
   * @param {Marionette.Client} client the marionette client to extend.
   * @param {Object} options plugin-specific configuration.
   * @return {MarionettePlugin} appropriately configured plugin.
   */
  setup: function(client, options) {}
};
  • They get exposed through the client (i.e. client.apps).
  • Existing plugins include

What is mocha's TDD execution model?

  • The mocha runner require()s all of the files it is told about in sequence in Mocha.prototype.loadFiles().  This consists of:
    • synchronously emitting a pre-require event.
      • The TDD interface setup/teardown/suiteSetup/suiteTeardown/suite/test into the global namespace.
    • synchronously requiring the file in question.
      • This means the top level of your file is executed.
      • Any marionette() calls to define a suite (since Marionette is just a wrapper around suite()) will in turn have their defining function executed synchronously.
      • The test() calls inside the marionette() calls are NOT executed.  Those don't happen until the test is actually run.
      • This does mean you have to be very careful not to do anything foolish in the top-level of your file or the marionette() file.
    • synchronously emitting a 'require' event passing the module from the file in question.
    • synchronously emitting a 'post-require' event.
  • The Runner runs the suites and tests in sequence.
    • emits start.
    • runSuite loops over all the suites, for each one:
      • emits suite.
      • emits beforeAll, which is what suiteSetup maps to.
      • calls runTests loops over all the tests, for each one:
        • emits test.
        • emits beforeEach, which is what setup maps to.
        • calls runTest, which runs your test function.
        • emits test end.
          • This runs checkGlobals() to make sure you didn't clobber anything unexpected into the global state.  It fails your test if you did.
        • emits afterEach, which is what teardown maps to.
      • emits afterAll, which is what suiteTeardown maps to.
      • emits suite end.
    • emits end.

What is the life-cycle of the Gecko processes?

For your test suite (aka each top-level marionette('description', function() {...}) in your file), a new profile is created.

For each test (aka each test('description', function() {...}) in your marionette('description', ...)), the host gets restarted after each test().

The nitty gritty of this is that your call to marionette.client() invokes HostManager.createHost() in lib/runtime/hostmanager.js, which uses the mocha TDD infrastructure to decorate your suite with the following:

  • suiteSetup():
    • Host.create() in lib/runtime/host.js gets called, which causes createHost() in ChildRunner to get invoked.
      • This builds a profile from scratch if it doesn't exist, otherwise the existing profile is reused but settings/etc. are clobbered to be how we want them.
      • The "runner" process waits for the host to start-up.  It connects to the host with the async API and starts a session, then deletes it, and only generates the callback notification that will allow the "mocha test" process to know the host is ready.
  • setup():
    • Causes the driver to connect to the host at startup.
      • driver.resetWithDriver() gets called: this sounds scary, but it just resets internal state.  See marionette-js-client's lib/marionette/client.js.
      • All the plugins on record get registered with the client.
      • client.startSession() gets called, presumably doing something.
  • teardown() #1:
    • client.deleteSession() gets called, presumably doing something.
  • teardown() #2:
    • Host.restart() in lib/runtime/host.js gets called, which causes restartHost() in ChildRunner to get invoked.
      • This calls stopHost under the hood and then reuses much of the createHost() logic except the existing remotes handle is reused in the call to _buildRemote.
  • suiteTeardown():

Writing integration tests

marionette-js-runner has some simple overviews that are worth checking out.  You can find js marionette client documentation here!

File naming

Integration tests are located in the test/integration/ directory.

See also

Revision Source

<div class="summary">
 <p><span class="seoSummary">This document covers running and writing integration tests for Gaia apps, and provides a detailed explanation of how the integration tests are run.</span></p>
</div>
<h2 id="Running_tests">Running tests</h2>
<p>This section looks at setting up your envinronment correctly to run the existing integration test suite. Note that tests live with the rest of the apps code (in <code>apps/my_app/test/marionette</code> for example) and test files always end in <code>_test.js</code>. Shared code for tests lives under <code>shared/test/integration</code>.</p>
<h3 id="Setup">Setup</h3>
<p>To setup your environment for running tests:</p>
<ol>
 <li>Make sure you have <a href="https://github.com/mozilla-b2g/marionette-js-runner">marionette-js-runner</a> installed by using the below command:
  <pre class="brush: bash">
npm install --save-dev marionette-client marionette-js-runner
</pre>
  Gaia uses&nbsp;this to run the integration tests with a custom profile builder for Gaia.</li>
 <li>All integration tests run under a node environment. You need node 0.10 or higher for this to work predictably (<a href="https://github.com/joyent/node/wiki/Installing-Node.js-via-package-manager">detailed instructions on installing node on different platforms</a>).</li>
</ol>
<h3 id="Running_all_the_tests">Running all the tests</h3>
<p>The following command runs all the integration tests for all the apps:</p>
<pre class="brush: bash">
make test-integration
</pre>
<p>You will get to watch the tests pop up on your screen and you will probably also get to hear various sounds as they run.&nbsp; This can be distracting or possibly maddening. To run specific tests you can set the TEST_FILES environment variable. To do so in the bash shell (default on Mac OS X), use a command of the form:</p>
<pre class="brush: bash">
<span class="nv">TEST_FILE</span><span class="o">S=</span>&lt;<span class="nb">test</span>&gt; make <span class="nb">test</span>-integration</pre>
<p>For example, you could run the <code>day_view_test.js</code> test in calendar app with the below command.</p>
<pre class="brush: bash">
TEST_FILES=./apps/calendar/test/marionette/day_view_test.js make test-integration</pre>
<p>If you would like to run more than one test, you could do the following:</p>
<pre class="brush: bash">
TEST_FILES="./apps/calendar/test/marionette/day_view_test.js ./apps/calendar/test/marionette/today_test.js" make test-integration</pre>
<h3 id="Invoking_tests_for_a_specific_app">Invoking tests for a specific app</h3>
<p>To run just the tests for a specific app, you can use the following form:</p>
<div class="highlight highlight-sh">
 <pre class="brush: bash">
make <span class="nb">test</span>-integration <span class="nv">APP</span><span class="o">=</span>&lt;APP&gt;
</pre>
</div>
<p>For example, you could run all tests for the calendar app with</p>
<pre class="brush: bash">
make test-integration APP=calendar</pre>
<h3 id="Running_the_tests_in_the_background.2C_quietly">Running the tests in the background, quietly</h3>
<p>You might want to run the tests in the background, meaning that you won't have to see the tests run, worry about them stealing your focus, or risk impacting the running tests. One solution is to use Xvfb:</p>
<div class="highlight highlight-sh">
 <pre class="brush: bash">
xvfb-run make <span class="nb">test</span>-integration
</pre>
</div>
<p>If you are using PulseAudio and want to keep the tests quiet, then you can specify <code><span class="nv">PULSE_SERVER</span><span class="o">=</span><span class="s2">":"</span></code> to force an invalid server so no sound is output:</p>
<pre class="brush: bash">
PULSE_SERVER=":" make test-integration</pre>
<p>You can of course combine both:</p>
<pre class="brush: bash">
PULSE_SERVER=":" xvfb-run make test-integration</pre>
<h3 id="Running_tests_without_building_a_profile">Running tests without building a profile</h3>
<p>if you would like to run tests without building a profile, use <code>make test-integration-test</code>:</p>
<pre class="brush: bash">
PROFILE_FOLDER=profile-test make # generate profile directory in first time
make test-integration-test</pre>
<h3 id="Debugging_Tests">Debugging Tests</h3>
<p>To view gecko log output from a test, use the following:</p>
<pre class="brush: bash">
VERBOSE=1 make test-integration</pre>
<h2 id="Use_cases_and_design_patterns">Use cases and design patterns</h2>
<p>This section explores some use cases and design patterns that can help make the js marionette tests we write more maintainable.</p>
<h3 id="What_are_common_things_for_ui_tests_to_do.3F">What are common things for ui tests to do?</h3>
<div class="note">
 <p>Marionette tests simulate users interacting with apps.</p>
</div>
<p>In the abstract, your marionette tests will test your app by interacting with it as a user does. What do Firefox OS app users do? Here are some examples:</p>
<ul>
 <li>launch an app,</li>
 <li>click inside of an app,</li>
 <li>read text in an area of an app,</li>
 <li>wait for the ui to change,</li>
 <li>type in input fields,</li>
 <li>fill out forms,</li>
 <li>switch between apps,</li>
 <li>close apps, and</li>
 <li>swipe inside of an app</li>
</ul>
<p>Amazingly, you can simulate all of these user actions using marionette! You can find detailed, code-level documentation for the basic tasks <a href="http://mozilla-b2g.github.io/marionette-js-client/api-docs/">here</a>. For more advanced interactions, marionette js has a rich collection of plugins including ones for <a href="https://github.com/mozilla-b2g/marionette-apps">app management</a>, <a href="https://github.com/mozilla-b2g/marionette-plugin-forms">form completion</a>, <a href="https://github.com/mozilla-b2g/marionette-content-script">injecting scripts</a>, <a href="https://github.com/mozilla-b2g/marionette-file-manager">reading and writing to device storage</a>, <a href="https://github.com/mozilla-b2g/marionette-settings-api">editing settings</a>, and <a href="https://github.com/mozilla-b2g/marionette-helper">anything else your heart desires</a>!</p>
<h3 id="Marionette_is_faster_than_my_application!">Marionette is faster than my application!</h3>
<p>An extremely common issue is that marionette performs UI actions much faster than the average user. Imagine, for instance, that we had an application with two simple views each with a button to navigate to the other. We might expect for the following snippet of pseudocode to work:</p>
<div class="warning">
 <p>This is an example of what not to do. Most marionette tests that fail intermittently exhibit the following behavior.</p>
</div>
<pre style="margin-left: 40px;">
<code class="brush: js">view1.navigateToOther();</code>
<code>view2.navigateToOther();</code>
<code>assert.ok(view1.isDisplayed());</code></pre>
<p>This simple test tries to navigate from one view to the other, navigate back, and then make sure that the original view is displayed. This test will fail intermittently since there's a race condition between your application code which renders the two views and your test code which makes the implicit (and incorrect) assumption that the ui is responding synchronously. Instead, you need to program your test to behave like a user. Users don't try to click things or read things before they show up in the UI. Users "poll" the UI by looking at it until the thing they're waiting for shows up. Here's a better version of our test:</p>
<pre style="margin-left: 40px;">
<code class="brush: js">client.waitFor(view1.isDisplayed.bind(view1));</code>
<code>view1.navigateToOther();</code>
<code>client.waitFor(view2.isDisplayed.bind(view2));</code>
<code>view2.navigateToOther();</code>
<code>client.waitFor(view1.isDisplayed.bind(view1));</code></pre>
<div class="note">
 <p><a href="http://mozilla-b2g.github.io/marionette-js-client/api-docs/classes/Marionette.Client.html#method_waitFor">Marionette.Client#waitFor</a> is a utility that blocks your test execution until the parameter function returns true.</p>
</div>
<h3 id="How_should_I_structure_my_ui_test_libraries.3F">How should I structure my ui test libraries?</h3>
<p>Most of the test code you will write won't be general purpose enough to warrant abstracting into a general-purpose plugin. For app-specific code, we recommend having a separate file/class/module for each view. We demonstrate this pattern in gaia's <a href="https://github.com/mozilla-b2g/gaia/tree/bc5c83ad7e38cb588794f3e4692b3f0d63025c6a/apps/calendar/test/marionette">calendar test libraries</a>. The <a href="https://github.com/mozilla-b2g/gaia/tree/bc5c83ad7e38cb588794f3e4692b3f0d63025c6a/apps/calendar/test/marionette/lib/views">views folder</a> has different a class for each calendar app view. The general <a href="https://github.com/mozilla-b2g/gaia/tree/bc5c83ad7e38cb588794f3e4692b3f0d63025c6a/apps/calendar/test/marionette/lib/calendar.js">calendar object</a> has methods to navigate between the different views. In each of the views, we "scope" methods to search for elements in the DOM to the view's root element.</p>
<p>Oftentimes the views will contain getter methods to read an element or data from the view and setter methods to write data to an &lt;input&gt; or &lt;select&gt; tag. A good example is <a href="https://github.com/mozilla-b2g/gaia/tree/bc5c83ad7e38cb588794f3e4692b3f0d63025c6a/apps/calendar/test/marionette/lib/calendar.js#L119">creating a calendar event</a> which involves calling setter methods on the <a href="https://github.com/mozilla-b2g/gaia/tree/bc5c83ad7e38cb588794f3e4692b3f0d63025c6a/apps/calendar/test/marionette/lib/views/edit_event.js">edit event view</a> and saving. It's also idiomatic to abstract multiple ui actions which form a single "logical" action into view methods. For instance, <a href="https://github.com/mozilla-b2g/gaia/blob/master/apps/calendar/test/marionette/lib/views/edit_event.js#L47">setting a collection of reminders for a calendar event</a> involves tapping a select option for each reminder.</p>
<h2 id="How_the_test_runner_works">How the test runner works</h2>
<p>This section provides a detailed review of how the test runner works, for those that are interested.</p>
<div class="note">
 <p><strong>Note</strong>: All of the various ways the Marionette JavaScript integration tests are run happens via the same make <code>test-integration</code> path.&nbsp; Only parameters are changed.</p>
</div>
<h3 id="What_triggers_the_tests.3F">What triggers the tests?</h3>
<p>On <a href="/en-US/docs/">travis</a>:</p>
<ul>
 <li>Travis gets told what to do in our <a href="https://github.com/mozilla-b2g/gaia/blob/master/.travis.yml">.travis.yml</a> file.</li>
 <li>This sets up an xvfb environment.</li>
 <li>All of our jobs are run through the <code>travisaction</code> helper script from our <a href="https://github.com/mozilla-b2g/travis-project-jobs">travis-project-jobs repo</a>.&nbsp; The key things to know about this script are that it looks in a special directory for a subdirectory matching whatever <code>CI_ACTION</code> it is given.&nbsp; The config file that controls this directory is <a href="https://github.com/mozilla-b2g/gaia/blob/master/travisaction.opts">gaia/travisaction.opts</a>, which points at <a href="https://github.com/mozilla-b2g/gaia/tree/master/tests/travis_ci">gaia/tests/travis_ci</a> for us.</li>
 <li>So for <code>CI_ACTION=marionette_js</code> we look for the scripts in <a href="https://github.com/mozilla-b2g/gaia/tree/master/tests/travis_ci/marionette_js">gaia/tests/travis_ci/marionette_js</a>.</li>
 <li>The current contents of <a href="https://github.com/mozilla-b2g/gaia/blob/master/tests/travis_ci/marionette_js/script">gaia/tests/travis_ci/marionette_js/script</a> are:
  <pre>
<span class="nv">TEST_MANIFEST</span><span class="o">=</span>./shared/test/integration/travis-manifest.json
    <span class="nv">REPORTER</span><span class="o">=</span>spec make <span class="nb">test</span>-integration</pre>
 </li>
 <li><a href="https://github.com/mozilla-b2g/gaia/blob/master/shared/test/integration/travis-manifest.json">travis-manifest.json</a> is a JSON file containing a list of blacklisted tests.&nbsp; There are also manifests in the <a href="https://github.com/mozilla-b2g/gaia/tree/master/shared/test/integration">gaia/shared/test/integration directory</a> for <a href="https://github.com/mozilla-b2g/gaia/blob/master/shared/test/integration/local-manifest.json">local</a> and <a href="https://github.com/mozilla-b2g/gaia/blob/master/shared/test/integration/tbpl-manifest.json">tbpl</a> runs.</li>
</ul>
<p>On TBPL:</p>
<ul>
 <li>The build automation runs the following:
  <pre>
make test-integration NPM_REGISTRY=http://npm-mirror.pub.build.mozilla.org
REPORTER=mocha-tbpl-reporter TEST_MANIFEST=./shared/test/integration/tbpl-manifest.json</pre>
 </li>
 <li>The custom <code>NPM_REGISTRY</code> may or may not continue to exist; our <code>node_modules</code> are now sourced from our <a href="https://github.com/mozilla-b2g/gaia-node-modules">gaia-node-modules repo</a>.</li>
 <li>The custom reporter <a href="https://github.com/mozilla-b2g/mocha-tbpl-reporter">mocha-tbpl-reporter</a> just prints out <em>TEST-START / TEST-PASS / TEST-UNEXPECTED-FAIL / TEST-END</em> that the parsers consuming TBPL output expect, instead of all of those cool checkmarks you get when using the (default) spec reporter we use locally and on Travis.&nbsp; Note that the time-stamps from the TBPL output are much more realistic than what the spec reporter claims for test durations.&nbsp; Do not believe the spec reporter: it doesn't include your setup times!</li>
 <li>The <a href="https://github.com/mozilla-b2g/gaia/blob/master/shared/test/integration/tbpl-manifest.json">tbpl-manifest.json</a> manifest lists tests specifically blacklisted on <code>TBPL/</code></li>
</ul>
<p>Locally:</p>
<ul>
 <li>Whatever you want, you got it.</li>
</ul>
<h3 id="Okay.2C_what_does_make_test-integration_actually_do.3F">Okay, what does make test-integration actually do?</h3>
<ul>
 <li>The <a href="https://github.com/mozilla-b2g/gaia/blob/master/Makefile">Makefile</a> invokes <a href="https://github.com/mozilla-b2g/gaia/blob/master/bin/gaia-marionette">bin/gaia-marionette</a>, passing <code>MARIONETTE_RUNNER_HOST</code>, <code>TEST_MANIFEST</code>, and <code>REPORTER</code> as command-line arguments pulled out of the make environment.
  <ul>
   <li><code>MARIONETTE_RUNNER_HOST</code> defaults to <a href="https://github.com/mozilla-b2g/marionette-b2gdesktop-host">marionette-b2gdesktop-host</a> (run in B2G Desktop) if you don't explicitly specifying something else like <a href="https://github.com/mozilla-b2g/marionette-firefox-host">marionette-firefox-host</a> (run in Firefox) or <a href="https://github.com/mozilla-b2g/marionette-device-host">marionette-device-host</a> (run on a device, not expected to work yet).</li>
   <li><code>TEST_MANIFEST</code> defaults to <a href="https://github.com/mozilla-b2g/gaia/blob/master/shared/test/integration/local-manifest.json">shared/test/integration/local-manifest.json</a> and right now is just a list of blacklisted tests.&nbsp; The logic for manifests lives in <a href="https://github.com/mozilla-b2g/marionette-js-runner">marionette-js-runner</a> / <a href="https://github.com/mozilla-b2g/marionette-js-runner/blob/master/lib/applymanifest.js">lib/applymanifest.js</a>.</li>
   <li><code>REPORTER</code> defaults to spec which is a standard <a href="http://visionmedia.github.io/mocha/">mocha</a> reporter.&nbsp; <a href="http://visionmedia.github.io/mocha/#spec-reporter">Check it out on the mocha docs page</a>.</li>
  </ul>
 </li>
 <li><a href="https://github.com/mozilla-b2g/gaia/blob/master/bin/gaia-marionette">bin/gaia-marionette</a> is a wrapper around <a href="https://github.com/mozilla-b2g/marionette-js-runner">marionette-js-runner</a>'s <a href="https://github.com/mozilla-b2g/marionette-js-runner/blob/master/bin/marionette-mocha">bin/marionette-mocha</a> and generally appears to be a place to add a few more defaults, duplicate a bunch of work already done in the Makefile, and generately be a place to cram stuff if you don't understand Makefiles / want to make things take longer to run by avoiding parallelization.&nbsp; The good news is that because of all this you can invoke it directly.&nbsp; The notable bits in here are:
  <ul>
   <li>It uses "find" to locate all potential tests.&nbsp; If <code>APP</code> is in the environment, it only looks for tests under a given app.&nbsp; It will find blacklisted test files, but these are filtered out by marionette-mocha.</li>
   <li><a href="https://github.com/mozilla-b2g/gaia/blob/master/shared/test/integration/setup.js">gaia/shared/test/integration/setup.js</a> is specified as the first script to run when we spin up the mocha runtime.&nbsp; This currently is just a place to require and add plugins for use across all app tests.</li>
   <li><a href="https://github.com/mozilla-b2g/gaia/blob/master/shared/test/integration/profile.js">gaia/shared/test/integration/profile.js</a> is specified as the base configuration for all marionette integration tests' Gecko profiles.&nbsp; Adding things to this file will cause all tests to have certain prefs set, settings set, and/or apps installed.&nbsp; You almost certainly should not add things to this file.</li>
   <li><a href="https://github.com/mozilla-b2g/gaia/blob/master/shared/test/integration/profile_builder.js">gaia/shared/test/integration/profile_builder.js</a> is specified as the profile builder.&nbsp; It uses the Gaia Makefile, <a href="https://github.com/mozilla-b2g/mozilla-profile-builder">mozilla-profile-builder</a> in conjunction with <a href="https://github.com/mozilla-b2g/gaia/blob/master/shared/test/integration/profile.js">profile.js</a> above and specific per-test-file settings you specify in your <code>marionette.client({ ... })</code> call to configure your profile.&nbsp; We discuss the actual steps, when they actually happen at runtime, below.</li>
  </ul>
 </li>
 <li><a href="https://github.com/mozilla-b2g/marionette-js-runner">marionette-js-runner</a>'s <a href="https://github.com/mozilla-b2g/marionette-js-runner/blob/master/bin/marionette-mocha">bin/marionette-mocha</a> applies its own defaults that don't matter to us, applies the manifest rules in <a href="https://github.com/mozilla-b2g/marionette-js-runner/blob/master/lib/applymanifest.js">lib/applymanifest.js</a> to filter out blacklisted tests and spins up a <a href="https://github.com/mozilla-b2g/marionette-js-runner/blob/master/lib/parentrunner.js">ParentRunner</a> instance.</li>
 <li>The <a href="https://github.com/mozilla-b2g/marionette-js-runner/blob/master/lib/parentrunner.js">ParentRunner</a> instantiates a <a href="https://github.com/mozilla-b2g/marionette-js-runner/blob/master/lib/childrunner.js">ChildRunner</a>.&nbsp; The ParentRunner is boring and its name is somewhat misleading.&nbsp; Both the ParentRunner and ChildRunner live entirely in the same "runner" process.&nbsp; There is currently only ever one ChildRunner.</li>
</ul>
<p>Now things get somewhat complicated, so let's take a second to get an overview of all the processes that can be active and how they work:</p>
<ul>
 <li>The runner process. This is the <a href="https://github.com/mozilla-b2g/marionette-js-runner/blob/master/bin/marionette-mocha">bin/marionette-mocha</a> process with the <a href="https://github.com/mozilla-b2g/marionette-js-runner/blob/master/lib/parentrunner.js">ParentRunner</a> and <a href="https://github.com/mozilla-b2g/marionette-js-runner/blob/master/lib/childrunner.js">ChildRunner</a>.
  <ul>
   <li>It is a node.js environment.</li>
   <li>None of your test code runs in this process!</li>
   <li>It forks the <a href="https://github.com/visionmedia/mocha">mocha</a> / <a href="https://github.com/visionmedia/mocha/blob/master/bin/_mocha">bin/_mocha</a> node.js script to be the "mocha test" (sub-)process.
    <ul>
     <li>This is done using the node.js <a href="http://nodejs.org/api/child_process.html#child_process_child_process_fork_modulepath_args_options">child_process.fork</a> mechanism.&nbsp; This allows the "runner" process to <a href="http://nodejs.org/api/child_process.html#child_process_child_send_message_sendhandle">send</a> messages to the child and receive <a href="http://nodejs.org/api/child_process.html#child_process_event_message">message</a> events in return.</li>
    </ul>
   </li>
   <li>It gets bossed around by the "mocha test" process.&nbsp; Literally.&nbsp; The messages get received and converted into lookups on the ChildRunner object with <code>apply()</code> then called and a callback generated that will marshal the result back down to the client.&nbsp; This means that the runner process can and is told to:
    <ul>
     <li>Start/stop/restart "hosts".&nbsp; Hosts are your B2G Desktop/Firefox/B2G devices.</li>
    </ul>
   </li>
  </ul>
 </li>
 <li>The mocha test process.&nbsp; This is the <a href="https://github.com/visionmedia/mocha">mocha</a> / <a href="https://github.com/visionmedia/mocha/blob/master/bin/_mocha">bin/_mocha</a> node.js script.
  <ul>
   <li>It is a node.js environment and this is the stock mocha test runner.</li>
   <li>This is where your test code runs!</li>
   <li>The mocha runner gets told about the following files, in the following order:
    <ul>
     <li><a href="https://github.com/mozilla-b2g/marionette-js-runner">marionette-js-runner</a>'s <a href="https://github.com/mozilla-b2g/marionette-js-runner/blob/master/lib/runtime.js">lib/runtime.js</a>
      <ul>
       <li>This is where that "marionette" global comes from that you see in your tests!</li>
      </ul>
     </li>
     <li><a href="https://github.com/mozilla-b2g/gaia/blob/master/shared/test/integration/setup.js">gaia/shared/test/integration/setup.js</a>
      <ul>
       <li>This just loads and configures a bunch of marionette plugins to run during every step.</li>
      </ul>
     </li>
     <li>All of the test files that <a href="https://github.com/mozilla-b2g/marionette-js-runner/blob/master/bin/marionette-mocha">bin/marionette-mocha</a> was made aware of, that made it through the manifest's blacklists/whitelists.</li>
    </ul>
   </li>
   <li>See below for details on the execution model and how a test run actually works.</li>
  </ul>
 </li>
 <li>The "host" process(es).&nbsp; These are B2G Desktop/Firefox/(magic adb control wrapper) instances.
  <ul>
   <li>These are Gecko processes, potentially a hierarchy of processes now that OOP is in play.</li>
   <li>They are communicated with via the Marionette protocol.</li>
  </ul>
 </li>
 <li>"server" processes: These are test/fake servers spun up by various tests.&nbsp; These currently all seem to be spun up by the "mocha test" processes directly, but it's conceivable the "runner" process could also spin some up.&nbsp; Some known examples:
  <ul>
   <li>A test web-server.&nbsp; <a href="https://github.com/mozilla-b2g/gaia/blob/master/shared/test/integration/server.js">gaia/shared/test/integration/server.js</a> forks off a subprocess that runs <a href="https://github.com/mozilla-b2g/gaia/blob/master/shared/test/integration/server_child.js">gaia/shared/test/integration/server_child.js</a> and communicates via the same <code>child_process</code> message-passing mechanism that the "runner" and "mocha test" processes use.
    <ul>
     <li>there are duplicates of this implementation in <a href="https://github.com/mozilla-b2g/gaia/blob/master/apps/browser/test/marionette/lib/server.js">browser</a> and <a href="https://github.com/mozilla-b2g/gaia/blob/master/apps/homescreen/test/marionette/lib/server.js">homescreen</a> for some reason, likely historical.</li>
    </ul>
   </li>
   <li>The e-mail app has fake IMAP/POP3/SMTP servers that it shares with its back-end implementation.&nbsp; These live in the <a href="https://github.com/mozilla-b2g/mail-fakeservers">mail-fakeservers repo</a>.&nbsp; The fake-servers actually run in an XPCShell Gecko-style instance that initially communicates via a low-level IPC interface using <a href="https://github.com/lightsofapollo/json-wire-protocol/">json-wire-protocol</a> to bootstrap equivalence with the back-end tests' mechanism and then afterwards with a synchronous HTTP REST interface for all the e-mail domain work.&nbsp; It's due for some cleanup.</li>
  </ul>
 </li>
</ul>
<h3 id="What_is_in_my_global_scope_in_my_test_file.3F">What is in my global scope in my test file?</h3>
<ul>
 <li>Marionette: this comes from <a href="https://github.com/mozilla-b2g/marionette-js-runner/blob/master/lib/runtime.js">lib/runtime.js</a>.&nbsp; It is a function that's a wrapper around mocha's TDD suite() method.&nbsp; Source is in <a href="https://github.com/mozilla-b2g/marionette-js-runner/blob/master/lib/runtime/marionette.js">lib/runtime/marionette.js</a>.&nbsp; It has the following notable properties annotated onto it:
  <ul>
   <li>client(stuffToAddToTheProfile, driver that defaults to the synchronous Marionette API): Bound to HostManager.createHost from <a href="https://github.com/mozilla-b2g/marionette-js-runner/blob/master/lib/runtime/hostmanager.js">lib/runtime/hostmanager.js</a>.</li>
   <li>plugin(name, module): This is a bound function that invokes HostManager.addPlugin from <a href="https://github.com/mozilla-b2g/marionette-js-runner/blob/master/lib/runtime/hostmanager.js">lib/runtime/hostmanager.js</a>.&nbsp; It adds the plugin for the duration of your test suite.</li>
  </ul>
 </li>
 <li>The <a href="https://github.com/visionmedia/mocha/blob/master/lib/interfaces/tdd.js">TDD interface</a> pokes setup/teardown/suiteSetup/suiteTeardown/suite/test into the global namespace.</li>
</ul>
<h3 id="What_are_marionette_plugins.3F">What are marionette plugins?</h3>
<ul>
 <li>Marionette plugins are node.js modules which extend the native abilities of the js marionette client.</li>
 <li>They conform to the following simple plugin API:</li>
</ul>
<pre>
MarionettePlugin = {
  /**
   * @param {Marionette.Client} client the marionette client to extend.
   * @param {Object} options plugin-specific configuration.
   * @return {MarionettePlugin} appropriately configured plugin.
   */
  setup: function(client, options) {}
};</pre>
<ul>
 <li>They get exposed through the client (i.e. client.apps).</li>
 <li>Existing plugins include
  <ul>
   <li><a href="https://github.com/mozilla-b2g/marionette-apps">marionette-apps</a> - utility to find, open, close, and switch between apps</li>
   <li><a href="https://github.com/mozilla-b2g/marionette-helper">marionette-helper</a> - miscellaneous helper functions for interacting with the DOM</li>
   <li><a href="https://github.com/mozilla-b2g/marionette-js-logger">marionette-js-logger</a> - proxies console.* calls from gecko to node.js (notably used internally by marionette-js-runner)</li>
   <li><a href="https://github.com/mozilla-b2g/marionette-plugin-forms">marionette-plugin-forms</a> - makes interacting with forms through marionette easier</li>
  </ul>
 </li>
</ul>
<h3 id="What_is_mocha's_TDD_execution_model.3F">What is mocha's TDD execution model?</h3>
<ul>
 <li>The mocha runner <code>require()</code>s all of the files it is told about in sequence in <a href="https://github.com/visionmedia/mocha/blob/master/lib/mocha.js">Mocha.prototype.loadFiles</a>().&nbsp; This consists of:
  <ul>
   <li>synchronously emitting a <code>pre-require</code> event.
    <ul>
     <li>The <a href="https://github.com/visionmedia/mocha/blob/master/lib/interfaces/tdd.js">TDD interface</a> setup/teardown/suiteSetup/suiteTeardown/suite/test into the global namespace.</li>
    </ul>
   </li>
   <li>synchronously requiring the file in question.
    <ul>
     <li>This means the top level of your file is executed.</li>
     <li>Any <code>marionette()</code> calls to define a suite (since Marionette is just a wrapper around <code>suite()</code>) will in turn have their defining function executed synchronously.</li>
     <li>The <code>test()</code> calls inside the <code>marionette()</code> calls are NOT executed.&nbsp; Those don't happen until the test is actually run.</li>
     <li>This does mean you have to be very careful not to do anything foolish in the top-level of your file or the <code>marionette()</code> file.</li>
    </ul>
   </li>
   <li>synchronously emitting a 'require' event passing the module from the file in question.</li>
   <li>synchronously emitting a 'post-require' event.</li>
  </ul>
 </li>
 <li>The <a href="https://github.com/visionmedia/mocha/blob/master/lib/runner.js">Runner</a> runs the suites and tests in sequence.
  <ul>
   <li>emits <code>start</code>.</li>
   <li><code>runSuite</code> loops over all the suites, for each one:
    <ul>
     <li>emits <code>suite</code>.</li>
     <li>emits <code>beforeAll</code>, which is what <code>suiteSetup</code> maps to.</li>
     <li>calls <code>runTests</code> loops over all the tests, for each one:
      <ul>
       <li>emits <code>test</code>.</li>
       <li>emits <code>beforeEach</code>, which is what <code>setup</code> maps to.</li>
       <li>calls <code>runTest</code>, which runs your test function.</li>
       <li>emits <code>test end</code>.
        <ul>
         <li>This runs <code>checkGlobals()</code> to make sure you didn't clobber anything unexpected into the global state.&nbsp; It fails your test if you did.</li>
        </ul>
       </li>
       <li>emits <code>afterEach</code>, which is what <code>teardown</code> maps to.</li>
      </ul>
     </li>
     <li>emits <code>afterAll</code>, which is what <code>suiteTeardown</code> maps to.</li>
     <li>emits <code>suite end</code>.</li>
    </ul>
   </li>
   <li>emits <code>end</code>.</li>
  </ul>
 </li>
</ul>
<h3 id="What_is_the_life-cycle_of_the_Gecko_processes.3F">What is the life-cycle of the Gecko processes?</h3>
<p>For your test suite (aka each top-level <code>marionette('description', function() {...})</code> in your file), a new profile is created.</p>
<p>For each test (aka each <code>test('description', function() {...})</code> in your <code>marionette('description', ...)</code>), the host gets restarted after each <code>test()</code>.</p>
<p>The nitty gritty of this is that your call to <code>marionette.client()</code> invokes <code>HostManager.createHost()</code> in <a href="https://github.com/mozilla-b2g/marionette-js-runner/blob/master/lib/runtime/hostmanager.js">lib/runtime/hostmanager.js</a>, which uses the mocha TDD infrastructure to decorate your suite with the following:</p>
<ul>
 <li><code>suiteSetup()</code>:
  <ul>
   <li><code>Host.create()</code> in <a href="https://github.com/mozilla-b2g/marionette-js-runner/blob/master/lib/runtime/host.js">lib/runtime/host.js</a> gets called, which causes <code>createHost()</code> in <a href="https://github.com/mozilla-b2g/marionette-js-runner/blob/master/lib/childrunner.js">ChildRunner</a> to get invoked.
    <ul>
     <li>This builds a profile from scratch if it doesn't exist, otherwise the existing profile is reused but settings/etc. are clobbered to be how we want them.</li>
     <li>The "runner" process waits for the host to start-up.&nbsp; It connects to the host with the async API and starts a session, then deletes it, and only generates the callback notification that will allow the "mocha test" process to know the host is ready.</li>
    </ul>
   </li>
  </ul>
 </li>
 <li><code>setup()</code>:
  <ul>
   <li>Causes the driver to connect to the host at startup.
    <ul>
     <li><code>driver.resetWithDriver()</code> gets called: this sounds scary, but it just resets internal state.&nbsp; See <a href="https://github.com/mozilla-b2g/marionette-js-client">marionette-js-client</a>'s <a href="https://github.com/mozilla-b2g/marionette-js-client/blob/master/lib/marionette/client.js">lib/marionette/client.js</a>.</li>
     <li>All the plugins on record get registered with the client.</li>
     <li><code>client.startSession()</code> gets called, presumably doing something.</li>
    </ul>
   </li>
  </ul>
 </li>
 <li><code>teardown()</code> #1:
  <ul>
   <li><code>client.deleteSession()</code> gets called, presumably doing something.</li>
  </ul>
 </li>
 <li><code>teardown()</code> #2:
  <ul>
   <li><code>Host.restart()</code> in <a href="https://github.com/mozilla-b2g/marionette-js-runner/blob/master/lib/runtime/host.js">lib/runtime/host.js</a> gets called, which causes <code>restartHost()</code> in <a href="https://github.com/mozilla-b2g/marionette-js-runner/blob/master/lib/childrunner.js">ChildRunner</a> to get invoked.
    <ul>
     <li>This calls <code>stopHost</code> under the hood and then reuses much of the <code>createHost()</code> logic except the existing <code>remotes</code> handle is reused in the call to <code>_buildRemote</code>.</li>
    </ul>
   </li>
  </ul>
 </li>
 <li><code>suiteTeardown()</code>:
  <ul>
   <li><code>Host.stop()</code> in <a href="https://github.com/mozilla-b2g/marionette-js-runner/blob/master/lib/runtime/host.js">lib/runtime/host.js</a> gets called, which causes <code>stopHost()</code> in <a href="https://github.com/mozilla-b2g/marionette-js-runner/blob/master/lib/childrunner.js">ChildRunner</a> to get invoked.</li>
  </ul>
 </li>
</ul>
<h2 id="Writing_integration_tests">Writing integration tests</h2>
<p>marionette-js-runner <a href="https://github.com/mozilla-b2g/marionette-js-runner#marionetteclient-marionette-client-interface">has some simple overviews</a> that are worth checking out.&nbsp; You can find js marionette client documentation <a href="http://mozilla-b2g.github.io/marionette-js-client/api-docs/">here</a>!</p>
<h3 id="File_naming"><strong>File naming</strong></h3>
<p>Integration tests are located in the <code>test/integration/</code> directory.</p>
<h2 id="See_also">See also</h2>
<ul>
 <li><a href="https://github.com/mozilla-b2g/gaia/#integration-tests">Gaia integration tests info on Github</a>.</li>
 <li><a href="/en-US/docs/Marionette/Marionette_JavaScript_Tools" title="/en-US/docs/Marionette/Marionette_JavaScript_Tools">Marionette Javascript Tools</a> for a high level overview.</li>
 <li><a href="https://github.com/mozilla-b2g/marionette-js-runner/blob/master/README.md#exposed-apis-for-writing-marionette-tests" title="https://github.com/mozilla-b2g/marionette-js-runner/blob/master/README.md#exposed-apis-for-writing-marionette-tests">Code samples</a> on Github.</li>
 <li><a class="link-https" href="https://wiki.mozilla.org/Gaia/Hacking" title="https://wiki.mozilla.org/Gaia/Hacking">Hacking Gaia</a> for getting started with Gaia.</li>
 <li><a href="http://visionmedia.github.io/mocha/">mocha: which is wrapped by marionette-js-runner.</a></li>
 <li><a href="https://github.com/mozilla-b2g/marionette-js-runner">marionette-js-runner: for the test framework.</a></li>
 <li><a href="http://lightsofapollo.github.io/marionette_js_client/api-docs/classes/Marionette.Client.html">marionette-client: for anything to do with client.X.</a></li>
</ul>
Revert to this revision