mozilla

Revision 330785 of Gaia unit tests

  • Revision slug: Mozilla/Boot_to_Gecko/Gaia_Unit_Tests
  • Revision title: Gaia Unit Tests
  • Revision id: 330785
  • Created:
  • Creator: gbrander
  • Is current revision? No
  • Comment Fix formatting of launch_tests function.

Revision Content

Warning: This document assumes you fully understand how to work within Gaia and B2G.

See Hacking Gaia to get started with Gaia.

Running unit tests

All you need is Gaia and B2G desktop/Firefox Nightly. Make sure you use the following to generate your profile:

(note you must use DEBUG=1)

DEBUG=1 make

Running with B2G Desktop

Launch Gaia and start the "Test Agent" app. From the test agent app you can select tests to run from the UI.

Running with Firefox Nightly

First you need to force the DNS test-agent.gaiamobile.org to your computer. On Linux you just need to add this line in your /etc/hosts file:

127.0.0.1 test-agent.gaiamobile.org

Then you can launch Firefox Nightly:

cd <path to gaia>
<path to nightly>/firefox --no-remote -profile <path to gaia>/profile/ http://test-agent.gaiamobile.org:8080/

Using Firebug is possible and makes it easy to debug the tests; use the debugger keyword to break in Firebug's debugger. Currently, Firefox' own debugger doesn't work well as it doesn't refresh the files.

Advanced: launch from the CLI and launch-as-you-save

For a more advanced test runner with CLI based reporting you must have Node.js and NPM installed.

(Note that if following commands fail with some cryptic errors while installing test-agent dependencies, it might be due because of too old Node.js/NPM version. Checkout this page in order to install very last versions and delete /tools/test-agent/node_modules folder.)

TLDR;

  1. run: make test-agent-server &
  2. run B2G desktop/Firefox Nightly and launch "Test Agent" app
  3. run: make test-agent-test

WebSocket server

Test agent (the test runner) ships with a built in WebSocket server that lets you remotely message the browser or device to queue a test run. Often you will want to develop with time saving features like a file watcher that will run your tests when a test file or implementation changes. To take advantage of these features you need to start the server.

make test-agent-server

Using the WebSocket server offers other tools such as a command line reporter for test results (watch the terminal you ran the command from), a Growl reporter, syntax error notifications and more.

The agent also watches for modifications in files, and automatically runs the associated tests. It runs when you save the test or if you save the tested file (we use the convention where the test filename is the tested filename with _test appended, see below for more examples). It watches only existing files so if you create a new file, you have to restart the agent.

Launch the server and Firefox Nightly

Add this to your .bashrc and then use the launch_tests command:

GAIADIR="<path to your gaia clone>"
NIGHTLY="<path to your firefox nightly>"

launch_tests() {
    cd $GAIADIR
    if [ ! -d profile/extensions/httpd ] ; then
        DEBUG=1 make
    fi

    $NIGHTLY/firefox --no-remote -profile $GAIADIR/profile/ http://test-agent.gaiamobile.org:8080/ &

    make test-agent-server
}

Running all the tests

Before checking your code in, it's a good idea to run all the tests. Doing this from the UI can be painful (right now anyway), so there is a CLI command to run all available tests.

Make sure the WebSocket server is running.

Run B2G desktop/Firefox Nightly and launch "Test Agent" app (see first part).

Then:

make test-agent-test

If you only want to run one app you can specify which via the APP env variable

make test-agent-test APP=calendar

You can also optionally provide a reporter to use to format the test output.

make REPORTER=List test-agent-test
Note: Not all reporters work, since we currently do not support Doc.

Setting up your Gaia app

Although this guide should help make things easier, the best way to learn how to write, set up, and run tests is currently still to look at the source code; in particular, take a look at the gallery tests.

Writing unit tests

The unit test runner is mocha using the TDD interface. Mocha doesn't ship with an assertion library (there is no ok or assert_equals), so we use chai to provide assertions.

It's highly recommended that you read though the mocha site, since all tests are really mocha tests. The documentation here is focused on getting you started, and about our special integrations with test-agent and Gaia.

It's also important to note that we add special functions (like require() and requireApp()) to make writing tests easier. All test helper scripts can be found in the /common/test directory.

File naming

Tests are usually one to one. One implementation, which lives in the js/ directory, and one test, which lives in the test/ directory.

Examples
Implementation Test
apps/app/js/file.js apps/app/test/unit/file_test.js
apps/app/js/nested/thing/file.js apps/app/test/unit/nested/thing/file_test.js

Example Implementation (gist):

//apps/example/js/example.js
var Example = (function(){
  return {
    math: function() {
      
    },
    
    complexMethod: function() {
      
    },
    
    asyncMethod: function(callback) {
    
    }
  }
}());

Example Test (gist):

//apps/example/test/unit/example_test.js
requireApp('example/js/example.js');


//suite/setup/test/done are standard mocha functionality.

suite('Example', function() {
  var subject;
  
  //will be called before each "test" block. 
  setup(function() {
    subject = Example();
  });
  
  //for a simple method
  test('#math', function() {
    var result = subject.math('1', '+', '1');
    //assert functionality is provided by chai
    assert.equal(result, 2, 'addition should work');
  });
  
  //there is full support for async tests using done
  //when you an argument to your test function it is
  //assumed that the given test is async and will only
  //complete once done is called.
  test('#asyncMethod', function(done) {
    subject.asyncMethod(function(err, value) {
      done(function() {
        assert.ok(value, 'sending message failed');      
      });
    });
  });
  
  //when you have a method that will
  //require complex setup/teardown logic
  suite('#complexMethod', function() {
    var result;
    setup(function() {
      //complex setup stuff
      result = subject.complexMethod();
    });
    
    test('stuff works', function() {
      assert.typeOf(result, 'string');
      //it is good practice to add the third argument which is
      //a description of why a test failed.
      assert.equal(result, 'real value', 'should output real value');
    });
  });
  
});

Using mocks

TBD

Revision Source

<div class="warning">
  <strong>Warning:</strong> This document assumes you fully understand how to work within Gaia and B2G.</div>
<p>See <a class="link-https" href="https://wiki.mozilla.org/Gaia/Hacking" title="https://wiki.mozilla.org/Gaia/Hacking">Hacking Gaia</a> to get started with Gaia.</p>
<h2 id="Running_unit_tests">Running unit tests</h2>
<p>All you need is Gaia and B2G desktop/Firefox Nightly. Make sure you use the following to generate your profile:</p>
<p>(note you <strong>must</strong> use DEBUG=1)</p>
<pre>
DEBUG=1 make
</pre>
<h3 id="Running_with_B2G_Desktop">Running with B2G Desktop</h3>
<p>Launch Gaia and start the "Test Agent" app. From the test agent app you can select tests to run from the UI.</p>
<h3 id="Running_with_Firefox_Nightly">Running with Firefox Nightly</h3>
<p>First you need to force the DNS <code>test-agent.gaiamobile.org</code> to your computer. On Linux you just need to add this line in your <code>/etc/hosts</code> file:</p>
<pre>
127.0.0.1 test-agent.gaiamobile.org</pre>
<p>Then you can launch Firefox Nightly:</p>
<pre>
cd &lt;path to gaia&gt;
&lt;path to nightly&gt;/firefox --no-remote -profile &lt;path to gaia&gt;/profile/ http://test-agent.gaiamobile.org:8080/</pre>
<p>Using Firebug is possible and makes it easy to debug the tests; use the <code>debugger</code> keyword to break in Firebug's debugger. Currently, Firefox' own debugger doesn't work well as it doesn't refresh the files.</p>
<h2 id="Advanced.3A_launch_from_the_CLI_and_launch-as-you-save">Advanced: launch from the CLI and launch-as-you-save</h2>
<p>For a more advanced test runner with CLI based reporting you must have <a class="external" href="http://nodejs.org/" title="http://nodejs.org/">Node.js</a> and <a class="external" href="http://npmjs.org/" title="http://npmjs.org/">NPM</a> installed.</p>
<p>(Note that if following commands fail with some cryptic errors while installing test-agent dependencies, it might be due because of too old Node.js/NPM version. Checkout <a href="https://github.com/joyent/node/wiki/Installing-Node.js-via-package-manager" title="https://github.com/joyent/node/wiki/Installing-Node.js-via-package-manager">this page</a> in order to install very last versions and delete <code>/tools/test-agent/node_modules</code> folder.)</p>
<h3 id="TLDR.3B">TLDR;</h3>
<ol>
  <li>run: <code>make test-agent-server &amp;</code></li>
  <li>run B2G desktop/Firefox Nightly and launch "Test Agent" app</li>
  <li>run: <code>make test-agent-test</code></li>
</ol>
<h3 id="WebSocket_server">WebSocket server</h3>
<p>Test agent (the test runner) ships with a built in WebSocket server that lets you remotely message the browser or device to queue a test run. Often you will want to develop with time saving features like a file watcher that will run your tests when a test file or implementation changes. To take advantage of these features you need to start the server.</p>
<pre>
make test-agent-server
</pre>
<p>Using the WebSocket server offers other tools such as a command line reporter for test results (watch the terminal you ran the command from), a Growl reporter, syntax error notifications and more.</p>
<p>The agent also watches for modifications in files, and automatically runs the associated tests. It runs when you save the test or if you save the tested file (we use the convention where the test filename is the tested filename with <code>_test</code> appended,<code> </code>see below for more examples). It watches only existing files so if you create a new file, you have to restart the agent.</p>
<h3 id="Launch_the_server_and_Firefox_Nightly">Launch the server and Firefox Nightly</h3>
<p>Add this to your .bashrc and then use the <code>launch_tests</code> command:</p>
<pre class="brush: bash">
GAIADIR="&lt;path to your gaia clone&gt;"
NIGHTLY="&lt;path to your firefox nightly&gt;"

launch_tests() {
    cd $GAIADIR
    if [ ! -d profile/extensions/httpd ] ; then
        DEBUG=1 make
    fi

    $NIGHTLY/firefox --no-remote -profile $GAIADIR/profile/ http://test-agent.gaiamobile.org:8080/ &amp;

    make test-agent-server
}</pre>
<h3 id="Running_all_the_tests">Running all the tests</h3>
<p>Before checking your code in, it's a good idea to run all the tests. Doing this from the UI can be painful (right now anyway), so there is a CLI command to run all available tests.</p>
<p>Make sure the WebSocket server is running.</p>
<p>Run B2G desktop/Firefox Nightly and launch "Test Agent" app (see first part).</p>
<p>Then:</p>
<pre>
make test-agent-test
</pre>
<p>If you only want to run one app you can specify which via the APP env variable</p>
<pre>
make test-agent-test APP=calendar
</pre>
<p>You can also optionally provide a <a class="external" href="http://visionmedia.github.com/mocha/#reporters" title="http://visionmedia.github.com/mocha/#reporters">reporter</a> to use to format the test output.</p>
<pre>
make REPORTER=List test-agent-test
</pre>
<div class="note">
  <strong>Note:</strong> Not all reporters work, since we currently do not support Doc.</div>
<h2 id="Setting_up_your_Gaia_app">Setting up your Gaia app</h2>
<p>Although this guide should help make things easier, the best way to learn how to write, set up, and run tests is currently still to look at the source code; in particular, take a look at the <a class="link-https" href="https://github.com/lightsofapollo/gaia/tree/master/apps/gallery/test" title="https://github.com/lightsofapollo/gaia/tree/master/apps/gallery/test">gallery tests</a>.</p>
<h3 id="Writing_unit_tests">Writing unit tests</h3>
<p>The unit test runner is <a class="external" href="http://visionmedia.github.com/mocha/" title="http://visionmedia.github.com/mocha/">mocha</a> using the <a class="external" href="http://visionmedia.github.com/mocha/index.html#tdd-interface" title="http://visionmedia.github.com/mocha/index.html#tdd-interface">TDD interface</a>. Mocha doesn't ship with an assertion library (there is no <code>ok</code> or <code>assert_equals</code>), so we use <a class="external" href="http://chaijs.com/api/assert/" title="http://chaijs.com/code/assert.html">chai</a> to provide assertions.</p>
<p>It's highly recommended that you read though the <a class="external" href="http://visionmedia.github.com/mocha/" title="http://visionmedia.github.com/mocha/">mocha</a> site, since all tests are really mocha tests. The documentation here is focused on getting you started, and about our special integrations with test-agent and Gaia.</p>
<p>It's also important to note that we add special functions (like <code>require()</code> and <code>requireApp()</code>) to make writing tests easier. All test helper scripts can be found in the <code><a class="link-https" href="https://github.com/andreasgal/gaia/tree/master/common/test" title="https://github.com/andreasgal/gaia/tree/master/common/test">/common/test</a></code> directory.</p>
<h4 id="File_naming"><strong>File naming</strong></h4>
<p>Tests are usually one to one. One implementation, which lives in the <code>js/</code> directory, and one test, which lives in the <code>test/</code> directory.</p>
<h5 id="Examples">Examples</h5>
<table class="standard-table" style="width: auto;">
  <tbody>
    <tr>
      <td class="header">Implementation</td>
      <td class="header">Test</td>
    </tr>
    <tr>
      <td>apps/app/js/file.js</td>
      <td>apps/app/test/unit/file_test.js</td>
    </tr>
    <tr>
      <td>apps/app/js/nested/thing/file.js</td>
      <td>apps/app/test/unit/nested/thing/file_test.js</td>
    </tr>
  </tbody>
</table>
<p><strong>Example Implementation (<a class="link-https" href="https://gist.github.com/2703499" title="https://gist.github.com/2703499">gist</a>):</strong></p>
<pre class="brush: js">
//apps/example/js/example.js
var Example = (function(){
  return {
    math: function() {
      
    },
    
    complexMethod: function() {
      
    },
    
    asyncMethod: function(callback) {
    
    }
  }
}());
</pre>
<p><strong>Example Test (<a class="link-https" href="https://gist.github.com/2703499" title="https://gist.github.com/2703499">gist</a>):</strong></p>
<pre class="brush: js">
//apps/example/test/unit/example_test.js
requireApp('example/js/example.js');


//suite/setup/test/done are standard mocha functionality.

suite('Example', function() {
  var subject;
  
  //will be called before each "test" block. 
  setup(function() {
    subject = Example();
  });
  
  //for a simple method
  test('#math', function() {
    var result = subject.math('1', '+', '1');
    //assert functionality is provided by chai
    assert.equal(result, 2, 'addition should work');
  });
  
  //there is full support for async tests using done
  //when you an argument to your test function it is
  //assumed that the given test is async and will only
  //complete once done is called.
  test('#asyncMethod', function(done) {
  &nbsp; subject.asyncMethod(function(err, value) {
  &nbsp; &nbsp; done(function() {
  &nbsp; &nbsp;   assert.ok(value, 'sending message failed');  &nbsp; &nbsp; 
  &nbsp; &nbsp; });
  &nbsp; });
  });
  
  //when you have a method that will
  //require complex setup/teardown logic
  suite('#complexMethod', function() {
    var result;
    setup(function() {
      //complex setup stuff
      result = subject.complexMethod();
    });
    
    test('stuff works', function() {
      assert.typeOf(result, 'string');
      //it is good practice to add the third argument which is
      //a description of why a test failed.
      assert.equal(result, 'real value', 'should output real value');
    });
  });
  
});
</pre>
<h3 id="Using_mocks">Using mocks</h3>
<p>TBD</p>
Revert to this revision