mozilla

Revision 513559 of Gaia UI Tests: Running Tests

  • Revision slug: Mozilla/Firefox_OS/Platform/Automated_testing/gaia-ui-tests/Gaia_UI_Tests_Run_Tests
  • Revision title: Gaia UI Tests: Running Tests
  • Revision id: 513559
  • Created:
  • Creator: janjongboom
  • Is current revision? No
  • Comment

Revision Content

This page isn't finished yet.
Your feedback to gaia-ui-automation@mozilla.org would be greatly appreciated.

Running Tests - developing tests and framework

 

The gaia-ui-tests have been developed with the capability to run on a device, emulators, and desktop B2G. By far the easiest method is to run against desktopb2g.

However, please note that you should choose the right branch of gaia to run against the version of Firefox OS that you are using.

 

Testing on Desktopb2g (Firefox OS desktop client)

Prequisites:

  • Desktopb2g downloaded from this location and unpacked to a directory on your computer or virtual machine

  • Gaia git repository cloned and checked out

  • testvars.json file created

The desktopb2g / desktop client is ideal for developing tests where a device is unavailable and there is no need for phone-level functions. It is fast, accessible and will run on Linux, Mac and Windows!

We want to develop new tests and that will involve modifying gaiatest. We'll install the in-tree version of gaiatest directly from the gaia repository so that our changes are picked up immediately. In your cloned version of gaia, navigate to the gaia-ui-tests directory and run the setup command:

> ~/Mozilla/gaia/tests/python/gaia-ui-tests$ python setup.py develop

Via Marionette, gaiatest is able to launch desktopb2g when we start the test. To do this we need to pass in the path to the binary file and the path to the profile. In this example I have included the $HOME variable as the path to the b2g binary and the b2g profile. This will simply be the location that you unzipped desktopb2g to after you downloaded it. The command to run all the tests is:

> ~/Mozilla/gaia/tests/python/gaia-ui-tests$ gaiatest --binary=$HOME/b2g/b2g-bin --profile=$HOME/b2g/gaia/profile --testvars=testvars_master.json --restart ./gaiatest/tests/functional/manifest.ini

Tests that are incompatible with the desktopb2g client, for example those that make calls and send SMS are automatically skipped from the test run.

You can run a single test file directly by calling the file directly on the command line or even several tests by calling them all on the command line

> ~/Mozilla/gaia/tests/python/gaia-ui-tests$ gaiatest --binary=$HOME/b2g/b2g-bin --profile=$HOME/b2g/gaia/profile --testvars=testvars_master.json --restart ./gaiatest/tests/functional/clock/test_clock_set_alarm.py

 

Testing on Firefox OS Devices

Testing on Firefox OS devices is by far the most complicated

Prerequisites:

  • Android debug bridge (adb) installed and udev rules configured
  • A Firefox OS device with a Marionette-enabled build
  • Gaia Git repository checked out for the branch matching your Firefox OS device
  • testvars.json file created

The gaiatest package is needed to run the tests. We need to install this for the branch that we want to run the tests on. Navigate to the tests/python/gaia-ui-tests/ directory and install gaiatest with this command:

> ~/Mozilla/gaia/tests/python/gaia-ui-tests$ python setup.py develop

Before you start to run the test, read through this warning in order to acknowledge that running gaia-ui-tests on a device may cause data to be deleted from the device!

Marionette on the device awaits commands on port 2828. We need to forward our local port to the remote port on the device using adb:

> $ adb forward tcp:2828 tcp:2828

Now, you can send this command to execute all gaia-ui-tests:

> ~/Mozilla/gaia/tests/python/gaia-ui-tests$ gaiatest --address=localhost:2828 --testvars=<TESTVARS_FILE> --restart --type=b2g gaiatest/tests/functional/manifest.ini

Or, if you want to run solely one test file, just do:

> ~/Mozilla/gaia/tests/python/gaia-ui-tests$ gaiatest --address=localhost:2828 --testvars=<TESTVARS_FILE> --restart gaiatest/functional/clock/test_clock_create_new_alarm.py

After the test, you will see all the pass/fail results and stacktraces for each file.

 

Running tests - debugging CI failures

 

Running tests using the Travis scripts (for Gecko and Gaia developers)

 

 

Testing locally with TBPL configuration (for Gecko and Gaia developers)

Prerequisites:

  • A desktopb2g build including the Gecko patch you are testing, either self-built or downloaded from TBPL

  • mozilla-b2g/gaia git repository checked out (for making the Gaia profile, installing gaiatest and the test files)

  • testvars.json file created

In this documentation I will focus upon replicating a TBPL test run locally. Before testing locally with TBPL configuration you need to be aware that it builds a gaia profile separately. The profile that comes packaged with the TBPL build or rel-eng build is NOT the same Gaia profile that TBPL uses.

Building a Gaia profile with TBPL's  settings

Before building a profile we need to check the environment variables that were used to build the profile. Searching in the "brief log" of the Gu test run, find this command block:

Using env: {'DEBUG': '0',
            'DESKTOP': '0',
            'DESKTOP_SHIMS': '1',
            'NOFTU': '0'}

These settings can drastically change the way desktopb2g behaves. It is very important to match TBPL's settings.

The next step is to checkout the git commit that matches the hg commit that TBPL ran with. You can also find the hg commit in the "brief log" of the TBPL run but you will need to correlate that with the git commit yourself.

The gaia test repository includes a make script for building a Gaia profile. Navigate to the gaia git repository that you have cloned locally. Build the profile using the environment variables we found on TBPL:

> ~/Mozilla/gaia$ DEBUG=0 DESKTOP=0 DESKTOP_SHIMS=1 NOFTU=0 make

Running the tests

Using the same git commit that TBPL used to run the test, install gaiatest from the gaia-ui-tests directory:

> ~/Mozilla/gaia/tests/python/gaia-ui-tests$ python setup.py develop

Now that we have built the Gaia profile, the only significant difference to running these tests is that we must direct gaiatest to the binary location and profile location. As TBPL runs an isolated environment it uses its own set of tests, set in tbpl-manifest.ini.

$BINARY_PATH - Path to the "b2g-bin" file in the directory you have built or unpacked desktopb2g to
$PROFILE_PATH - Path to your locally build Gaia profile. If built from gaia repository it will be ~/gaia/profile

Navigate to the gaia-ui-tests directory and run this command:

> ~/Mozilla/gaia/tests/python/gaia-ui-tests$ gaiatest --binary $BINARY_PATH/b2g/b2g-bin --profile $PROFILE_PATH/profile --restart --testvars testvars.json gaiatest/tests/tbpl-manifest.ini --type=b2g

There is no need to forward port 2828 unless you have set the marionette server to use a different default port.

Try to keep your mouse cursor clear of the b2g window so that pointer events do not pollute the test run.

 

Testing against an already running desktopb2g binary (advanced debugging for developers)

Prerequisites:

  • A B2G binary built with your Gecko changes

  • Debugger of choice attached to the B2G binary

  • mozilla-b2g/gaia git repository checked out (for installing gaiatest and the test files)

  • testvars.json file created

Running against an existing b2g process is quite easy - you've already gone through the hard parts to build B2G!

If you have not already installed gaiatest then do so now by running `python setup.py develop` from the gaia-ui-tests directory.

> ~/Mozilla/gaia/tests/python/gaia-ui-tests$ python setup.py develop

Once you have started the b2g binary process and attached your debugger we direct gaiatest to the port using the --address command. Do not use --binary or --profile, gaiatest will just send commands directly to the port without attempting to start or close the binary.

> ~/Mozilla/gaia/tests/python/gaia-ui-tests$ gaiatest --testvars testvars.json gaiatest/tests/functional/test_that_is_being_debugged.py

Note that we have omitted the --restart command too. As gaiatest does not attempt to stop or start the binary some data from your test run may be left behind in Firefox OS databases or even in the DOM. It is your responsibility to 'reset' B2G back to a basic state before trying to run the test for a second time.

 

Test Variables (testvars.json)

The gaia-ui-tests will wipe the databases on your phone in order to give the test a clean profile to run against. Gaiatest contains protection against running and wiping your device's data . Please do read the warning before you set up your test variables file.

We use the --testvars option to pass in local variables, particularly those that cannot be checked into the repository. For example in gaia-ui-tests these variables can be your private login credentials, phone number, or details of your WiFi connection.

To use it, copy testvars_template.json to a different filename but add it into .gitignore so you don't check it into your repository.

When running your tests add the argument: --testvars=(filename).json

How to configure WiFi using testvars.json (for devices)

By setting the wifi key in testvars.json we can configure the Firefox OS Settings database with your WiFi configuration. Then the test cases can use your WiFi connection.

No WiFi authentication:

"wifi": { "ssid": "MyNetwork"} 

WEP authentication:

"wifi": { "ssid": "MyNetwork", "keyManagement": "WEP", "wep": "MyPassword" } 

WPA-PSK authentication:

"wifi": { "ssid": "MyNetwork", "keyManagement": "WPA-PSK", "psk": "MyPassword" } 

Due to Bug 775499, WiFi connections via WPA-EAP are not possible at this time.

 

Troubleshooting (common error messages)

Common problems before the test has started

Error Message Corresponding Actions
error: [Errno 111] Connection refused Reissue the "adb forward tcp:2828 tcp:2828" command
Element ... not visible before timeout make sure the element would display on the app you test
TimeoutException: Condition timed out make sure the condition on the app is the same as your expectations
broken pip mostly bad marionette server issue

 

Common problems and error messages after the test has started

 

Glossary

adb Marionette uses the Android Debug Bridge to communicate with Firefox OS using TCP requests
gaiatest The test runner package
   

 

Revision Source

<div class="warning">
 <p><em><strong>This page isn't finished yet.<br />
  Your feedback to gaia-ui-automation@mozilla.org would be greatly appreciated.</strong></em></p>
</div>
<h2 id="Running_Tests_-_developing_tests_and_framework">Running Tests - developing tests and framework</h2>
<p>&nbsp;</p>
<p>The gaia-ui-tests have been developed with the capability to run on a device, emulators, and desktop B2G. By far the easiest method is to run against desktopb2g.</p>
<p>However, please note that you should <u><strong>choose the right branch</strong></u> of gaia to run against the version of Firefox OS that you are using.</p>
<p>&nbsp;</p>
<h3 id="Testing_on_Desktopb2g_(Firefox_OS_desktop_client)">Testing on Desktopb2g (Firefox OS desktop client)</h3>
<div class="note">
 <p>Prequisites:</p>
 <ul>
  <li>
   <p>Desktopb2g downloaded from <a href="http://ftp.mozilla.org/pub/mozilla.org/b2g/nightly/latest-mozilla-central/">this location</a> and unpacked to a directory on your computer or virtual machine</p>
  </li>
  <li>
   <p>Gaia git repository cloned and checked out</p>
  </li>
  <li>
   <p><a href="#testvars">testvars.json file</a> created</p>
  </li>
 </ul>
</div>
<p>The desktopb2g / desktop client is ideal for developing tests where a device is unavailable and there is no need for phone-level functions. It is fast, accessible and will run on Linux, Mac and Windows!</p>
<p>We want to develop new tests and that will involve modifying gaiatest. We'll install the in-tree version of gaiatest directly from the gaia repository so that our changes are picked up immediately. In your cloned version of gaia, navigate to the gaia-ui-tests directory and run the setup command:</p>
<pre>
&gt; ~/Mozilla/gaia/tests/python/gaia-ui-tests$ python setup.py develop</pre>
<p>Via Marionette, gaiatest is able to launch desktopb2g when we start the test. To do this we need to pass in the path to the binary file and the path to the profile. In this example I have included the $HOME variable as the path to the b2g binary and the b2g profile. This will simply be the location that you unzipped desktopb2g to after you downloaded it. The command to run all the tests is:</p>
<pre>
&gt; ~/Mozilla/gaia/tests/python/gaia-ui-tests$ gaiatest --binary=$HOME/b2g/b2g-bin --profile=$HOME/b2g/gaia/profile --testvars=testvars_master.json --restart ./gaiatest/tests/functional/manifest.ini
</pre>
<p>Tests that are incompatible with the desktopb2g client, for example those that make calls and send SMS are automatically skipped from the test run.</p>
<p>You can run a single test file directly by calling the file directly on the command line or even several tests by calling them all on the command line</p>
<pre>
&gt; ~/Mozilla/gaia/tests/python/gaia-ui-tests$ gaiatest --binary=$HOME/b2g/b2g-bin --profile=$HOME/b2g/gaia/profile --testvars=testvars_master.json --restart ./gaiatest/tests/functional/clock/test_clock_set_alarm.py
</pre>
<p>&nbsp;</p>
<h3 id="Testing_on_Firefox_OS_Devices">Testing on Firefox OS Devices</h3>
<p>Testing on Firefox OS devices is by far the most complicated</p>
<div class="note">
 <p>Prerequisites:</p>
 <ul>
  <li>Android debug bridge (adb) installed and <a href="/en-US/Firefox_OS/Firefox_OS_build_prerequisites#For_Linux.3A_configure_the_udev_rule_for_your_phone">udev rules configured</a></li>
  <li>A Firefox OS device with a Marionette-enabled build</li>
  <li>Gaia Git repository checked out for the branch matching your Firefox OS device</li>
  <li>
   <p><a href="#testvars">testvars.json file</a> created</p>
  </li>
 </ul>
</div>
<p>The gaiatest package is needed to run the tests. We need to install this for the branch that we want to run the tests on. Navigate to the tests/python/gaia-ui-tests/ directory and install gaiatest with this command:</p>
<pre>
&gt; ~/Mozilla/gaia/tests/python/gaia-ui-tests$ python setup.py develop</pre>
<p>Before you start to run the test, read through <a href="https://developer.mozilla.org/en-US/docs/Gaia_Test_Runner#Risks" title="https://developer.mozilla.org/en-US/docs/Gaia_Test_Runner#Risks">this warning</a> in order to acknowledge that running gaia-ui-tests on a device may cause data to be deleted from the device!</p>
<p>Marionette on the device awaits commands on port 2828. We need to forward our local port to the remote port on the device using adb:</p>
<pre>
&gt; $ adb forward tcp:2828 tcp:2828</pre>
<p>Now, you can send this command to execute all gaia-ui-tests:</p>
<pre>
&gt; ~/Mozilla/gaia/tests/python/gaia-ui-tests$ gaiatest --address=localhost:2828 --testvars=&lt;TESTVARS_FILE&gt; --restart --type=b2g gaiatest/tests/functional/manifest.ini
</pre>
<p>Or, if you want to run solely one test file, just do:</p>
<pre>
&gt; ~/Mozilla/gaia/tests/python/gaia-ui-tests$ gaiatest --address=localhost:2828 --testvars=&lt;TESTVARS_FILE&gt; --restart gaiatest/functional/clock/test_clock_create_new_alarm.py
</pre>
<p>After the test, you will see all the pass/fail results and stacktraces for each file.</p>
<p>&nbsp;</p>
<h2 id="Running_tests_-_debugging_CI_failures">Running tests - debugging CI failures</h2>
<p id=".C2.A0">&nbsp;</p>
<h3 id="Running_tests_using_the_Travis_scripts_(for_Gecko_and_Gaia_developers)">Running tests using the Travis scripts (for Gecko and Gaia developers)</h3>
<p>&nbsp;</p>
<p>&nbsp;</p>
<h3 id="Testing_locally_with_TBPL_configuration_(for_Gecko_and_Gaia_developers)"><a name="RunningTBPL"></a>Testing locally with TBPL configuration (for Gecko and Gaia developers)</h3>
<div class="note">
 <p class="syntaxbox" id="Prerequisites.3A">Prerequisites:</p>
 <ul>
  <li>
   <p>A desktopb2g build including the Gecko patch you are testing, either self-built or downloaded from TBPL</p>
  </li>
  <li>
   <p>mozilla-b2g/gaia git repository checked out (for making the Gaia profile, installing gaiatest and the test files)</p>
  </li>
  <li>
   <p><a href="#testvars">testvars.json file</a> created</p>
  </li>
 </ul>
</div>
<p>In this documentation I will focus upon replicating a TBPL test run locally. Before testing locally with TBPL configuration you need to be aware that it builds a gaia profile separately. The profile that comes packaged with the TBPL build or rel-eng build is NOT the same Gaia profile that TBPL uses.</p>
<h4 id="Building_a_Gaia_profile_with_TBPL's.C2.A0_settings">Building a Gaia profile with TBPL's&nbsp; settings</h4>
<p>Before building a profile we need to check the environment variables that were used to build the profile. Searching in the "brief log" of the Gu test run, find this command block:</p>
<pre>
Using env: {'DEBUG': '0',
            'DESKTOP': '0',
            'DESKTOP_SHIMS': '1',
            'NOFTU': '0'}
</pre>
<p>These settings can drastically change the way desktopb2g behaves. It is very important to match TBPL's settings.</p>
<p>The next step is to checkout the git commit that matches the hg commit that TBPL ran with. You can also find the hg commit in the "brief log" of the TBPL run but you will need to correlate that with the git commit yourself.</p>
<p>The gaia test repository includes a <strong>make</strong> script for building a Gaia profile. Navigate to the gaia git repository that you have cloned locally. Build the profile using the environment variables we found on TBPL:</p>
<pre>
&gt; ~/Mozilla/gaia$ DEBUG=0 DESKTOP=0 DESKTOP_SHIMS=1 NOFTU=0 make</pre>
<h4 id="Running_the_tests">Running the tests</h4>
<p>Using the same git commit that TBPL used to run the test, install gaiatest from the gaia-ui-tests directory:</p>
<pre>
&gt; ~/Mozilla/gaia/tests/python/gaia-ui-tests$ python setup.py develop</pre>
<p>Now that we have built the Gaia profile, the only significant difference to running these tests is that we must direct gaiatest to the binary location and profile location. As TBPL runs an isolated environment it uses its own set of tests, set in tbpl-manifest.ini.</p>
<p>$BINARY_PATH - Path to the "b2g-bin" file in the directory you have built or unpacked desktopb2g to<br />
 $PROFILE_PATH - Path to your locally build Gaia profile. If built from gaia repository it will be ~/gaia/profile</p>
<p>Navigate to the gaia-ui-tests directory and run this command:</p>
<pre>
&gt; ~/Mozilla/gaia/tests/python/gaia-ui-tests$ gaiatest --binary $BINARY_PATH/b2g/b2g-bin --profile $PROFILE_PATH/profile --restart --testvars testvars.json gaiatest/tests/tbpl-manifest.ini --type=b2g</pre>
<p>There is no need to forward port 2828 unless you have set the marionette server to use a different default port.</p>
<p>Try to keep your mouse cursor clear of the b2g window so that pointer events do not pollute the test run.</p>
<p>&nbsp;</p>
<h3 id="Testing_against_an_already_running_desktopb2g_binary_(advanced_debugging_for_developers)">Testing against an already running desktopb2g binary (advanced debugging for developers)</h3>
<div class="note">
 <p class="syntaxbox" id="Prerequisites.3A">Prerequisites:</p>
 <ul>
  <li>
   <p>A B2G binary built with your Gecko changes</p>
  </li>
  <li>
   <p>Debugger of choice attached to the B2G binary</p>
  </li>
  <li>
   <p>mozilla-b2g/gaia git repository checked out (for installing gaiatest and the test files)</p>
  </li>
  <li>
   <p><a href="#testvars">testvars.json file</a> created</p>
  </li>
 </ul>
</div>
<p>Running against an existing b2g process is quite easy - you've already gone through the hard parts to build B2G!</p>
<p>If you have not already installed gaiatest then do so now by running `python setup.py develop` from the gaia-ui-tests directory.</p>
<pre>
&gt; ~/Mozilla/gaia/tests/python/gaia-ui-tests$ python setup.py develop</pre>
<p>Once you have started the b2g binary process and attached your debugger we direct gaiatest to the port using the <strong>--address</strong> command. Do not use --binary or --profile, gaiatest will just send commands directly to the port without attempting to start or close the binary.</p>
<pre>
&gt; ~/Mozilla/gaia/tests/python/gaia-ui-tests$ gaiatest --testvars testvars.json gaiatest/tests/functional/test_that_is_being_debugged.py</pre>
<p>Note that we have omitted the <strong>--restart</strong> command too. As gaiatest does not attempt to stop or start the binary some data from your test run may be left behind in Firefox OS databases or even in the DOM. It is your responsibility to 'reset' B2G back to a basic state before trying to run the test for a second time.</p>
<p>&nbsp;</p>
<h2 id="Test_Variables_(testvars.json)"><a name="testvars">Test Variables (testvars.json)</a></h2>
<p>The gaia-ui-tests will wipe the databases on your phone in order to give the test a clean profile to run against. Gaiatest contains protection against running and wiping your device's data . Please do read <a href="https://developer.mozilla.org/en-US/docs/Gaia_Test_Runner#Risks" title="https://developer.mozilla.org/en-US/docs/Gaia_Test_Runner#Risks">the warning</a> before you set up your test variables file.</p>
<p>We use the --testvars option to pass in local variables, particularly those that cannot be checked into the repository. For example in gaia-ui-tests these variables can be your private login credentials, phone number, or details of your WiFi connection.</p>
<p>To use it, copy testvars_template.json to a different filename but add it into .gitignore so you don't check it into your repository.</p>
<p>When running your tests add the argument: --testvars=(filename).json</p>
<h3 id="How_to_configure_WiFi_using_testvars.json_(for_devices)"><strong>How to configure WiFi using </strong>testvars.json (for devices)</h3>
<p>By setting the wifi key in testvars.json we can configure the Firefox OS Settings database with your WiFi configuration. Then the test cases can use your WiFi connection.</p>
<p>No WiFi authentication:</p>
<pre>
<code>"wifi": { "ssid": "MyNetwork"} </code></pre>
<p><code>WEP authentication:</code></p>
<pre>
<code>"wifi": { "ssid": "MyNetwork", "keyManagement": "WEP", "wep": "MyPassword" } </code></pre>
<p><code>WPA-PSK authentication:</code></p>
<pre>
<code>"wifi": { "ssid": "MyNetwork", "keyManagement": "WPA-PSK", "psk": "MyPassword" } </code></pre>
<div class="note">
 <p>Due to <a href="http://bugzil.la/775499">Bug 775499</a>, WiFi connections via WPA-EAP are not possible at this time.</p>
</div>
<p>&nbsp;</p>
<h2 id="Troubleshooting_(common_error_messages)">Troubleshooting (common error messages)</h2>
<p>Common problems before the test has started</p>
<table class="standard-table">
 <tbody>
  <tr>
   <td style="text-align: center;"><strong>Error Message </strong></td>
   <td style="text-align: center;"><strong>Corresponding Actions</strong></td>
  </tr>
  <tr>
   <td>error: [Errno 111] Connection refused</td>
   <td>Reissue the "adb forward tcp:2828 tcp:2828" command</td>
  </tr>
  <tr>
   <td>Element ... not visible before timeout</td>
   <td>make sure the element would display on the app you test</td>
  </tr>
  <tr>
   <td>TimeoutException: Condition timed out</td>
   <td>make sure the condition on the app is the same as your expectations</td>
  </tr>
  <tr>
   <td>broken pip</td>
   <td>mostly bad marionette server issue</td>
  </tr>
 </tbody>
</table>
<p>&nbsp;</p>
<p>Common problems and error messages after the test has started</p>
<p>&nbsp;</p>
<h2 id="Glossary">Glossary</h2>
<table class="standard-table">
 <tbody>
  <tr>
   <td>adb</td>
   <td>Marionette uses the Android Debug Bridge to communicate with Firefox OS using TCP requests</td>
  </tr>
  <tr>
   <td>gaiatest</td>
   <td>The test runner package</td>
  </tr>
  <tr>
   <td>&nbsp;</td>
   <td>&nbsp;</td>
  </tr>
 </tbody>
</table>
<p>&nbsp;</p>
Revert to this revision