mozilla

Revision 390805 of Gaia UI Tests: Writing Tests

  • Revision slug: Mozilla/Firefox_OS/Platform/Testing/gaia-ui-tests/Gaia_UI_Tests_Write_Tests
  • Revision title: Gaia UI Tests Write Tests
  • Revision id: 390805
  • Created:
  • Creator: ypwalter
  • Is current revision? No
  • Comment

Revision Content

This page hasn't finished yet.
Your feedback to gaia-ui-automation@mozilla.org would be greatly appreciated.

Writing Tests

Tests Guidelines

    Your contribution is a big step to us! You may reference this page. I will brief out something here since that page is having some similiar topics as this MDN document.

    You can find related programming guide for marionette. This shows the basic layout and how you would write a test on gaia-ui-tests. Also, you will find we strictly go accord to PEP8 style except the 80 characters, E501. However, we do want to keep it within 100 charaters or so. If you are using Ubuntu, you can install a basic check program for PEP8 by doing:

apt-get install pep8

    and, then you can check your program style by doing this command:

pep8 --ignore=E501 <your test file location and name>

    After you finished writing the test(s), you can send a pull request to the branch in Mozilla! Also, we sometime open issues in github of Mozilla/gaia-ui-tests to discuss in advance. In that case, we can understand if our contribution was finished by others before, if there are better suggestion, and if this contribution worth a lot!

Test Examples

# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.

from gaiatest import GaiaTestCase

class TestTemplate(GaiaTestCase):

    def setUp(self):
        # you should do something here to set up the environment
        GaiaTestCase.setUp(self)

    def test_template_1(self):
        # your first test regarding for this category of test
        ''' (description of where this test case is from, you may put a MozTrap link here)'''

    def test_template_2(self):
        # your second test regarding for this category of test
        ''' (description of where this test case is from, you may put a MozTrap link here)'''

    def tearDown(self):
        # you should do something here to clean the environment
        GaiaTestCase.tearDown(self)

Functions Reference

    At this point of time, we change codes dramatically everyday. We will try to come out a real gaia-ui-test methods overview soon. Before that comes into reality, you may take marionette methods page for reference. Furthermore, please do notice that marionette methods page has some out-of-date stuffs, too. Please kindly help us to correct them if possible.

    Another good reference is to dig into the code to know common libraries we are using. I suggest that you can start with gaia_test.py. Moreover, you can look into gaia_data_layer.js and gaia_apps.js. Also, all the files under apps/ folder are good-to-have materials.

TODO: Gaia-UI-Tests functions/methods/attributes reference

Debugging

    Everytime when you have no idea to start with in Gaia-UI-Tests, you can put python debugger(pdb) and run it:

# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.

from gaiatest import GaiaTestCase

class TestDebug(GaiaTestCase):

    def setUp(self):
        GaiaTestCase.setUp(self)

    def test_debug(self):
        import pdb; pdb.set_trace()

    def tearDown(self):
        GaiaTestCase.tearDown(self)

    You will find yourself in a console after running this. You can send single line code and it would return the result and control the device/emulator immediately. However, please do understand there are some variable restriction in this case. There are some reserved words in pdb that you can't use. Here is the reference of pdb from official Python website.

    When you tried to debug in your program, you may just put import pdb; pdb.set_trace() before the line it crashed to help you better debug it.

    Also, you can read the gaia(html) from B2G project to better helping you with realizing the code. But, here is another way to get the realtime code. You may send import re; import codecs; f = codecs.open("temp","w","utf8"); f.write(re.sub(r'<(script).*?</\1>(?s)', r'', self.marionette.page_source)) in pdb, and it should write the current code without localization stuffs to a file named "temp".

CSS and XPath Locators

     When we are trying to get locators of web elements, the most simple ways to do so is by linktext, id, or name. However, we do need to get webelement that sit in complex structure of HTML without linktext, id, or name. There are two main ways to find web elements in this case - CSS selector or XPath selector.

    You can find basic XPath reference in w3schools, CSS selectors reference in w3schools, or CSS selectors reference in W3C standard. I don't want to get into this too deep. You will find it easy to pick up knowledge from others' test scripts.
 

Note: CSS selector is preferrable over XPath selector, the reason is css selectors is usually faster. (external link for explanation)
You can also find this cheatsheet easy to use or keep you away from chaos.

 

Manifest files

    You will find manifest.ini files under all folders inside tests folder. It's the manifest files we are talking about. Inside the manifest files, we can find that it might indicate the required status when running tests under this folder, it might include test files needed, and it might include the other manifest files to be read.

Here is a list of the types used, and when to use them:

  • b2g - this means the test is a B2G (Firefox OS) test. All tests must include this type.
  • antenna - these tests require an antenna (headphones) to be connected.
  • bluetooth - requires bluetooth to be available.
  • camera - these tests require use of a camera.
  • carrier - an active SIM card with carrier connection is required.
  • lan - a local area connection (not cell data) is required by these tests (see note below).
  • offline - specifically requires no online connection.
  • online - some sort of online connection (lan or carrier) is required.
  • qemu - these tests require the Firefox OS emulator to run.
  • sdcard - a storage device must be present.
  • wifi - this means a WiFi connection is required.
  • xfail - a special type that indicates the test is expected to fail.

Note: Please do read the Mozilla/gaia-ui-tests github readme for most up-to-date types and help us to update in this page.

 

Revision Source

<div class="warning">
  <p><em><strong>This page hasn't finished yet.<br />
    Your feedback to gaia-ui-automation@mozilla.org would be greatly appreciated.</strong></em></p>
</div>
<h2 id="Writing_Tests">Writing Tests</h2>
<h3 id="Tests_Guidelines">Tests Guidelines</h3>
<p>&nbsp;&nbsp;&nbsp; Your contribution is a big step to us! You may reference <a href="https://github.com/mozilla/gaia-ui-tests/blob/master/CONTRIBUTING.md" title="https://github.com/mozilla/gaia-ui-tests/blob/master/CONTRIBUTING.md">this page</a>. I will brief out something here since that page is having some similiar topics as this MDN document.<br />
  <br />
  &nbsp; &nbsp; You can find related <a href="https://developer.mozilla.org/en-US/docs/Marionette/Marionette_Python_Tests" title="https://developer.mozilla.org/en-US/docs/Marionette/Marionette_Python_Tests">programming guide for marionette</a>. This shows the basic layout and how you would write a test on gaia-ui-tests. Also, you will find we strictly go accord to <a href="http://www.python.org/dev/peps/pep-0008/" title="http://www.python.org/dev/peps/pep-0008/">PEP8</a> style except the 80 characters, E501. However, we do want to keep it within 100 charaters or so. If you are using Ubuntu, you can install a basic check program for PEP8 by doing:</p>
<pre>
apt-get install pep8
</pre>
<p>&nbsp;&nbsp;&nbsp; and, then you can check your program style by doing this command:</p>
<pre>
pep8 --ignore=E501 &lt;your test file location and name&gt;</pre>
<p>&nbsp;&nbsp;&nbsp; After you finished writing the test(s), you can send a pull request to the branch in Mozilla! Also, we sometime open issues in github of <a href="https://github.com/mozilla/gaia-ui-tests/issues?state=open" title="https://github.com/mozilla/gaia-ui-tests/issues?state=open">Mozilla/gaia-ui-tests</a> to discuss in advance. In that case, we can understand if our contribution was finished by others before, if there are better suggestion, and if this contribution worth a lot!</p>
<h3 id="Test_Examples">Test Examples</h3>
<pre class="brush: python">
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.

from gaiatest import GaiaTestCase

class TestTemplate(GaiaTestCase):

&nbsp;&nbsp;&nbsp; def setUp(self):
        # you should do something here to set up the environment
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; GaiaTestCase.setUp(self)

&nbsp;&nbsp;&nbsp; def test_template_1(self):
        # your first test regarding for this category of test
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ''' (description of where this test case is from, you may put a MozTrap link here)'''

&nbsp;&nbsp;&nbsp; def test_template_2(self):
        # your second test regarding for this category of test
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ''' (description of where this test case is from, you may put a MozTrap link here)'''

&nbsp;&nbsp;&nbsp; def tearDown(self):
        # you should do something here to clean the environment
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; GaiaTestCase.tearDown(self)

</pre>
<h3 id="Functions_Reference">Functions Reference</h3>
<p>&nbsp;&nbsp;&nbsp; At this point of time, we change codes dramatically everyday. We will try to come out a real gaia-ui-test methods overview soon. Before that comes into reality, you may take <a href="https://developer.mozilla.org/en-US/docs/Marionette/Marionette" title="https://developer.mozilla.org/en-US/docs/Marionette/Marionette">marionette methods page</a> for reference. Furthermore, please do notice that marionette methods page has some out-of-date stuffs, too. Please kindly help us to correct them if possible.<br />
  <br />
  &nbsp;&nbsp;&nbsp; Another good reference is to dig into the code to know common libraries we are using. I suggest that you can start with <strong>gaia_test.py</strong>. Moreover, you can look into <strong>gaia_data_layer.js</strong> and <strong>gaia_apps.js</strong>. Also, all the files under <strong>apps/</strong> folder are good-to-have materials.</p>
<div class="geckoVersionNote">
  <p>TODO: Gaia-UI-Tests functions/methods/attributes reference</p>
</div>
<h3 id="Debugging">Debugging</h3>
<p>&nbsp;&nbsp;&nbsp; Everytime when you have no idea to start with in Gaia-UI-Tests, you can put python debugger(pdb) and run it:</p>
<pre class="brush: python">
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.

from gaiatest import GaiaTestCase

class TestDebug(GaiaTestCase):

&nbsp;&nbsp;&nbsp; def setUp(self):
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; GaiaTestCase.setUp(self)

&nbsp;&nbsp;&nbsp; def test_debug(self):
        import pdb; pdb.set_trace()

&nbsp;&nbsp;&nbsp; def tearDown(self):
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; GaiaTestCase.tearDown(self)
</pre>
<p>&nbsp;&nbsp;&nbsp; You will find yourself in a console after running this. You can send single line code and it would return the result and control the device/emulator immediately. However, please do understand there are some variable restriction in this case. There are some reserved words in pdb that you can't use. Here is the <a href="http://docs.python.org/2/library/pdb.html" title="http://docs.python.org/2/library/pdb.html">reference of pdb</a> from official Python website.<br />
  <br />
  &nbsp;&nbsp;&nbsp; When you tried to debug in your program, you may just put <strong>import pdb; pdb.set_trace()</strong> before the line it crashed to help you better debug it.<br />
  <br />
  &nbsp;&nbsp;&nbsp; Also, you can read the gaia(html) from B2G project to better helping you with realizing the code. But, here is another way to get the realtime code. You may send <strong>import re; import codecs; f = codecs.open("temp","w","utf8"); f.write(re.sub(r'&lt;(script).*?&lt;/\1&gt;(?s)', r'', self.marionette.page_source))</strong> in pdb, and it should write the current code without localization stuffs to a file named "temp".</p>
<h3 id="CSS_and_XPath_Locators">CSS and XPath Locators</h3>
<p>&nbsp;&nbsp;&nbsp;&nbsp; When we are trying to get locators of web elements, the most simple ways to do so is by linktext, id, or name. However, we do need to get webelement that sit in complex structure of HTML without linktext, id, or name. There are two main ways to find web elements in this case - CSS selector or XPath selector.<br />
  <br />
  &nbsp;&nbsp;&nbsp; You can find basic <a href="http://www.w3schools.com/xpath/xpath_syntax.asp" title="http://www.w3schools.com/xpath/xpath_syntax.asp">XPath reference in w3schools,</a> <a href="http://www.w3schools.com/cssref/css_selectors.asp" title="http://www.w3schools.com/cssref/css_selectors.asp">CSS selectors reference in w3schools</a>, or <a href="http://www.w3.org/TR/CSS2/selector.html" title="http://www.w3.org/TR/CSS2/selector.html">CSS selectors reference in W3C standard</a>. I don't want to get into this too deep. You will find it easy to pick up knowledge from others' test scripts.<br />
  &nbsp;</p>
<div class="note">
  <p>Note: CSS selector is preferrable over XPath selector, the reason is css selectors is usually faster. (<a href="http://sauceio.com/index.php/2011/05/why-css-locators-are-the-way-to-go-vs-xpath/" title="http://sauceio.com/index.php/2011/05/why-css-locators-are-the-way-to-go-vs-xpath/">external link for explanation</a>)<br />
    You can also find <a href="https://www.simple-talk.com/dotnet/.net-framework/xpath,-css,-dom-and-selenium-the-rosetta-stone/" title="https://www.simple-talk.com/dotnet/.net-framework/xpath,-css,-dom-and-selenium-the-rosetta-stone/">this cheatsheet</a> easy to use or keep you away from chaos.</p>
</div>
<p style="text-align: center;">&nbsp;</p>
<h2 id="Manifest_files">Manifest files</h2>
<p>&nbsp;&nbsp;&nbsp; You will find <strong>manifest.ini</strong> files under all folders inside tests folder. It's the manifest files we are talking about. Inside the manifest files, we can find that it might indicate the required status when running tests under this folder, it might include test files needed, and it might include the other manifest files to be read.<br />
  <br />
  Here is a list of the types used, and when to use them:</p>
<ul>
  <li>b2g - this means the test is a B2G (Firefox OS) test. All tests must include this type.</li>
  <li>antenna - these tests require an antenna (headphones) to be connected.</li>
  <li>bluetooth - requires bluetooth to be available.</li>
  <li>camera - these tests require use of a camera.</li>
  <li>carrier - an active SIM card with carrier connection is required.</li>
  <li>lan - a local area connection (not cell data) is required by these tests (see note below).</li>
  <li>offline - specifically requires no online connection.</li>
  <li>online - some sort of online connection (lan or carrier) is required.</li>
  <li>qemu - these tests require the Firefox OS emulator to run.</li>
  <li>sdcard - a storage device must be present.</li>
  <li>wifi - this means a WiFi connection is required.</li>
  <li>xfail - a special type that indicates the test is expected to fail.</li>
</ul>
<div class="note">
  <p>Note: Please do read the <a href="https://github.com/mozilla/gaia-ui-tests" title="https://github.com/mozilla/gaia-ui-tests">Mozilla/gaia-ui-tests</a> github readme for most up-to-date types and help us to update in this page.</p>
</div>
<p>&nbsp;</p>
Revert to this revision