mozilla

Revision 538711 of MTBF tests

  • Revision slug: Mozilla/Firefox_OS/Platform/Automated_testing/MTBF_tests
  • Revision title: MTBF tests
  • Revision id: 538711
  • Created:
  • Creator: ypwalter
  • Is current revision? No
  • Comment First Commit

Revision Content

The MTBF tests are a suite of Firefox OS tests built on top of the Gaiatest (Gaia UI Tests) Framework. The tests run on real Firefox OS devices, and use Marionette to drive the device's UI.

Mean time between failures (MTBF) is the predicted elapsed time between inherent failures of a system during operation. MTBF can be calculated as the arithmetic mean (average) time between failures of a system. The MTBF is typically part of a model that assumes the failed system is immediately repaired (mean time to repair, or MTTR), as a part of a renewal process. This is in contrast to the mean time to failure (MTTF), which measures average time to failures with the modeling assumption that the failed system is not repaired (infinite repair time).

How often are the tests run?

MTBF will not run until we get a all pass in smoke test and the code should be in the aurora (next release branch) before we start to run it.

Where can I see the results?

Currently, only intranet can see some of the trial runs in MTBF clusters in MTBF Jenkins CI server.
http://mtbf-1.corp.tpe1.mozilla.com:8080

Running the tests

Let's go through the steps required to set up the Gaia-UI Endurance test environment and run the tests on your local machine and Firefox OS device.

Prerequisites

  • Ubuntu 12.04 (or better) x64, or Mac OS X (these instructions are confirmed for 10.8 but should work on previous versions of 10, theoretically.)
  • A Firefox OS device ALREADY FLASHED with an ENGINEERING build of Firefox OS B2G-18 V1-Train (1.1.)
  • ADB installed, and the environment variable ADB_PATH pointing to your ADB location.

If you are on Ubuntu, you need to check that it is configured to support the USB connection to the Firefox OS device. To verify this, connect the device to your computer via USB, open a terminal and enter the adb logcat command to see if it connects.

Note: To start with, the Firefox OS device should not be connected to your computer. You will be told when to connect it in the steps below.

Step 1: Clone the Gaia repository from Mozilla-TWQA

The Gaia-UI Endurance Tests are located in the Mozilla Github Gaia repository. Assuming that you haven’t done so already, the first step is to clone that repo:

git clone https://github.com/mozilla-b2g/gaia.git

You may want to go grab a coffee and come back in five minutes. Furthermore, you can get all the branch and try to switch to current MTBF branch like mtbf-v1.3-20140218 or mtbf-v1.4-trial.

Step 2: Run the GaiaTest setup

The Gaia-UI Endurance tests are built upon the GaiaTest framework (which uses Marionette). The next step is to run the setup script to install GaiaTest and all of the required dependencies. You may wish to create a new virtual environment to use with the Gaia-UI Endurance Tests. If you don’t, you may need to use sudo while running the setup command. In your terminal, type:

cd gaia/tests/python/gaia-ui-tests
python setup.py develop

Step 3: Get MTBF Driver

git clone https://github.com/Mozilla-TWQA/MTBF-Driver.git

After you get this you can now procede to next step and we will explain this further later.

Step 4: Set test vars and acknowledge risks

GaiaTest uses a special file to set certain variables that are required for test configuration. For example, to tell the device which WiFi network it should use. Before running the Gaia-UI Endurance Tests, you must setup the test vars file. Make a copy of the gaia/tests/python/gaia-ui-tests/gaiatest/testvars_template.json file in the same location (call it what you like) and edit it:

  • If the Firefox OS device has a SIM card, add the corresponding phone number in the phone number entry in the carrier section, e.g. "phone_number": "5551234567".
  • Add the same phone number for the "remote_phone_number" entry.
  • In the wifi section, add the SSID for the Wifi network that the Firefox OS device is to connect to during the tests in the ssid line, for example "ssid": "Wifi Network Name".

As mentioned in the warning above, running the Gaia-UI Endurance tests will result in data being erased from the Firefox OS device and microSD card. This is to ensure that the tests start cleanly each time. For example, running a test on a device that already has 10,000 contacts will have very different memory value results vs running the same test on a device with no existing contacts. In order to run the tests, you must acknowledge that you are aware of this data loss risk.

To acknowledge the risks, add the following entry to your testvars file as the first entry in the list: "acknowledged_risks": true.

Note: If the risks are not acknowledged in the testvars file, the tests will not run.

Step 5: Connect to USB and ADB Forward the Device

Attach the Firefox OS device to your computer via USB.

Note: If you’re using an Ubuntu VM, after attaching the device ensure the VM sees the device and connects to it; in the VM select VM > Removable Devices > Your Device > Connect and wait for the device to connect to the VM.

Now tell adb to forward the device port to GaiaTest using the following command:

adb forward tcp:2828 tcp:2828

Note: If you are using the Firefox OS Leo device, you must first tell ADB to be the root user, like so:

adb root
adb forward tcp:2828 tcp:2828

Step 6: Run a Test

Now you’re ready to actually try running a test. To run the add_contact endurance test, with a single iteration, use the following commands:

cd {MTBF Driver Folder}
MTBF_TIME=1d python MTBF.py --address=localhost:2828 --testvars=mytestvars.json tests/MTBF/{any folder or test}

We can parse the MTBF_TIME by d(ay), h(our), or m(inute).

If you get a “connection refused” error it means the device USB connection didn’t work; just repeat the device connection and try again.

The Firefox OS device b2g process should now restart, then the add_contact endurance test will run with a single iteration. If you watch the Firefox OS device, you’ll see the device UI being manipulated by Marionette. After the test finishes, a memory checkpoint will be performed.

Note: The Gaia-UI Endurance tests now grab the Firefox OS device’s b2g process RSS value for the memory use checkpoint (it used to be the V-SIZE value.)

The test result will be displayed in the terminal window. Note that this result doesn’t include the b2g process memory value; this value is stored in a text file that was created at the time of the checkpoint in the checkpoints directory. To see the resulting b2g process, open this file. This "suite_summary" file will contain the average b2g process memory use (RSS) value, averaged from all of the test checkpoints (in our example there was only one checkpoint anyway).

There are two other files present in the checkpoints folder:

  • The checkpoint_add_contact_(datetime)_summary.log file contains a summary for the test run. This includes the number of iterations, all of the RSS value readings from each checkpoint, the average RSS value, etc.
  • The checkpoint_add_contact_(datetime).log file contains the raw data received from each of the device checkpoints, from which the summary files were produced.

Contributing to the project

If you have any questions about the Firefox OS Endurance tests or are interested in contributing to this important automation development effort, feel free to contact us at wachen@mozilla.com

How to migrate test cases from Gaia-ui-tests

<1> Rename

  • from MtbfTestCase import GaiaMtbfTestCase

  • (original) from gaiatest import GaiaTestCase

  • class XXX(GaiaMtbfTestCase):

  • (original) class XXX(GaiaTestCase)

  • if it has setUp or tearDown method-> GaiaMtbfTestCase (original: GaiaTestCase)


<2> Add

  • if there's no setUp method , add it before the general test function

  • please add GaiaMtbfTestCase.setUp(self) as the 1st step of setUp

    • Environment check/recovery and app initialization

    • launch -> launch_by_touch if possible

    • get the handle by self.app_id

  • if there's no tearDown method, add it after the general test function

  • please add GaiaMtbfTestCase.tearDown(self) as the last step of tearDown

    • handle data after finished (delete/remove/move…)


<3> Principles

  • modify assert()s in test (make sure it works for multiple test cases)

  • if porting cases from Gaia-UI-Test, you need to declare app as a test class member

  • multiple apps init/launch in a case, please refer to the card_view cases

  • unless your clean actions don't involve UI interactions, but please avoid it if you can

  • for example, use device manager to remove the file on the device

  • in those cases, you may need to call marionette.refresh() to get the latest update

  • replace datetime.fromtimestamp to datetime.utcfromtimestamp if you wanna implement calendar related cases


<4> About apps

apps -> mtbf_apps if needed

    a. import original apps

    b. add __init__() and functions you need


<5> After you finished

1. Test each test case -> Test full suite -> Test full suite with shuffle

2. check PEP8 errors

3. pull request! don’t not push directly

**Fix issues in a more general way

 

Revision Source

<p>The <strong>MTBF tests</strong> are a suite of Firefox OS tests built on top of the <a href="https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Platform/Automated_testing/gaia-ui-tests" title="https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Platform/Automated_testing/gaia-ui-tests">Gaiatest (Gaia UI Tests) </a>Framework. The tests run on real Firefox OS devices, and use <a href="/en-US/docs/Marionette" title="/en-US/docs/Marionette">Marionette</a> to drive the device's UI.</p>
<p><strong>Mean time between failures (MTBF)</strong> is the predicted elapsed time between inherent failures of a system during operation. MTBF can be calculated as the arithmetic mean (average) time between failures of a system. The MTBF is typically part of a model that assumes the failed system is immediately repaired (mean time to repair, or MTTR), as a part of a renewal process. This is in contrast to the mean time to failure (MTTF), which measures average time to failures with the modeling assumption that the failed system is not repaired (infinite repair time).</p>
<h2 id="How_often_are_the_tests_run.3F">How often are the tests run?</h2>
<p>MTBF will not run until we get a all pass in smoke test and the code should be in the aurora (next release branch) before we start to run it.</p>
<h2 id="Where_can_I_see_the_results.3F">Where can I see the results?</h2>
<p>Currently, only intranet can see some of the trial runs in MTBF clusters in MTBF Jenkins CI server.<br />
 <a href="http://mtbf-1.corp.tpe1.mozilla.com:8080">http://mtbf-1.corp.tpe1.mozilla.com:8080</a></p>
<h2 id="Running_the_tests">Running the tests</h2>
<p>Let's go through the steps required to set up the Gaia-UI Endurance test environment and run the tests on your local machine and Firefox OS device.</p>
<h3 id="Prerequisites">Prerequisites</h3>
<ul>
 <li>Ubuntu 12.04 (or better) x64, or Mac OS X (these instructions are confirmed for 10.8 but should work on previous versions of 10, theoretically.)</li>
 <li>A Firefox OS device ALREADY FLASHED with an ENGINEERING build of Firefox OS B2G-18 V1-Train (1.1.)</li>
 <li><a href="/en-US/Firefox_OS/Debugging/Installing_ADB">ADB installed</a>, and the environment variable <code>ADB_PATH</code> pointing to your ADB location.</li>
</ul>
<p>If you are on Ubuntu, you need to check that it is configured to support the USB connection to the Firefox OS device. To verify this, connect the device to your computer via USB, open a terminal and enter the <code>adb logcat</code> command to see if it connects.</p>
<div class="note">
 <p><strong>Note</strong>: To start with, the Firefox OS device should not be connected to your computer. You will be told when to connect it in the steps below.</p>
</div>
<h3 id="Step_1.3A_Clone_the_Gaia_repository">Step 1: Clone the Gaia repository from Mozilla-TWQA</h3>
<p>The Gaia-UI Endurance Tests are located in the Mozilla Github Gaia repository. Assuming that you haven’t done so already, the first step is to clone that repo:</p>
<pre>
git clone <a href="https://github.com/mozilla-b2g/gaia.git" rel="nofollow">https://github.com/mozilla-b2g/gaia.git</a></pre>
<p>You may want to go grab a coffee and come back in five minutes. Furthermore, you can get all the branch and try to switch to current MTBF branch like mtbf-v1.3-20140218 or mtbf-v1.4-trial.</p>
<h3 id="Step_2.3A_Run_the_GaiaTest_setup">Step 2: Run the GaiaTest setup</h3>
<p>The Gaia-UI Endurance tests are built upon the GaiaTest framework (which uses <a href="https://developer.mozilla.org/en-US/docs/Marionette" target="_blank" title="Marionette">Marionette</a>). The next step is to run the setup script to install GaiaTest and all of the required dependencies. You may wish to create a new virtual environment to use with the Gaia-UI Endurance Tests. If you don’t, you may need to use <code>sudo</code> while running the setup command. In your terminal, type:</p>
<pre style="padding-left: 30px;">
cd gaia/tests/python/gaia-ui-tests
python setup.py develop</pre>
<h3 id="Step_5.3A_Connect_to_USB_and_ADB_Forward_the_Device">Step 3: Get MTBF Driver</h3>
<pre style="padding-left: 30px;">
git clone <a href="https://github.com/Mozilla-TWQA/MTBF-Driver.git">https://github.com/Mozilla-TWQA/MTBF-Driver.git</a></pre>
<p>After you get this you can now procede to next step and we will explain this further later.</p>
<h3 id="Step_3.3A_Set_test_vars_and_acknowledge_risks">Step 4: Set test vars and acknowledge risks</h3>
<p>GaiaTest uses a special file to set certain variables that are required for test configuration. For example, to tell the device which WiFi network it should use. Before running the Gaia-UI Endurance Tests, you must setup the test vars file. Make a copy of the <code>gaia/tests/python/gaia-ui-tests/gaiatest/testvars_template.json</code> file in the same location (call it what you like) and edit it:</p>
<ul>
 <li>If the Firefox OS device has a SIM card, add the corresponding phone number in the phone number entry in the <code>carrier</code> section, e.g. <code>"phone_number": "5551234567"</code>.</li>
 <li>Add the same phone number for the <code>"remote_phone_number"</code> entry.</li>
 <li>In the <code>wifi</code> section, add the SSID for the Wifi network that the Firefox OS device is to connect to during the tests in the <code>ssid</code> line, for example <code>"ssid": "Wifi Network Name"</code>.</li>
</ul>
<p>As mentioned in the warning above, running the Gaia-UI Endurance tests will result in data being erased from the Firefox OS device and microSD card. This is to ensure that the tests start cleanly each time. For example, running a test on a device that already has 10,000 contacts will have very different memory value results vs running the same test on a device with no existing contacts. In order to run the tests, you must acknowledge that you are aware of this data loss risk.</p>
<p>To acknowledge the risks, add the following entry to your <code>testvars</code> file as the first entry in the list: <code>"acknowledged_risks": true</code>.</p>
<div class="note">
 <p><strong>Note</strong>: If the risks are not acknowledged in the <code>testvars</code> file, the tests will not run.</p>
</div>
<h3 id="Step_5.3A_Connect_to_USB_and_ADB_Forward_the_Device">Step 5: Connect to USB and ADB Forward the Device</h3>
<p>Attach the Firefox OS device to your computer via USB.</p>
<div class="note">
 <p><strong>Note</strong>: If you’re using an Ubuntu VM, after attaching the device ensure the VM sees the device and connects to it; in the VM select <em>VM &gt; Removable Devices &gt; Your Device &gt; Connect</em> and wait for the device to connect to the VM.</p>
</div>
<p>Now tell <code>adb</code> to forward the device port to GaiaTest using the following command:</p>
<pre class="brush: bash" style="padding-left: 30px;">
adb forward tcp:2828 tcp:2828</pre>
<div class="note">
 <p><strong>Note</strong>: If you are using the Firefox OS Leo device, you must first tell ADB to be the root user, like so:</p>
 <pre style="padding-left: 30px;">
adb root
adb forward tcp:2828 tcp:2828</pre>
</div>
<h3 id="Step_6.3A_Run_a_Test">Step 6: Run a Test</h3>
<p>Now you’re ready to actually try running a test. To run the <code>add_contact</code> endurance test, with a single iteration, use the following commands:</p>
<pre class="brush: bash" style="padding-left: 30px;">
cd {MTBF Driver Folder}
MTBF_TIME=1d python MTBF.py --address=localhost:2828 --testvars=mytestvars.json tests/MTBF/{any folder or test}</pre>
<p>We can parse the MTBF_TIME by d(ay), h(our), or m(inute).</p>
<p>If you get a “connection refused” error it means the device USB connection didn’t work; just repeat the device connection and try again.</p>
<p>The Firefox OS device b2g process should now restart, then the <code>add_contact</code> endurance test will run with a single iteration. If you watch the Firefox OS device, you’ll see the device UI being manipulated by Marionette. After the test finishes, a memory checkpoint will be performed.</p>
<div class="note">
 <p><strong>Note</strong>: The Gaia-UI Endurance tests now grab the Firefox OS device’s b2g process RSS value for the memory use checkpoint (it used to be the V-SIZE value.)</p>
</div>
<p>The test result will be displayed in the terminal window. Note that this result doesn’t include the b2g process memory value; this value is stored in a text file that was created at the time of the checkpoint in the <code>checkpoints</code> directory. To see the resulting b2g process, open this file. This "suite_summary" file will contain the average b2g process memory use (RSS) value, averaged from all of the test checkpoints (in our example there was only one checkpoint anyway).</p>
<p>There are two other files present in the <code>checkpoints</code> folder:</p>
<ul>
 <li>The <code>checkpoint_add_contact_(datetime)_summary.log</code> file contains a summary for the test run. This includes the number of iterations, all of the RSS value readings from each checkpoint, the average RSS value, etc.</li>
 <li>The <code>checkpoint_add_contact_(datetime).log</code> file contains the raw data received from each of the device checkpoints, from which the summary files were produced.</li>
</ul>
<h2 id="Contributing_to_the_project">Contributing to the project</h2>
<p>If you have any questions about the Firefox OS Endurance tests or are interested in contributing to this important automation development effort, feel free to contact us at wachen@mozilla.com</p>
<h2 id="Contributing_to_the_project">How to migrate test cases from Gaia-ui-tests</h2>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">&lt;1&gt; Rename</span></p>
<ul style="margin-top:0pt;margin-bottom:0pt;">
 <li dir="ltr" style="list-style-type:disc;font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">
  <p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">from MtbfTestCase import GaiaMtbfTestCase</span></p>
 </li>
</ul>
<ul style="margin-top:0pt;margin-bottom:0pt;">
 <li dir="ltr" style="list-style-type:disc;font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">
  <p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">(original) from gaiatest import GaiaTestCase</span></p>
 </li>
</ul>
<ul style="margin-top:0pt;margin-bottom:0pt;">
 <li dir="ltr" style="list-style-type:disc;font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">
  <p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">class XXX(GaiaMtbfTestCase):</span></p>
 </li>
</ul>
<ul style="margin-top:0pt;margin-bottom:0pt;">
 <li dir="ltr" style="list-style-type:disc;font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">
  <p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">(original) class XXX(GaiaTestCase)</span></p>
 </li>
</ul>
<ul style="margin-top:0pt;margin-bottom:0pt;">
 <li dir="ltr" style="list-style-type:disc;font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">
  <p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">if it has setUp or tearDown method-&gt; GaiaMtbfTestCase (original: GaiaTestCase)</span></p>
 </li>
</ul>
<p><br />
 <p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">&lt;2&gt; Add</span></p>
</p>
<ul style="margin-top:0pt;margin-bottom:0pt;">
 <li dir="ltr" style="list-style-type:disc;font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">
  <p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">if there's no setUp method , add it before the general test function</span></p>
 </li>
 <li dir="ltr" style="list-style-type:disc;font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">
  <p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">please add GaiaMtbfTestCase.setUp(self) as the 1st step of setUp</span></p>
  <ul style="margin-top:0pt;margin-bottom:0pt;">
   <li dir="ltr" style="list-style-type:circle;font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">
    <p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">Environment check/recovery and app initialization</span></p>
   </li>
   <li dir="ltr" style="list-style-type:circle;font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">
    <p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">launch -&gt; launch_by_touch if possible</span></p>
   </li>
   <li dir="ltr" style="list-style-type:circle;font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">
    <p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">get the handle by self.app_id</span></p>
   </li>
  </ul>
 </li>
</ul>
<ul style="margin-top:0pt;margin-bottom:0pt;">
 <li dir="ltr" style="list-style-type:disc;font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">
  <p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">if there's no tearDown method, add it after the general test function</span></p>
 </li>
</ul>
<ul style="margin-top:0pt;margin-bottom:0pt;">
 <li dir="ltr" style="list-style-type:disc;font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">
  <p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">please add GaiaMtbfTestCase.tearDown(self) as the last step of tearDown</span></p>
  <ul style="margin-top:0pt;margin-bottom:0pt;">
   <li dir="ltr" style="list-style-type:circle;font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">
    <p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">handle data after finished (delete/remove/move…)</span></p>
   </li>
  </ul>
 </li>
</ul>
<p><br />
 <p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">&lt;3&gt; Principles</span></p>
</p>
<ul style="margin-top:0pt;margin-bottom:0pt;">
 <li dir="ltr" style="list-style-type:disc;font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">
  <p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">modify assert()s in test (make sure it works for multiple test cases)</span></p>
 </li>
</ul>
<ul style="margin-top:0pt;margin-bottom:0pt;">
 <li dir="ltr" style="list-style-type:disc;font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">
  <p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">if porting cases from Gaia-UI-Test, you need to declare app as a test class member</span></p>
 </li>
</ul>
<ul style="margin-top:0pt;margin-bottom:0pt;">
 <li dir="ltr" style="list-style-type:disc;font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">
  <p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">multiple apps init/launch in a case, please refer to the card_view cases</span></p>
 </li>
</ul>
<ul style="margin-top:0pt;margin-bottom:0pt;">
 <li dir="ltr" style="list-style-type:disc;font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">
  <p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">unless your clean actions don't involve UI interactions, but please avoid it if you can</span></p>
 </li>
</ul>
<ul style="margin-top:0pt;margin-bottom:0pt;">
 <li dir="ltr" style="list-style-type:disc;font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">
  <p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">for example, use device manager to remove the file on the device</span></p>
 </li>
</ul>
<ul style="margin-top:0pt;margin-bottom:0pt;">
 <li dir="ltr" style="list-style-type:disc;font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">
  <p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">in those cases, you may need to call marionette.refresh() to get the latest update</span></p>
 </li>
</ul>
<ul style="margin-top:0pt;margin-bottom:0pt;">
 <li dir="ltr" style="list-style-type:disc;font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">
  <p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">replace datetime.fromtimestamp to datetime.utcfromtimestamp if you wanna implement calendar related cases</span></p>
 </li>
</ul>
<p><br />
 <p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">&lt;4&gt; About apps</span></p>
</p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-indent: 36pt;"><span style="font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">apps -&gt; mtbf_apps if needed</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">&nbsp;&nbsp;&nbsp; a. import original apps</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">&nbsp;&nbsp;&nbsp; b. add __init__() and functions you need</span></p>
<p><br />
 <p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:bold;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">&lt;5&gt; After you finished</span></p>
</p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-indent: 36pt;"><span style="font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">1. Test each test case -&gt; Test full suite -&gt; Test full suite with shuffle</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-indent: 36pt;"><span style="font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">2. check PEP8 errors</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-indent: 36pt;"><span style="font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">3. pull request! don’t not push directly</span></p>
<p><span style="font-size:15px;font-family:Arial;color:#000000;background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;">**Fix issues in a more general way</span></p>
<p>&nbsp;</p>
Revert to this revision