Thunderbird MozMill Testing

  • Revision slug: Thunderbird/Thunderbird_MozMill_Testing
  • Revision title: Thunderbird MozMill Testing
  • Revision id: 455527
  • Created:
  • Creator: jmozmoz
  • Is current revision? No
  • Comment

Revision Content

MozMill is the framework that we're using to run interactive tests.  Mozmill tests effectively run the entire application through unit tests.

Installing MozMill

{{ obsolete_header("9") }}

Starting Thunderbird 9, MozMill no longer needs to be installed separately. It is included in the sources and will be installed into a virtual environment during the build process. So you can proceed with Running Mozmill Tests.

Thunderbird's MozMill test harness currently requires using MozMill 1.4.2b1; more recent versions will not work at this time.  If you need to use a more recent version of mozmill for other software development, you will want to look into virtualenv.

Windows

You'll need MozillaBuild 1.4 or higher for this. Start a MozillaBuild prompt and use it to install MozMill 1.4.2b1:

 easy_install http://pypi.rhaptos.org/mozrunner/mozrunner-2.5.2.tar.gz
 easy_install http://pypi.rhaptos.org/jsbridge/jsbridge-2.3.7b1.tar.gz
 easy_install http://pypi.rhaptos.org/mozmill/mozmill-1.4.2b1.tar.gz

Mac/Linux

You'll need root privileges for this.

 sudo easy_install http://pypi.rhaptos.org/mozrunner/mozrunner-2.5.2.tar.gz
 sudo easy_install http://pypi.rhaptos.org/jsbridge/jsbridge-2.3.7b1.tar.gz
 sudo easy_install http://pypi.rhaptos.org/mozmill/mozmill-1.4.2b1.tar.gz

If you, for some reason, need a different version, see MozMill Development Install.

Running Mozmill Tests

{{ Note("Although running a Thunderbird instance and a MozMill Thunderbird instance alongside each other in the same user login is possible, MozMill tends not to detect shutdown of the MozMill Instance correctly. If your test isn't finishing (even though Thunderbird has shut down) try closing any other instances of Thunderbird running in the user login.") }}

Running a test suite:

Run this from objdir/ to run all the tests in the test suite.

make mozmill

For builds on Windows (created by {{ tbsource("mozilla/mach", "mozilla/mach") }} and implicitly {{ tbsource("build/pymake/make.py", "build/pymake/make.py") }}):

../build/pymake/make.py mozmill

Running a specific test:

For example to run all the {{ tbsource("mail/test/mozmill/folder-display/", "folder-display") }} tests, go to the objdir and run:

make SOLO_TEST=folder-display mozmill-one

For builds on Windows (created by {{ tbsource("mozilla/mach", "mozilla/mach") }} and implicitly {{ tbsource("build/pymake/make.py", "build/pymake/make.py") }}):

../build/pymake/make.py SOLO_TEST=folder-display mozmill-one

SOLO_TEST may be a directory containing tests within {{ tbsource("mail/test/mozmill/", "mail/test/mozmill") }}, or a specific test file.

To run just a single test ("all the tests in a single file"), like test-message-window.js, again go to the objdir and run:

make SOLO_TEST=folder-display/test-message-window.js mozmill-one 

For builds on Windows (created by {{ tbsource("mozilla/mach", "mozilla/mach") }} and implicitly {{ tbsource("build/pymake/make.py", "build/pymake/make.py") }}):

../build/pymake/make.py SOLO_TEST=folder-display/test-message-window.js mozmill-one

Running MozMill Tests with pre-compiled Thunderbird ('packaged tests')

If you don't have a self-compiled build of Thunderbird, then you can run tests against pre-compiled Thunderbird builds. This is known as running in the 'packaged tests' style.

See this page for how to run in package test style.

Linux & VNC Server

Linux runs have a lot of issues with focus related problems in more recents of gecko.  Also, it can be annoying to lose your session to the tests.  With the landing of bug 548172 if you have Xvnc installed and exposed via /usr/bin/vncserver and a password defined in ~/.vnc/passwd, we will use it to spin up a VNC session and run the mozmill tests in that session.  On Fedora Core 12, this is provided by the tigervnc-server package.

There are two things you need to do / be aware of:

  1. You must create a password at ~/.vnc/passwd for this to work.  This is accomplished by invoking the "vncpasswd" script.
  2. The default xstartup script (~/.vnc/xstartup), at least on my system, uses the default X startup junk from /etc.  That may start up more than you really want spun up in the testing session.  For example, when I connected with vncviewer I noticed that my gnome-panel was in there running a system monitor.  I personally have whacked my xstartup file to lose the /etc stuff and just invoke xfwm4 (the xfce window manager which I specially installed):
#!/bin/sh

vncconfig -iconic &
unset SESSION_MANAGER
unset DBUS_SESSION_BUS_ADDRESS
OS=`uname -s`
xsetroot -solid grey
xterm -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" &
xfwm4 &

You can connect to the vnc server to see what it is doing by invoking "vncviewer :99".  (We always run mozmill tests on display 99, currently.)  You will need to enter the password.

Developing Tests

{{ draft() }}

The Thunderbird mozmill tests and infrastructure are stored in {{ tbsource("mail/test/mozmill") }}.

Each set of tests is stored in a sub-directory, the tests that are run on a full mozmill run are listed in {{ tbsource("mail/test/mozmill/mozmilltests.list") }}.

MozMill has its own documentation that is useful to examine to get used to working with MozMill. However, Thunderbird has its own set of APIs/styles that it uses to interact with mozmill, and these are contained in the {{ tbsource("mail/test/mozmill/shared-modules/") }} directory. This code is shared across the different test sub-directories being included in the tests where required, documentation for these functions is contained within the code in doxygen format.

These files give examples of MozMill tests in Thunderbird and are considered relatively simple examples to get started with:

  • {{ tbsource("mail/test/mozmill/junk-commands/test-junk-commands.js") }}
  • {{ tbsource("mail/test/mozmill/content-policy/test-js-content-policy.js") }}
  • {{ tbsource("mail/test/mozmill/search-window/test-right-click-to-open-search-window.js") }}

Need info on:

  • Support modules
  • Profiles
  • Running MozMill in a normal profile
  • ...

Revision Source

<p>MozMill is the framework that we're using to run interactive tests.&nbsp; Mozmill tests effectively run the entire application through unit tests.</p>
<h3 id="Installing_MozMill">Installing MozMill</h3>
<p>{{ obsolete_header("9") }}</p>
<p><strong>Starting Thunderbird 9, MozMill no longer needs to be installed separately. It is included in the sources and will be installed into a virtual environment during the build process. So you can proceed with <a href="/en/Thunderbird/Thunderbird_MozMill_Testing#Running_Mozmill_Tests">Running Mozmill Tests</a>.</strong></p>
<p>Thunderbird's MozMill test harness currently requires using MozMill 1.4.2b1; more recent versions will not work at this time. &nbsp;If you need to use a more recent version of mozmill for other software development, you will want to look into <a class="external" href="http://pypi.python.org/pypi/virtualenv" title="http://pypi.python.org/pypi/virtualenv">virtualenv</a>.</p>
<h4 id="Windows">Windows</h4>
<p>You'll need <a class="internal" href="/En/Developer_Guide/Build_Instructions/Windows_Prerequisites#MozillaBuild" title="En/Developer Guide/Build Instructions/Windows
Prerequisites#MozillaBuild">MozillaBuild 1.4 or higher</a> for this. Start a MozillaBuild prompt and use it to install MozMill 1.4.2b1:</p>
<pre>
 easy_install http://pypi.rhaptos.org/mozrunner/mozrunner-2.5.2.tar.gz
 easy_install http://pypi.rhaptos.org/jsbridge/jsbridge-2.3.7b1.tar.gz
&nbsp;easy_install http://pypi.rhaptos.org/mozmill/mozmill-1.4.2b1.tar.gz
</pre>
<h4 id="Mac.2FLinux">Mac/Linux</h4>
<p>You'll need root privileges for this.</p>
<pre>
 sudo easy_install http://pypi.rhaptos.org/mozrunner/mozrunner-2.5.2.tar.gz
 sudo easy_install http://pypi.rhaptos.org/jsbridge/jsbridge-2.3.7b1.tar.gz
&nbsp;sudo easy_install http://pypi.rhaptos.org/mozmill/mozmill-1.4.2b1.tar.gz
</pre>
<p>If you, for some reason, need a different version, see <a class="internal" href="/en/Thunderbird/Thunderbird_MozMill_Testing/MozMill_Development_Install" title="en/Thunderbird/Thunderbird MozMill Testing/MozMill Development Install">MozMill Development Install</a>.</p>
<h3 id="Running_Mozmill_Tests"><a name="runtests">Running Mozmill Tests</a></h3>
<p><a name="runtests"> </a></p>
<p><a name="runtests">{{ Note("Although running a Thunderbird instance and a MozMill Thunderbird instance alongside each other in the same user login is possible, MozMill tends not to detect shutdown of the MozMill Instance correctly. If your test isn't finishing (even though Thunderbird has shut down)&nbsp;try closing any other instances of Thunderbird running in the user login.") }}</a></p>
<h4 id="Running_a_test_suite.3A"><a name="runtests">Running a test suite:</a></h4>
<p><a name="runtests">Run this from <code>objdir/</code> to run all the tests in the test suite.</a></p>
<pre>
<a name="runtests"><code>make mozmill</code></a></pre>
<p>For builds on Windows (created by {{ tbsource("mozilla/mach", "mozilla/mach") }} and implicitly {{ tbsource("build/pymake/make.py", "build/pymake/make.py") }}):</p>
<pre>
<a name="runtests">../build/pymake/make.py<code> mozmill</code></a></pre>
<h4 id="Running_a_specific_test.3A"><a name="runtests">Running a specific test:</a></h4>
<p><a name="runtests">For example to run all the {{ tbsource("mail/test/mozmill/folder-display/", "folder-display") }} tests, go to the objdir and run:</a></p>
<pre>
<a name="runtests"><code>make SOLO_TEST=folder-display mozmill-one</code></a></pre>
<p>For builds on Windows (created by {{ tbsource("mozilla/mach", "mozilla/mach") }} and implicitly {{ tbsource("build/pymake/make.py", "build/pymake/make.py") }}):</p>
<pre>
<a name="runtests">../build/pymake/make.py<code> SOLO_TEST=folder-display mozmill-one</code></a></pre>
<p><a name="runtests"><code>SOLO_TEST</code> may be a directory containing tests within {{ tbsource("mail/test/mozmill/", "mail/test/mozmill") }}, or a specific test file.</a></p>
<p><a name="runtests">To run just a single test ("all the tests in a single file"), like test-message-window.js, again go to the objdir and run:</a></p>
<pre>
<a name="runtests"><code>make SOLO_TEST=folder-display/test-message-window.js mozmill-one </code></a></pre>
<p>For builds on Windows (created by {{ tbsource("mozilla/mach", "mozilla/mach") }} and implicitly {{ tbsource("build/pymake/make.py", "build/pymake/make.py") }}):</p>
<pre>
<a name="runtests">../build/pymake/make.py<code> SOLO_TEST=folder-display/test-message-window.js mozmill-one</code></a></pre>
<h2 id="Running_MozMill_Tests_with_pre-compiled_Thunderbird_('packaged_tests')"><a name="runtests">Running MozMill Tests with pre-compiled Thunderbird ('packaged tests')</a></h2>
<p><a name="runtests">If you don't have a self-compiled build of Thunderbird, then you can run tests against pre-compiled Thunderbird builds. This is known as running in the 'packaged tests' style.</a></p>
<p><a name="runtests"></a><a href="/en/Thunderbird/Thunderbird_MozMill_Testing/Running_Thunderbird_MozMill_tests_from_packaged_tests" title="https://developer.mozilla.org/en/Thunderbird/Thunderbird_MozMill_Testing/Running_Thunderbird_MozMill_tests_from_packaged_tests">See this page for how to run in package test style</a>.</p>
<h3 id="Linux_.26.C2.A0VNC_Server">Linux &amp;&nbsp;VNC Server</h3>
<p>Linux runs have a lot of issues with focus related problems in more recents of gecko.&nbsp; Also, it can be annoying to lose your session to the tests.&nbsp; With the landing of bug 548172 if you have Xvnc installed and exposed via /usr/bin/vncserver and a password defined in ~/.vnc/passwd, we will use it to spin up a VNC session and run the mozmill tests in that session.&nbsp; On Fedora Core 12, this is provided by the tigervnc-server package.</p>
<p>There are two things you need to do / be aware of:</p>
<ol>
  <li>You must create a password at ~/.vnc/passwd for this to work.&nbsp; This is accomplished by invoking the "vncpasswd" script.</li>
  <li>The default xstartup script (~/.vnc/xstartup), at least on my system, uses the default X startup junk from /etc.&nbsp; That may start up more than you really want spun up in the testing session.&nbsp; For example, when I connected with vncviewer I noticed that my gnome-panel was in there running a system monitor.&nbsp; I personally have whacked my xstartup file to lose the /etc stuff and just invoke xfwm4 (the xfce window manager which I specially installed):</li>
</ol>
<pre>
#!/bin/sh

vncconfig -iconic &amp;
unset SESSION_MANAGER
unset DBUS_SESSION_BUS_ADDRESS
OS=`uname -s`
xsetroot -solid grey
xterm -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" &amp;
xfwm4 &amp;
</pre>
<p>You can connect to the vnc server to see what it is doing by invoking "vncviewer :99". &nbsp;(We always run mozmill tests on display 99, currently.)&nbsp; You will need to enter the password.</p>
<h3 id="Developing_Tests">Developing Tests</h3>
<p>{{ draft() }}</p>
<p>The Thunderbird mozmill tests and infrastructure are stored in {{ tbsource("mail/test/mozmill") }}.</p>
<p>Each set of tests is stored in a sub-directory, the tests that are run on a full mozmill run are listed in {{ tbsource("mail/test/mozmill/mozmilltests.list") }}.</p>
<p>MozMill has its own <a href="/en/Mozmill" title="en/Mozmill">documentation</a> that is useful to examine to get used to working with MozMill. However, Thunderbird has its own set of APIs/styles that it uses to interact with mozmill, and these are contained in the {{ tbsource("mail/test/mozmill/shared-modules/") }} directory. This code is shared across the different test sub-directories being included in the tests where required, documentation for these functions is contained within the code in doxygen format.</p>
<p>These files give examples of MozMill tests in Thunderbird and are considered relatively simple examples to get started with:</p>
<ul>
  <li>{{ tbsource("mail/test/mozmill/junk-commands/test-junk-commands.js") }}</li>
  <li>{{ tbsource("mail/test/mozmill/content-policy/test-js-content-policy.js") }}</li>
  <li>{{ tbsource("mail/test/mozmill/search-window/test-right-click-to-open-search-window.js") }}</li>
</ul>
<p>Need info on:</p>
<ul>
  <li>Support modules</li>
  <li>Profiles</li>
  <li>Running MozMill in a normal profile</li>
  <li>...</li>
</ul>
Revert to this revision