Mozharness FAQ

  • Revision slug: Mozharness_FAQ
  • Revision title: Mozharness FAQ
  • Revision id: 357133
  • Created:
  • Creator: jgriffin
  • Is current revision? No
  • Comment

Revision Content

Design and Architecture

Q. What does mozharness do?  What problems does it solve for me?

A. "Mozharness is a configuration-driven python script harness with full logging that allows production infrastructure and individual developers to use the same scripts."
 
designed to be able to run in multiple environments/platforms/configurations
Since mozharness allows for complex configuration, scripts don't need to hardcode as much behavior, and running a script on a different-but-similar system has a better chance of running (with a new configuration file).
designed to be easier to debug or replicate problems
The logs are often comprehensive enough to debug problems without running.
When they're not enough, mozharness is designed to be able to run scripts standalone, without a full buildbot setup.
designed to be able to iterate over small sections of code more easily
When writing or debugging large pieces of automation, it can be time consuming to run through the entire thing over and over, when you really want to test and iterate on a section.  Mozharness' actions allow you to run each action individually, or skip specific actions, or run a subset of the actions easily.

Q. What are the parts of mozharness and how do they all fit together?

A. Mozharness lives at http://hg.mozilla.org/build/mozharness/; its primary parts are as follows:
 
  • mozharness.base.* contains generic script harness code: configuration, logging, vcs support, and the like.  These libraries could be used anywhere.
  • mozharness.mozilla.* contains mozilla-specific code.  Talos support, Firefox unittest support, Firefox localization, code to interface with the existing Mozilla buildbot infrastructure.
  • scripts/ contains the mozharness scripts.
  • configs/ contains configuration files to run mozharness scripts on different platforms/branches/environments.
There are three core classes:
 
mozharness.base.script.BaseScript
All scripts derive from this class.  It provides the logic for actions, creates a self.log_obj and self.config (from MultiFileLogger and BaseConfig+ReadOnlyDict), as well as methods to do basic scripting with logging.
 
For instance, self.mkdir_p(path) will basically os.makedirs(path), but will also log that we're doing so (via self.log_obj).  self.run_command() and self.get_output_from_command() allow the script to call other executables easily, while keeping information in the logs.
 
BaseScript.run() makes the script go.
mozharness.base.config.BaseConfig
This combines the initial script config, the config file, and the command line options into a single config, which becomes the BaseScript self.config.  This config is locked (via ReadOnlyDict) at the end of BaseScript.__init__() and dumped to the logs for ease of debugging and more predictable behavior.
mozharness.base.log.LogMixin
Almost every mozharness object inherits this mixin at some level, to allow for the various log-level methods.
 
If self.log_obj isn't set, we fall back to print's.  If it is set to a subclass of BaseLogger (or otherwise compatible logging object that has a log_message() method), then we log through that object.
 
By default, BaseScript uses MultiFileLogger as its log_obj, which creates a file per logging level.
 
There are also a number of other classes and mixins that provide useful functionality but aren't required for all script/job types.  This semi-complete list will most likely fall out of date at some point:
 
  • mozharness.base.log.OutputParser - parses for errors from BaseScript.run_command()
  • mozharness.base.parallel.ChunkingMixin - split a long list of items up into separate jobs.
  • mozharness.base.signing.AndroidSigningMixin - Android signing
  • mozharness.base.transfer.TransferMixin - uploading/downloading directories
  • mozharness.base.vcs.vcsbase.VCSScript - provides mercurial functionality.  More VCS support as they're added.
  • mozharness.mozilla.buildbot.BuildbotMixin - interface with buildbot and mozilla-specific buildbot setup
  • mozharness.mozilla.l10n.locales.LocalesMixin - localization support
  • mozharness.mozilla.l10n.multi_locale_build.MultiLocaleBuild
  • mozharness.mozilla.mock.MockMixin - mock_mozilla support
  • mozharness.mozilla.release.ReleaseMixin and SigningMixin - release automation support
  • mozharness.mozilla.testing.device.DeviceMixin - Android adb + sut device controlling support
  • mozharness.mozilla.testing.talos.Talos - talos support
  • mozharness.mozilla.testing.testbase.TestingMixin - unittest and talos support
  • mozharness.mozilla.tooltool.TooltoolMixin - tooltool support

Q. What classes of mozharness scripts are there (or should there be)?  Tests, tools, etc?

A. Right now mozharness is slated to replace buildbotcustom as MoCo releng's source of automation logic.

There are/will be scripts in mozharness for builds, tests, repacks, releases.  We may also put machine maintenance type scripts here; anything we want run on releng build+test farms.  At some point we will just be able to consider these compute farms, and mozharness will be what runs on that compute farm.
 
Since mozharness is just a python harness, almost anything that can be written in Python could potentially be added to mozharness.
 

Q. I want to write a mozharness script for a new automation problem.   How do I do this?

For your first mozharness script, it's probably easiest to use existing, similar scripts as examples; that's definitely easier than writing an entirely new type of script.
 
  • Split your script into logical chunks, or actions.
  • What actions are needed by some people or jobs, but not others?
  • What actions could potentially be independent of others?  For example, installing Firefox  to run tests.  You don't necessarily have to do this if you point to a  pre-installed Firefox to run those tests.  Or packaging after building.  Or uploading the logs somewhere.
  • Since actions can be run by themselves, or skipped, they're a logical place to split up a script's behavior.
  • These will be listed in the script's all_actions.  A default set of actions, if different from all_actions, is defined in default_actions.  But the user can turn any or all actions on or off via command line options or config file.
  • You need to have a method per action.  This method will be named the same as the action, with underscores replacing dashes.
  • If you want to define pre-action or post-action behavior, you can write optional preflight_actionname() and/or postflight_actionname() methods, again with underscores instead of dashes.
 
Figure out the different configurations that might be in play; these should be pre-defineable in either command line options or config files.  (Complex data structures are best suited in config files; commonly overridden options should be available via command line.)

Revision Source

<h3 id="Design_and_Architecture">Design and Architecture</h3>
<h4 id="Q._What_does_mozharness_do.3F.C2.A0_What_problems_does_it_solve_for_me.3F"><strong><span class="author-g-jti5gfjw5bf4o4x2">Q.</span><span class="author-g-gmeqsyoivrkikiba"> What does mozharness do?&nbsp; What problems does it solve for me?</span></strong></h4>
<div id="magicdomid12">
  <span class="author-g-ijx5rjfb8o6e7gv5 i">A. <i>"Mozharness is a configuration-driven python script harness with full logging that allows production infrastructure and individual developers to use the same scripts."</i></span></div>
<div>
  &nbsp;</div>
<dl>
  <dt>
    <span class="author-g-ijx5rjfb8o6e7gv5">designed to be able to run in multiple environments/platforms/configurations</span></dt>
  <dd>
    <span class="author-g-ijx5rjfb8o6e7gv5">Since mozharness allows for complex configuration, scripts don't need to hardcode as much behavior, and running a script on a different-but-similar system has a better chance of running (with a new configuration file).</span></dd>
</dl>
<dl>
  <dt id="magicdomid18">
    <span class="author-g-ijx5rjfb8o6e7gv5">designed to be easier to debug or replicate problems</span></dt>
  <dd>
    <div id="magicdomid20">
      <span class="author-g-ijx5rjfb8o6e7gv5">The logs are often comprehensive enough to debug problems without running.</span></div>
    <div id="magicdomid21">
      <span class="author-g-ijx5rjfb8o6e7gv5">When they're not enough, mozharness is designed to be able to run scripts standalone, without a full buildbot setup.</span></div>
  </dd>
</dl>
<dl>
  <dt id="magicdomid23">
    <span class="author-g-ijx5rjfb8o6e7gv5">designed to be able to iterate over small sections of code more easily</span></dt>
  <dd>
    <div id="magicdomid25">
      <span class="author-g-ijx5rjfb8o6e7gv5">When writing or debugging large pieces of automation, it can be time consuming to run through the entire thing over and over, when you really want to test and iterate on a section.&nbsp; Mozharness' actions allow you to run each action individually, or skip specific actions, or run a subset of the actions easily.</span></div>
  </dd>
</dl>
<div id="magicdomid24">
  <h4 id="Q._What_are_the_parts_of_mozharness_and_how_do_they_all_fit_together.3F"><span class="author-g-jti5gfjw5bf4o4x2">Q.</span><span class="author-g-gmeqsyoivrkikiba"> What are the parts of mozharness and how do they all fit together?</span></h4>
  <div id="magicdomid31">
    <span class="author-g-ijx5rjfb8o6e7gv5">A. Mozharness lives at </span><a href="http://hg.mozilla.org/build/mozharness/" title="/en-US/docs/">http://hg.mozilla.org/build/mozharness/</a>; its primary parts are as follows:</div>
  <div>
    &nbsp;</div>
  <ul>
    <li><span class="author-g-ijx5rjfb8o6e7gv5">mozharness.base.* contains generic script harness code: configuration, logging, vcs support, and the like.&nbsp; These libraries could be used anywhere.</span></li>
    <li><span class="author-g-ijx5rjfb8o6e7gv5">mozharness.mozilla.* contains mozilla-specific code.&nbsp; Talos support, Firefox unittest support, Firefox localization, code to interface with the existing Mozilla buildbot infrastructure.</span></li>
    <li><span class="author-g-ijx5rjfb8o6e7gv5">scripts/ contains the mozharness scripts.</span></li>
    <li><span class="author-g-ijx5rjfb8o6e7gv5">configs/ contains configuration files to run mozharness scripts on different platforms/branches/environments.</span></li>
  </ul>
  <div id="magicdomid39">
    <span class="author-g-ijx5rjfb8o6e7gv5">There are three core classes:</span></div>
  <div>
    &nbsp;</div>
  <dl>
    <dt id="magicdomid41">
      <span class="author-g-ijx5rjfb8o6e7gv5">mozharness.base.script.BaseScript</span></dt>
    <dd>
      <div id="magicdomid43">
        <span class="author-g-ijx5rjfb8o6e7gv5">All scripts derive from this class.&nbsp; It provides the logic for actions, creates a self.log_obj and self.config (from MultiFileLogger and BaseConfig+ReadOnlyDict), as well as methods to do basic scripting with logging.</span></div>
      <div id="magicdomid44">
        &nbsp;</div>
      <div id="magicdomid45">
        <span class="author-g-ijx5rjfb8o6e7gv5">For instance, self.mkdir_p(path) will basically os.makedirs(path), but will also log that we're doing so (via self.log_obj).&nbsp; self.run_command() and self.get_output_from_command() allow the script to call other executables easily, while keeping information in the logs.</span></div>
      <div id="magicdomid46">
        &nbsp;</div>
      <div id="magicdomid47">
        <span class="author-g-ijx5rjfb8o6e7gv5">BaseScript.run() makes the script go.</span></div>
    </dd>
  </dl>
  <dl>
    <dt id="magicdomid49">
      <span class="author-g-ijx5rjfb8o6e7gv5">mozharness.base.config.BaseConfig</span></dt>
    <dd>
      <span class="author-g-ijx5rjfb8o6e7gv5">This combines the initial script config, the config file, and the command line options into a single config, which becomes the BaseScript self.config.&nbsp; This config is locked (via ReadOnlyDict) at the end of BaseScript.__init__() and dumped to the logs for ease of debugging and more predictable behavior.</span></dd>
  </dl>
  <dl>
    <dt id="magicdomid53">
      <span class="author-g-ijx5rjfb8o6e7gv5">mozharness.base.log.LogMixin</span></dt>
    <dd>
      <div id="magicdomid55">
        <span class="author-g-ijx5rjfb8o6e7gv5">Almost every mozharness object inherits this mixin at some level, to allow for the various log-level methods.</span></div>
      <div id="magicdomid56">
        &nbsp;</div>
      <div id="magicdomid57">
        <span class="author-g-ijx5rjfb8o6e7gv5">If self.log_obj isn't set, we fall back to print's.&nbsp; If it is set to a subclass of BaseLogger (or otherwise compatible logging object that has a log_message() method), then we log through that object.</span></div>
      <div id="magicdomid58">
        &nbsp;</div>
      <div id="magicdomid59">
        <span class="author-g-ijx5rjfb8o6e7gv5">By default, BaseScript uses MultiFileLogger as its log_obj, which creates a file per logging level.</span></div>
    </dd>
  </dl>
  <div>
    &nbsp;</div>
  <div id="magicdomid61">
    <span class="author-g-ijx5rjfb8o6e7gv5">There are also a number of other classes and mixins that provide useful functionality but aren't required for all script/job types.&nbsp; This semi-complete list will most likely fall out of date at some point:</span></div>
  <div>
    &nbsp;</div>
  <ul>
    <li id="magicdomid63"><span class="author-g-ijx5rjfb8o6e7gv5">mozharness.base.log.OutputParser - parses for errors from BaseScript.run_command()</span></li>
    <li id="magicdomid64"><span class="author-g-ijx5rjfb8o6e7gv5">mozharness.base.parallel.ChunkingMixin - split a long list of items up into separate jobs.</span></li>
    <li id="magicdomid65"><span class="author-g-ijx5rjfb8o6e7gv5">mozharness.base.signing.AndroidSigningMixin - Android signing</span></li>
    <li id="magicdomid66"><span class="author-g-ijx5rjfb8o6e7gv5">mozharness.base.transfer.TransferMixin - uploading/downloading directories</span></li>
    <li id="magicdomid67"><span class="author-g-ijx5rjfb8o6e7gv5">mozharness.base.vcs.vcsbase.VCSScript - provides mercurial functionality.&nbsp; More VCS support as they're added.</span></li>
    <li id="magicdomid68"><span class="author-g-ijx5rjfb8o6e7gv5">mozharness.mozilla.buildbot.BuildbotMixin - interface with buildbot and mozilla-specific buildbot setup</span></li>
    <li id="magicdomid69"><span class="author-g-ijx5rjfb8o6e7gv5">mozharness.mozilla.l10n.locales.LocalesMixin - localization support</span></li>
    <li id="magicdomid70"><span class="author-g-ijx5rjfb8o6e7gv5">mozharness.mozilla.l10n.multi_locale_build.MultiLocaleBuild</span></li>
    <li id="magicdomid71"><span class="author-g-ijx5rjfb8o6e7gv5">mozharness.mozilla.mock.MockMixin - mock_mozilla support</span></li>
    <li id="magicdomid72"><span class="author-g-ijx5rjfb8o6e7gv5">mozharness.mozilla.release.ReleaseMixin and SigningMixin - release automation support</span></li>
    <li id="magicdomid73"><span class="author-g-ijx5rjfb8o6e7gv5">mozharness.mozilla.testing.device.DeviceMixin - Android adb + sut device controlling support</span></li>
    <li id="magicdomid74"><span class="author-g-ijx5rjfb8o6e7gv5">mozharness.mozilla.testing.talos.Talos - talos support</span></li>
    <li id="magicdomid75"><span class="author-g-ijx5rjfb8o6e7gv5">mozharness.mozilla.testing.testbase.TestingMixin - unittest and talos support</span></li>
    <li id="magicdomid76"><span class="author-g-ijx5rjfb8o6e7gv5">mozharness.mozilla.tooltool.TooltoolMixin - tooltool support</span></li>
  </ul>
</div>
<h4><span class="author-g-gmeqsyoivrkikiba">Q. What classes of mozharness scripts are there (or should there be)?&nbsp; Tests, tools, etc?</span></h4>
<p><span class="author-g-gmeqsyoivrkikiba">A. </span><span class="author-g-ijx5rjfb8o6e7gv5">Right now mozharness is slated to replace buildbotcustom as MoCo releng's source of automation logic.</span></p>
<div id="magicdomid83">
  <span class="author-g-ijx5rjfb8o6e7gv5">There are/will be scripts in mozharness for builds, tests, repacks, releases.&nbsp; We may also put machine maintenance type scripts here; anything we want run on releng build+test farms.&nbsp; At some point we will just be able to consider these compute farms, and mozharness will be what runs on that compute farm.</span></div>
<div id="magicdomid84">
  &nbsp;</div>
<div id="magicdomid85">
  <span class="author-g-ijx5rjfb8o6e7gv5">Since mozharness is just a python harness, almost anything that can be written in Python could potentially be added to mozharness.</span></div>
<div>
  &nbsp;</div>
<h4>Q. <span class="author-g-gmeqsyoivrkikiba">I want to write a mozharness script for a new automation problem.&nbsp;&nbsp; </span>How do I do this?</h4>
<div id="magicdomid90">
  <span class="author-g-ijx5rjfb8o6e7gv5">For your first mozharness script, it's probably easiest to use existing, similar scripts as examples; that's definitely easier than writing an entirely new type of script.</span></div>
<div id="magicdomid91">
  &nbsp;</div>
<ul>
  <li id="magicdomid92"><span class="author-g-ijx5rjfb8o6e7gv5">Split your script into logical chunks, or actions.</span></li>
  <li id="magicdomid93"><span class="author-g-ijx5rjfb8o6e7gv5">What actions are needed by some people or jobs, but not others?</span></li>
  <li id="magicdomid94"><span class="author-g-ijx5rjfb8o6e7gv5">What actions could potentially be independent of others?&nbsp; For example, installing Firefox&nbsp; to run tests.&nbsp; You don't necessarily have to do this if you point to a&nbsp; pre-installed Firefox to run those tests.&nbsp; Or packaging after building.&nbsp; Or uploading the logs somewhere.</span></li>
  <li id="magicdomid95"><span class="author-g-ijx5rjfb8o6e7gv5">Since actions can be run by themselves, or skipped, they're a logical place to split up a script's behavior.</span></li>
  <li><span class="author-g-ijx5rjfb8o6e7gv5">These will be listed in the script's all_actions.&nbsp; A default set of actions, if different from all_actions, is defined in default_actions.&nbsp; But the user can turn any or all actions on or off via command line options or config file.</span></li>
  <li id="magicdomid97"><span class="author-g-ijx5rjfb8o6e7gv5">You need to have a method per action.&nbsp; This method will be named the same as the action, with underscores replacing dashes.</span></li>
  <li id="magicdomid98"><span class="author-g-ijx5rjfb8o6e7gv5">If you want to define pre-action or post-action behavior, you can write optional preflight_actionname() and/or postflight_actionname() methods, again with underscores instead of dashes.</span></li>
</ul>
<div id="magicdomid99">
  &nbsp;</div>
<div id="magicdomid100">
  <span class="author-g-ijx5rjfb8o6e7gv5">Figure out the different configurations that might be in play; these should be pre-defineable in either command line options or config files.&nbsp; (Complex data structures are best suited in config files; commonly overridden options should be available via command line.)</span></div>
Revert to this revision