Mozharness FAQ

  • Revision slug: Mozharness_FAQ
  • Revision title: Mozharness FAQ
  • Revision id: 357147
  • Created:
  • Creator: jgriffin
  • Is current revision? No
  • Comment

Revision Content

Design and Architecture

Q. What does mozharness do?  What problems does it solve for me?

A. "Mozharness is a configuration-driven python script harness with full logging that allows production infrastructure and individual developers to use the same scripts."
 
designed to be able to run in multiple environments/platforms/configurations
Since mozharness allows for complex configuration, scripts don't need to hardcode as much behavior, and running a script on a different-but-similar system has a better chance of running (with a new configuration file).
designed to be easier to debug or replicate problems
The logs are often comprehensive enough to debug problems without running.
When they're not enough, mozharness is designed to be able to run scripts standalone, without a full buildbot setup.
designed to be able to iterate over small sections of code more easily
When writing or debugging large pieces of automation, it can be time consuming to run through the entire thing over and over, when you really want to test and iterate on a section.  Mozharness' actions allow you to run each action individually, or skip specific actions, or run a subset of the actions easily.

Q. What are the parts of mozharness and how do they all fit together?

A. Mozharness lives at http://hg.mozilla.org/build/mozharness/; its primary parts are as follows:
 
  • mozharness.base.* contains generic script harness code: configuration, logging, vcs support, and the like.  These libraries could be used anywhere.
  • mozharness.mozilla.* contains mozilla-specific code.  Talos support, Firefox unittest support, Firefox localization, code to interface with the existing Mozilla buildbot infrastructure.
  • scripts/ contains the mozharness scripts.
  • configs/ contains configuration files to run mozharness scripts on different platforms/branches/environments.
There are three core classes:
 
mozharness.base.script.BaseScript
All scripts derive from this class.  It provides the logic for actions, creates a self.log_obj and self.config (from MultiFileLogger and BaseConfig+ReadOnlyDict), as well as methods to do basic scripting with logging.
 
For instance, self.mkdir_p(path) will basically os.makedirs(path), but will also log that we're doing so (via self.log_obj).  self.run_command() and self.get_output_from_command() allow the script to call other executables easily, while keeping information in the logs.
 
BaseScript.run() makes the script go.
mozharness.base.config.BaseConfig
This combines the initial script config, the config file, and the command line options into a single config, which becomes the BaseScript self.config.  This config is locked (via ReadOnlyDict) at the end of BaseScript.__init__() and dumped to the logs for ease of debugging and more predictable behavior.
mozharness.base.log.LogMixin
Almost every mozharness object inherits this mixin at some level, to allow for the various log-level methods.
 
If self.log_obj isn't set, we fall back to print's.  If it is set to a subclass of BaseLogger (or otherwise compatible logging object that has a log_message() method), then we log through that object.
 
By default, BaseScript uses MultiFileLogger as its log_obj, which creates a file per logging level.
 
There are also a number of other classes and mixins that provide useful functionality but aren't required for all script/job types.  This semi-complete list will most likely fall out of date at some point:
 
  • mozharness.base.log.OutputParser - parses for errors from BaseScript.run_command()
  • mozharness.base.parallel.ChunkingMixin - split a long list of items up into separate jobs.
  • mozharness.base.signing.AndroidSigningMixin - Android signing
  • mozharness.base.transfer.TransferMixin - uploading/downloading directories
  • mozharness.base.vcs.vcsbase.VCSScript - provides mercurial functionality.  More VCS support as they're added.
  • mozharness.mozilla.buildbot.BuildbotMixin - interface with buildbot and mozilla-specific buildbot setup
  • mozharness.mozilla.l10n.locales.LocalesMixin - localization support
  • mozharness.mozilla.l10n.multi_locale_build.MultiLocaleBuild
  • mozharness.mozilla.mock.MockMixin - mock_mozilla support
  • mozharness.mozilla.release.ReleaseMixin and SigningMixin - release automation support
  • mozharness.mozilla.testing.device.DeviceMixin - Android adb + sut device controlling support
  • mozharness.mozilla.testing.talos.Talos - talos support
  • mozharness.mozilla.testing.testbase.TestingMixin - unittest and talos support
  • mozharness.mozilla.tooltool.TooltoolMixin - tooltool support

Q. What classes of mozharness scripts are there (or should there be)?  Tests, tools, etc?

A. Right now mozharness is slated to replace buildbotcustom as MoCo releng's source of automation logic.

There are/will be scripts in mozharness for builds, tests, repacks, releases.  We may also put machine maintenance type scripts here; anything we want run on releng build+test farms.  At some point we will just be able to consider these compute farms, and mozharness will be what runs on that compute farm.
 
Since mozharness is just a python harness, almost anything that can be written in Python could potentially be added to mozharness.
 

Q. I want to write a mozharness script for a new automation problem.   How do I do this?

For your first mozharness script, it's probably easiest to use existing, similar scripts as examples; that's definitely easier than writing an entirely new type of script.

Split your script into logical chunks, or actions:

  • What actions are needed by some people or jobs, but not others?
  • What actions could potentially be independent of others?  For example, installing Firefox  to run tests.  You don't necessarily have to do this if you point to a  pre-installed Firefox to run those tests.  Or packaging after building.  Or uploading the logs somewhere.
  • Since actions can be run by themselves, or skipped, they're a logical place to split up a script's behavior.
  • These will be listed in the script's all_actions.  A default set of actions, if different from all_actions, is defined in default_actions.  But the user can turn any or all actions on or off via command line options or config file.
  • You need to have a method per action.  This method will be named the same as the action, with underscores replacing dashes.
  • If you want to define pre-action or post-action behavior, you can write optional preflight_actionname() and/or postflight_actionname() methods, again with underscores instead of dashes.
 
Figure out the different configurations that might be in play; these should be pre-defineable in either command line options or config files.  (Complex data structures are best suited in config files; commonly overridden options should be available via command line.)
 

Coding

Q. What are all the possible mozharness actions, and what dependencies exist between them?

A. This is very much script- or script-family- specific.  Any mozharness script can define any arbitrary list of all_actions.  Each action has a matching method of the same name (after substituting underscores for dashes).

 
We attempt to make the actions standalone, so you could potentially run action #4 over and over without having to run actions 1-3 over and over as well.  However, action #4 probably needs actions 1-3 to have run at some point (clone repos, download installers, etc.), or for the user to have manually done those steps beforehand.
 
If you're asking what actions have been defined in existing scripts, a 
    grep -r 'all_actions\s*=' .
might be a good start.  This will change, so a static list here will be obsoleted.
 
Some actions with pre-written methods inside of mozharness.base.* or mozharness.mozilla.* include:
 
# clobber
# pull
# list-locales (l10n)
# pull-locale-source (l10n)
# passphrase (android signing)
# create-virtualenv (python)
# read-buildbot-config (buildbot interface)
# download-and-extract (tests)
# install (tests)
 
Perhaps a mozharness- level document like this won't be as useful for questions like this, as much as script-family level documentation.  For instance, "What actions do TestMixin- (or LocalesMixin-, or ...?) based mozharness scripts have defined already and what do they do?"
 

Q. I want to use a python library in a mozharness script.  How can I do that? 

If it's a python built-in, you can import as normal.  We currently support python 2.5.1 -> 2.7.x, but we're moving towards having python 2.7.3 everywhere.
 
If  it's an external package, you most likely need mozharness to install it  into your virtualenv (see PythonMixin.create_virtualenv()).
 
If there's a commandline tool that's also installed (say, mozinstall), you can call it from self.run_command().  For instance, TestMixin.install() calls mozinstall like this:  http://hg.mozilla.org/build/mozharness/file/3334bfde4eed/mozharness/mozilla/testing/testbase.py#l229 with a preflight check to make sure mozinstall is in the virtualenv, first: http://hg.mozilla.org/build/mozharness/file/3334bfde4eed/mozharness/mozilla/testing/testbase.py#l213
 
If the external package doesn't have a commandline tool, you need to import it.  However, since the module won't be available at the beginning of the script run, a bare |import module| will throw an exception before you create the virtualenv.
 
I solved this in SUTDeviceHandler by importing the module in a method that isn't called until after the virtualenv is created.  http://hg.mozilla.org/build/mozharness/file/b8104340b600/mozharness/mozilla/testing/device.py#l444
 

Q. I want to use a python library in a patch I'd like to contribute to the core mozharness classes.  How can I do that?

A. Same as above.  A core mozharness class will usually need to handle things in a more generic way than standalone scripts, but that's true of most core classes.

Q. I've just finished my mozharness script, and think it should live inside mozilla-central. Is this possible?

A. It's certainly possible, with the following caveats:

 
  • if the script lives in mozilla-central, it still (probably) depends on mozharness.base.* and mozharness.mozilla.* .  There needs to be some way to support that, whether it's cloning mozharness or checking a copy into m-c and maintaining that, pushing up to pypi, or some other approach.
  • as of this writing, mozharness is undergoing constant change.  anything checked into mozilla-central or pushed to pypi would probably go out-of-date fairly quickly, and would need regular code drops.
  • as described below, production mozharness jobs clone hg.m.o/build/mozharness, then run the script.  If the script doesn't live in mozharness, we would have to alter this process.  There would need to be a strong reason to do this.
 
This is complicated by the fact that only a subset of mozharness jobs require the tree, and most of those require a specific branch or revision in production.  If the scripts and logic to do this lived in the tree, we would either need external logic to do this, or have the script update itself to a potentially conflicting version, and reload itself mid-script.
 

Configuration

Q. What should/can I put in a production configuration file?  A test configuration file?

A. This actually is fairly script-dependent.

If we were to run, say, http://hg.mozilla.org/build/mozharness/file/bb6851655e8d/scripts/configtest.py configtest.py in production, we wouldn't necessarily need a config file at all.  We'd just need a python2 >=2.5 with json or simplejson support.
 
As you add functionality, the requirements grow.
 
If you need to upload:
  • Currently this is only doable on the build pool.
  • The settings needed vary depending on try/non-try, project, etc.; this is probably best for questions or a longer external document.
 
If you need to create a virtualenv:
 
If your production script will be running a different set of actions than developers:
  • Define a superset of all_actions
  • Define your in-script default_actions as the set that someone would use on the command line (developer centric)
  • Define a set of production config files that overrides default_actions for the set of actions pertinent to production.
 
If you're running on the test pool or build pool:
  • There are large differences between the test pool and build pool (and try pool), as the machines are configured differently.
  • We need to document these changes more to allow for easier usage.
 
If the script needs mercurial:
 
If your script will be run in ScriptFactory:
  • it needs to inherit BuildbotMixin
  • it needs to run self.read_buildbot_config() at some point before referring to revision, branch, installer_url, or test_url.  We've been doing this by adding a non-default read-buildbot-config action.
    • This will populate self.buildbot_config, which you can refer to.
  • the buildbot_json_path needs to be set to "buildprops.json".
These lists may not be complete and may change, but we can update this documentation over time.
 

Q. How do I inherit from one configuration file in another one?

A. Currently you can't, and I'd like to discourage that approach.

 
The  upside of flat config files is it's easy to annotate/blame/diff to see  exactly who changed what, when.  This is highly useful.  The downside is  there are more files to update when changing something that's shared  amongst multiple config files.
 
Inheriting/including/programmatically  creating config has the reverse upside/downside.  However, RelEng has  gone so far down this route that we're forever trying to figure out what  will change if we pull *this* string or who changed what, when, that  affected this seemingly unrelated thing.  We're moving towards  dynamically creating flat configs (and checking those flat configs into  revision control), which should hopefully give us a combination of  upsides with fewer downsides.
 
If  we have to choose one upside/downside combination, it's my (Aki's)  strong belief that optimizing for debugging + reading (read:  diff/blame/annotate) is more important than optimizing for writing,  since we're going to read these configs a lot more than we're going to  write them.
 
However, when https://bugzilla.mozilla.org/show_bug.cgi?id=779294 is resolved, you should be able to specify multiple config files on the command line, and there will be rules as to what to do with conflicting config entries.  Hopefully this will allow us to easily override things in certain cases, without potentially duplicating large swaths of configs in dozens of files.
 
Also, I don't mind having tools to easily compare config files, or build new ones programmatically, as long as the final flat configs are checked in for easy reading, diff, annotate, blame, etc.
 

Q. Is there a 'master' production configuration file that I can use?

A. No, and this doesn't necessarily make sense when you consider that "production configuration file" could refer to an Android multilocale build, Android signing, an l10n repack, a b2g build, a Firefox windows desktop unittest, or any other of a number of wildly disparate tasks.

 
 
We are starting to see "families" of scripts forming that do have a significant amount of overlap, however.  We can consolidate, or share, or logically split these up to take advantage of the multiple-config-files in bug 779294 when that's done.  We haven't tried to do this at the outset until obvious patterns and use cases emerged, but once those are apparent people can spend time to make this easier or more logical.

Revision Source

<h3 id="Design_and_Architecture">Design and Architecture</h3>
<h4 id="Q._What_does_mozharness_do.3F.C2.A0_What_problems_does_it_solve_for_me.3F"><strong><span class="author-g-jti5gfjw5bf4o4x2">Q.</span><span class="author-g-gmeqsyoivrkikiba"> What does mozharness do?&nbsp; What problems does it solve for me?</span></strong></h4>
<div id="magicdomid12">
  <span class="author-g-ijx5rjfb8o6e7gv5 i">A. <i>"Mozharness is a configuration-driven python script harness with full logging that allows production infrastructure and individual developers to use the same scripts."</i></span></div>
<div>
  &nbsp;</div>
<dl>
  <dt>
    <span class="author-g-ijx5rjfb8o6e7gv5">designed to be able to run in multiple environments/platforms/configurations</span></dt>
  <dd>
    <span class="author-g-ijx5rjfb8o6e7gv5">Since mozharness allows for complex configuration, scripts don't need to hardcode as much behavior, and running a script on a different-but-similar system has a better chance of running (with a new configuration file).</span></dd>
</dl>
<dl>
  <dt id="magicdomid18">
    <span class="author-g-ijx5rjfb8o6e7gv5">designed to be easier to debug or replicate problems</span></dt>
  <dd>
    <div id="magicdomid20">
      <span class="author-g-ijx5rjfb8o6e7gv5">The logs are often comprehensive enough to debug problems without running.</span></div>
    <div id="magicdomid21">
      <span class="author-g-ijx5rjfb8o6e7gv5">When they're not enough, mozharness is designed to be able to run scripts standalone, without a full buildbot setup.</span></div>
  </dd>
</dl>
<dl>
  <dt id="magicdomid23">
    <span class="author-g-ijx5rjfb8o6e7gv5">designed to be able to iterate over small sections of code more easily</span></dt>
  <dd>
    <div id="magicdomid25">
      <span class="author-g-ijx5rjfb8o6e7gv5">When writing or debugging large pieces of automation, it can be time consuming to run through the entire thing over and over, when you really want to test and iterate on a section.&nbsp; Mozharness' actions allow you to run each action individually, or skip specific actions, or run a subset of the actions easily.</span></div>
  </dd>
</dl>
<div id="magicdomid24">
  <h4 id="Q._What_are_the_parts_of_mozharness_and_how_do_they_all_fit_together.3F"><span class="author-g-jti5gfjw5bf4o4x2">Q.</span><span class="author-g-gmeqsyoivrkikiba"> What are the parts of mozharness and how do they all fit together?</span></h4>
  <div id="magicdomid31">
    <span class="author-g-ijx5rjfb8o6e7gv5">A. Mozharness lives at </span><a href="http://hg.mozilla.org/build/mozharness/" title="/en-US/docs/">http://hg.mozilla.org/build/mozharness/</a>; its primary parts are as follows:</div>
  <div>
    &nbsp;</div>
  <ul>
    <li><span class="author-g-ijx5rjfb8o6e7gv5">mozharness.base.* contains generic script harness code: configuration, logging, vcs support, and the like.&nbsp; These libraries could be used anywhere.</span></li>
    <li><span class="author-g-ijx5rjfb8o6e7gv5">mozharness.mozilla.* contains mozilla-specific code.&nbsp; Talos support, Firefox unittest support, Firefox localization, code to interface with the existing Mozilla buildbot infrastructure.</span></li>
    <li><span class="author-g-ijx5rjfb8o6e7gv5">scripts/ contains the mozharness scripts.</span></li>
    <li><span class="author-g-ijx5rjfb8o6e7gv5">configs/ contains configuration files to run mozharness scripts on different platforms/branches/environments.</span></li>
  </ul>
  <div id="magicdomid39">
    <span class="author-g-ijx5rjfb8o6e7gv5">There are three core classes:</span></div>
  <div>
    &nbsp;</div>
  <dl>
    <dt id="magicdomid41">
      <span class="author-g-ijx5rjfb8o6e7gv5">mozharness.base.script.BaseScript</span></dt>
    <dd>
      <div id="magicdomid43">
        <span class="author-g-ijx5rjfb8o6e7gv5">All scripts derive from this class.&nbsp; It provides the logic for actions, creates a self.log_obj and self.config (from MultiFileLogger and BaseConfig+ReadOnlyDict), as well as methods to do basic scripting with logging.</span></div>
      <div id="magicdomid44">
        &nbsp;</div>
      <div id="magicdomid45">
        <span class="author-g-ijx5rjfb8o6e7gv5">For instance, self.mkdir_p(path) will basically os.makedirs(path), but will also log that we're doing so (via self.log_obj).&nbsp; self.run_command() and self.get_output_from_command() allow the script to call other executables easily, while keeping information in the logs.</span></div>
      <div id="magicdomid46">
        &nbsp;</div>
      <div id="magicdomid47">
        <span class="author-g-ijx5rjfb8o6e7gv5">BaseScript.run() makes the script go.</span></div>
    </dd>
  </dl>
  <dl>
    <dt id="magicdomid49">
      <span class="author-g-ijx5rjfb8o6e7gv5">mozharness.base.config.BaseConfig</span></dt>
    <dd>
      <span class="author-g-ijx5rjfb8o6e7gv5">This combines the initial script config, the config file, and the command line options into a single config, which becomes the BaseScript self.config.&nbsp; This config is locked (via ReadOnlyDict) at the end of BaseScript.__init__() and dumped to the logs for ease of debugging and more predictable behavior.</span></dd>
  </dl>
  <dl>
    <dt id="magicdomid53">
      <span class="author-g-ijx5rjfb8o6e7gv5">mozharness.base.log.LogMixin</span></dt>
    <dd>
      <div id="magicdomid55">
        <span class="author-g-ijx5rjfb8o6e7gv5">Almost every mozharness object inherits this mixin at some level, to allow for the various log-level methods.</span></div>
      <div id="magicdomid56">
        &nbsp;</div>
      <div id="magicdomid57">
        <span class="author-g-ijx5rjfb8o6e7gv5">If self.log_obj isn't set, we fall back to print's.&nbsp; If it is set to a subclass of BaseLogger (or otherwise compatible logging object that has a log_message() method), then we log through that object.</span></div>
      <div id="magicdomid58">
        &nbsp;</div>
      <div id="magicdomid59">
        <span class="author-g-ijx5rjfb8o6e7gv5">By default, BaseScript uses MultiFileLogger as its log_obj, which creates a file per logging level.</span></div>
    </dd>
  </dl>
  <div>
    &nbsp;</div>
  <div id="magicdomid61">
    <span class="author-g-ijx5rjfb8o6e7gv5">There are also a number of other classes and mixins that provide useful functionality but aren't required for all script/job types.&nbsp; This semi-complete list will most likely fall out of date at some point:</span></div>
  <div>
    &nbsp;</div>
  <ul>
    <li id="magicdomid63"><span class="author-g-ijx5rjfb8o6e7gv5">mozharness.base.log.OutputParser - parses for errors from BaseScript.run_command()</span></li>
    <li id="magicdomid64"><span class="author-g-ijx5rjfb8o6e7gv5">mozharness.base.parallel.ChunkingMixin - split a long list of items up into separate jobs.</span></li>
    <li id="magicdomid65"><span class="author-g-ijx5rjfb8o6e7gv5">mozharness.base.signing.AndroidSigningMixin - Android signing</span></li>
    <li id="magicdomid66"><span class="author-g-ijx5rjfb8o6e7gv5">mozharness.base.transfer.TransferMixin - uploading/downloading directories</span></li>
    <li id="magicdomid67"><span class="author-g-ijx5rjfb8o6e7gv5">mozharness.base.vcs.vcsbase.VCSScript - provides mercurial functionality.&nbsp; More VCS support as they're added.</span></li>
    <li id="magicdomid68"><span class="author-g-ijx5rjfb8o6e7gv5">mozharness.mozilla.buildbot.BuildbotMixin - interface with buildbot and mozilla-specific buildbot setup</span></li>
    <li id="magicdomid69"><span class="author-g-ijx5rjfb8o6e7gv5">mozharness.mozilla.l10n.locales.LocalesMixin - localization support</span></li>
    <li id="magicdomid70"><span class="author-g-ijx5rjfb8o6e7gv5">mozharness.mozilla.l10n.multi_locale_build.MultiLocaleBuild</span></li>
    <li id="magicdomid71"><span class="author-g-ijx5rjfb8o6e7gv5">mozharness.mozilla.mock.MockMixin - mock_mozilla support</span></li>
    <li id="magicdomid72"><span class="author-g-ijx5rjfb8o6e7gv5">mozharness.mozilla.release.ReleaseMixin and SigningMixin - release automation support</span></li>
    <li id="magicdomid73"><span class="author-g-ijx5rjfb8o6e7gv5">mozharness.mozilla.testing.device.DeviceMixin - Android adb + sut device controlling support</span></li>
    <li id="magicdomid74"><span class="author-g-ijx5rjfb8o6e7gv5">mozharness.mozilla.testing.talos.Talos - talos support</span></li>
    <li id="magicdomid75"><span class="author-g-ijx5rjfb8o6e7gv5">mozharness.mozilla.testing.testbase.TestingMixin - unittest and talos support</span></li>
    <li id="magicdomid76"><span class="author-g-ijx5rjfb8o6e7gv5">mozharness.mozilla.tooltool.TooltoolMixin - tooltool support</span></li>
  </ul>
</div>
<h4 id="Q._What_classes_of_mozharness_scripts_are_there_(or_should_there_be).3F.C2.A0_Tests.2C_tools.2C_etc.3F"><span class="author-g-gmeqsyoivrkikiba">Q. What classes of mozharness scripts are there (or should there be)?&nbsp; Tests, tools, etc?</span></h4>
<p><span class="author-g-gmeqsyoivrkikiba">A. </span><span class="author-g-ijx5rjfb8o6e7gv5">Right now mozharness is slated to replace buildbotcustom as MoCo releng's source of automation logic.</span></p>
<div id="magicdomid83">
  <span class="author-g-ijx5rjfb8o6e7gv5">There are/will be scripts in mozharness for builds, tests, repacks, releases.&nbsp; We may also put machine maintenance type scripts here; anything we want run on releng build+test farms.&nbsp; At some point we will just be able to consider these compute farms, and mozharness will be what runs on that compute farm.</span></div>
<div id="magicdomid84">
  &nbsp;</div>
<div id="magicdomid85">
  <span class="author-g-ijx5rjfb8o6e7gv5">Since mozharness is just a python harness, almost anything that can be written in Python could potentially be added to mozharness.</span></div>
<div>
  &nbsp;</div>
<h4 id="Q._I_want_to_write_a_mozharness_script_for_a_new_automation_problem..C2.A0.C2.A0_How_do_I_do_this.3F">Q. <span class="author-g-gmeqsyoivrkikiba">I want to write a mozharness script for a new automation problem.&nbsp;&nbsp; </span>How do I do this?</h4>
<div id="magicdomid90">
  <span class="author-g-ijx5rjfb8o6e7gv5">For your first mozharness script, it's probably easiest to use existing, similar scripts as examples; that's definitely easier than writing an entirely new type of script.</span></div>
<div id="magicdomid91">
  <p><span class="author-g-ijx5rjfb8o6e7gv5">Split your script into logical chunks, or actions:</span></p>
</div>
<ul>
  <li id="magicdomid93"><span class="author-g-ijx5rjfb8o6e7gv5">What actions are needed by some people or jobs, but not others?</span></li>
  <li id="magicdomid94"><span class="author-g-ijx5rjfb8o6e7gv5">What actions could potentially be independent of others?&nbsp; For example, installing Firefox&nbsp; to run tests.&nbsp; You don't necessarily have to do this if you point to a&nbsp; pre-installed Firefox to run those tests.&nbsp; Or packaging after building.&nbsp; Or uploading the logs somewhere.</span></li>
  <li id="magicdomid95"><span class="author-g-ijx5rjfb8o6e7gv5">Since actions can be run by themselves, or skipped, they're a logical place to split up a script's behavior.</span></li>
  <li><span class="author-g-ijx5rjfb8o6e7gv5">These will be listed in the script's all_actions.&nbsp; A default set of actions, if different from all_actions, is defined in default_actions.&nbsp; But the user can turn any or all actions on or off via command line options or config file.</span></li>
  <li id="magicdomid97"><span class="author-g-ijx5rjfb8o6e7gv5">You need to have a method per action.&nbsp; This method will be named the same as the action, with underscores replacing dashes.</span></li>
  <li id="magicdomid98"><span class="author-g-ijx5rjfb8o6e7gv5">If you want to define pre-action or post-action behavior, you can write optional preflight_actionname() and/or postflight_actionname() methods, again with underscores instead of dashes.</span></li>
</ul>
<div id="magicdomid99">
  &nbsp;</div>
<div id="magicdomid100">
  <span class="author-g-ijx5rjfb8o6e7gv5">Figure out the different configurations that might be in play; these should be pre-defineable in either command line options or config files.&nbsp; (Complex data structures are best suited in config files; commonly overridden options should be available via command line.)</span></div>
<div>
  &nbsp;</div>
<h3 id="Coding">Coding</h3>
<h4 id="Q._What_are_all_the_possible_mozharness_actions.2C_and_what_dependencies_exist_between_them.3F">Q. <span class="author-g-gmeqsyoivrkikiba">What are all the possible mozharness actions, and what dependencies exist between them?</span></h4>
<p>A. <span class="author-g-ijx5rjfb8o6e7gv5">This is very much script- or script-family- specific.&nbsp; Any mozharness script can define any arbitrary list of all_actions.&nbsp; Each action has a matching method of the same name (after substituting underscores for dashes).</span></p>
<div id="magicdomid107">
  &nbsp;</div>
<div id="magicdomid108">
  <span class="author-g-ijx5rjfb8o6e7gv5">We attempt to make the actions standalone, so you could potentially run action #4 over and over without having to run actions 1-3 over and over as well.&nbsp; However, action #4 probably needs actions 1-3 to have run at some point (clone repos, download installers, etc.), or for the user to have manually done those steps beforehand.</span></div>
<div id="magicdomid109">
  &nbsp;</div>
<div id="magicdomid110">
  <span class="author-g-ijx5rjfb8o6e7gv5">If you're asking what actions have been defined in existing scripts, a&nbsp;</span></div>
<div id="magicdomid111">
  <span class="author-g-ijx5rjfb8o6e7gv5 b"><b>&nbsp;&nbsp;&nbsp; grep -r 'all_actions\s*=' .</b></span></div>
<div id="magicdomid112">
  <span class="author-g-ijx5rjfb8o6e7gv5">might be a good start.&nbsp; This will change, so a static list here will be obsoleted.</span></div>
<div id="magicdomid113">
  &nbsp;</div>
<div id="magicdomid114">
  <span class="author-g-ijx5rjfb8o6e7gv5">Some actions with pre-written methods inside of mozharness.base.* or mozharness.mozilla.* include:</span></div>
<div id="magicdomid115">
  &nbsp;</div>
<div id="magicdomid116">
  <span class="author-g-ijx5rjfb8o6e7gv5"># clobber</span></div>
<div id="magicdomid117">
  <span class="author-g-ijx5rjfb8o6e7gv5"># pull</span></div>
<div id="magicdomid118">
  <span class="author-g-ijx5rjfb8o6e7gv5"># list-locales (l10n)</span></div>
<div id="magicdomid119">
  <span class="author-g-ijx5rjfb8o6e7gv5"># pull-locale-source (l10n)</span></div>
<div id="magicdomid120">
  <span class="author-g-ijx5rjfb8o6e7gv5"># passphrase (android signing)</span></div>
<div id="magicdomid121">
  <span class="author-g-ijx5rjfb8o6e7gv5"># create-virtualenv (python)</span></div>
<div id="magicdomid122">
  <span class="author-g-ijx5rjfb8o6e7gv5"># read-buildbot-config (buildbot interface)</span></div>
<div id="magicdomid123">
  <span class="author-g-ijx5rjfb8o6e7gv5"># download-and-extract (tests)</span></div>
<div id="magicdomid124">
  <span class="author-g-ijx5rjfb8o6e7gv5"># install (tests)</span></div>
<div id="magicdomid125">
  &nbsp;</div>
<div id="magicdomid126">
  <span class="author-g-ijx5rjfb8o6e7gv5">Perhaps a mozharness- level document like this won't be as useful for questions like this, as much as script-family level documentation.&nbsp; For instance, "What actions do TestMixin- (or LocalesMixin-, or ...?) based mozharness scripts have defined already and what do they do?"</span></div>
<div>
  &nbsp;</div>
<h4 id="Q._I_want_to_use_a_python_library_in_a_mozharness_script..C2.A0_How_can_I_do_that.3F.C2.A0">Q. <span class="author-g-gmeqsyoivrkikiba">I want to use a python library in a mozharness script.&nbsp; How can I do that?&nbsp;</span></h4>
<div id="magicdomid130">
  <span class="author-g-ijx5rjfb8o6e7gv5">If it's a python built-in, you can import as normal.&nbsp; We currently support python 2.5.1 -&gt; 2.7.x, but we're moving towards having python 2.7.3 everywhere.</span></div>
<div id="magicdomid131">
  &nbsp;</div>
<div id="magicdomid132">
  <span class="author-g-ijx5rjfb8o6e7gv5">If&nbsp; it's an external package, you most likely need mozharness to install it&nbsp; into your virtualenv (see PythonMixin.create_virtualenv()).</span></div>
<div id="magicdomid133">
  &nbsp;</div>
<div id="magicdomid134">
  <span class="author-g-ijx5rjfb8o6e7gv5">If there's a commandline tool that's also installed (say, mozinstall), you can call it from self.run_command().&nbsp; For instance, TestMixin.install() calls mozinstall like this:&nbsp; </span><span class="author-g-ijx5rjfb8o6e7gv5 url"><a href="http://hg.mozilla.org/build/mozharness/file/3334bfde4eed/mozharness/mozilla/testing/testbase.py#l229">http://hg.mozilla.org/build/mozharness/file/3334bfde4eed/mozharness/mozilla/testing/testbase.py#l229</a></span><span class="author-g-ijx5rjfb8o6e7gv5"> with a preflight check to make sure mozinstall is in the virtualenv, first: </span><span class="author-g-ijx5rjfb8o6e7gv5 url"><a href="http://hg.mozilla.org/build/mozharness/file/3334bfde4eed/mozharness/mozilla/testing/testbase.py#l213">http://hg.mozilla.org/build/mozharness/file/3334bfde4eed/mozharness/mozilla/testing/testbase.py#l213</a></span></div>
<div id="magicdomid135">
  &nbsp;</div>
<div id="magicdomid136">
  <span class="author-g-ijx5rjfb8o6e7gv5">If the external package doesn't have a commandline tool, you need to import it.&nbsp; However, since the module won't be available at the beginning of the script run, a bare |import module| will throw an exception before you create the virtualenv.</span></div>
<div>
  &nbsp;</div>
<div>
  <div id="magicdomid140">
    <span class="author-g-ijx5rjfb8o6e7gv5">I solved this in SUTDeviceHandler by importing the module in a method that isn't called until after the virtualenv is created.&nbsp; </span><span class="author-g-ijx5rjfb8o6e7gv5 url"><a href="http://hg.mozilla.org/build/mozharness/file/b8104340b600/mozharness/mozilla/testing/device.py#l444">http://hg.mozilla.org/build/mozharness/file/b8104340b600/mozharness/mozilla/testing/device.py#l444</a></span></div>
  <div>
    &nbsp;</div>
</div>
<h4 id="Q._I_want_to_use_a_python_library_in_a_patch_I'd_like_to_contribute_to_the_core_mozharness_classes..C2.A0_How_can_I_do_that.3F">Q. <span class="author-g-gmeqsyoivrkikiba">I want to use a python library in a patch I'd like to contribute to the core mozharness classes.&nbsp; How can I do that?</span></h4>
<p>A. <span class="author-g-ijx5rjfb8o6e7gv5">Same as above.&nbsp; A core mozharness class will usually need to handle things in a more generic way than standalone scripts, but that's true of most core classes.</span></p>
<h4 id="Q._I've_just_finished_my_mozharness_script.2C_and_think_it_should_live_inside_mozilla-central._Is_this_possible.3F">Q. <span class="author-g-gmeqsyoivrkikiba">I've just finished my mozharness script, and think it should live inside mozilla-central. Is this possible?</span></h4>
<p>A. <span class="author-g-ijx5rjfb8o6e7gv5">It's certainly possible, with the following caveats:</span></p>
<div id="magicdomid150">
  &nbsp;</div>
<ul>
  <li id="magicdomid151"><span class="author-g-ijx5rjfb8o6e7gv5">if the script lives in mozilla-central, it still (probably) depends on mozharness.base.* and mozharness.mozilla.* .&nbsp; There needs to be some way to support that, whether it's cloning mozharness or checking a copy into m-c and maintaining that, pushing up to pypi, or some other approach.</span></li>
  <li id="magicdomid152"><span class="author-g-ijx5rjfb8o6e7gv5">as of this writing, mozharness is undergoing constant change.&nbsp; anything checked into mozilla-central or pushed to pypi would probably go out-of-date fairly quickly, and would need regular code drops.</span></li>
  <li id="magicdomid153"><span class="author-g-ijx5rjfb8o6e7gv5">as described below, production mozharness jobs clone hg.m.o/build/mozharness, then run the script.&nbsp; If the script doesn't live in mozharness, we would have to alter this process.&nbsp; There would need to be a strong reason to do this.</span></li>
</ul>
<div>
  &nbsp;</div>
<div id="magicdomid154">
  <span class="author-g-ijx5rjfb8o6e7gv5">This is complicated by the fact that only a subset of mozharness jobs require the tree, and most of those require a specific branch or revision in production.&nbsp; If the scripts and logic to do this lived in the tree, we would either need external logic to do this, or have the script update itself to a potentially conflicting version, and reload itself mid-script.</span></div>
<div>
  &nbsp;</div>
<h3 id="Configuration">Configuration</h3>
<h4 id="Q._What_should.2Fcan_I_put_in_a_production_configuration_file.3F.C2.A0_A_test_configuration_file.3F">Q. <span class="author-g-gmeqsyoivrkikiba">What should/can I put in a production configuration file?&nbsp; A </span><span class="author-g-gmeqsyoivrkikiba">test configuration file?</span></h4>
<p>A. <span class="author-g-ijx5rjfb8o6e7gv5">This actually is fairly script-dependent.</span></p>
<div id="magicdomid167">
  <span class="author-g-ijx5rjfb8o6e7gv5">If we were to run, say, </span><span class="author-g-ijx5rjfb8o6e7gv5 url"><a href="http://hg.mozilla.org/build/mozharness/file/bb6851655e8d/scripts/configtest.py">http://hg.mozilla.org/build/mozharness/file/bb6851655e8d/scripts/configtest.py</a></span><span class="author-g-ijx5rjfb8o6e7gv5"> configtest.py in production, we wouldn't necessarily need a config file at all.&nbsp; We'd just need a python2 &gt;=2.5 with json or simplejson support.</span></div>
<div id="magicdomid168">
  &nbsp;</div>
<div id="magicdomid169">
  <span class="author-g-ijx5rjfb8o6e7gv5">As you add functionality, the requirements grow.</span></div>
<div id="magicdomid170">
  &nbsp;</div>
<div id="magicdomid171">
  <span class="author-g-ijx5rjfb8o6e7gv5">If you need to upload:</span></div>
<ul>
  <li id="magicdomid172"><span class="author-g-ijx5rjfb8o6e7gv5">Currently this is only doable on the build pool.</span></li>
  <li id="magicdomid173"><span class="author-g-ijx5rjfb8o6e7gv5">The settings needed vary depending on try/non-try, project, etc.; this is probably best for questions or a longer external document.</span></li>
</ul>
<div id="magicdomid174">
  &nbsp;</div>
<div id="magicdomid175">
  <span class="author-g-ijx5rjfb8o6e7gv5">If you need to create a virtualenv:</span></div>
<ul>
  <li id="magicdomid176"><span class="author-g-ijx5rjfb8o6e7gv5">You need to specify virtualenv_modules.</span></li>
  <li id="magicdomid177"><span class="author-g-ijx5rjfb8o6e7gv5">We can't rely on pypi being up, and will be blocking access to the internet from the production pool, so specify a find_links list of internal servers like this: </span><span class="author-g-ijx5rjfb8o6e7gv5 url"><a href="http://hg.mozilla.org/build/mozharness/file/bb6851655e8d/configs/talos/linux_config.py#l13">http://hg.mozilla.org/build/mozharness/file/bb6851655e8d/configs/talos/linux_config.py#l13</a></span>
    <ul>
      <li id="magicdomid178"><span class="author-g-ijx5rjfb8o6e7gv5">Currently also specify the pypi_url to an internal url, until </span><span class="author-g-ijx5rjfb8o6e7gv5 url"><a href="https://bugzil.la/795153">https://bugzil.la/795153</a></span><span class="author-g-ijx5rjfb8o6e7gv5"> is fixed.</span></li>
    </ul>
  </li>
  <li id="magicdomid179"><span class="author-g-ijx5rjfb8o6e7gv5">Python and virtualenv aren't in PATH for the test pool.&nbsp; You can specify these in your 'exes' definition in your config files.</span>
    <ul>
      <li id="magicdomid180"><span class="author-g-ijx5rjfb8o6e7gv5">For the mac and linux test pools, python is /tools/buildbot/bin/python; virtualenv is&nbsp;</span><span class="author-g-ijx5rjfb8o6e7gv5">['/tools/buildbot/bin/python', '/tools/misc-python/virtualenv.py']</span></li>
      <li><span class="author-g-ijx5rjfb8o6e7gv5">For the windows test pool, python is c:/mozilla-build/python27/python and virtualenv is [''c:/mozilla-build/python27/python', 'c:/mozilla-build/buildbotve/virtualenv.py']</span></li>
      <li><span class="author-g-ijx5rjfb8o6e7gv5">You also need to specify virtualenv_python_dll as 'c:/mozilla-build/python27/python27.dll' due to </span><span class="author-g-ijx5rjfb8o6e7gv5 url"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=700415#c50">https://bugzilla.mozilla.org/show_bug.cgi?id=700415#c50</a></span><span class="author-g-ijx5rjfb8o6e7gv5"> and </span><span class="author-g-ijx5rjfb8o6e7gv5 url"><a href="https://bugs.launchpad.net/virtualenv/+bug/352844/comments/3">https://bugs.launchpad.net/virtualenv/+bug/352844/comments/3</a></span></li>
      <li><span class="author-g-ijx5rjfb8o6e7gv5">We probably want to have a fixed python install everywhere, but this works around this issue.</span></li>
      <li><span class="author-g-ijx5rjfb8o6e7gv5">you may need to specify the distribute_url and pip_url:</span>
        <ul>
          <li><span class="author-g-ijx5rjfb8o6e7gv5">"distribute_url": "</span><span class="author-g-ijx5rjfb8o6e7gv5 url"><a href="http://puppetagain.pub.build.mozilla.org/data/python/packages/distribute-0.6.26.tar.gz">http://puppetagain.pub.build.mozilla.org/data/python/packages/distribute-0.6.26.tar.gz</a></span><span class="author-g-ijx5rjfb8o6e7gv5">"</span></li>
          <li><span class="author-g-ijx5rjfb8o6e7gv5">"pip_url": "</span><span class="author-g-ijx5rjfb8o6e7gv5 url"><a href="http://puppetagain.pub.build.mozilla.org/data/python/packages/pip-0.8.2.tar.gz">http://puppetagain.pub.build.mozilla.org/data/python/packages/pip-0.8.2.tar.gz</a></span><span class="author-g-ijx5rjfb8o6e7gv5">"</span></li>
        </ul>
      </li>
      <li><span class="author-g-ijx5rjfb8o6e7gv5">If you require pywin32, you need to add it to your virtualenv_modules (for windows only), and define easy_install like here: </span><span class="author-g-ijx5rjfb8o6e7gv5 url"><a href="http://hg.mozilla.org/build/mozharness/file/bb6851655e8d/configs/talos/windows_config.py#l24">http://hg.mozilla.org/build/mozharness/file/bb6851655e8d/configs/talos/windows_config.py#l24</a></span></li>
      <li><span class="author-g-ijx5rjfb8o6e7gv5">If you're using mozinstall to install Firefox, you need to define mozinstall like here: </span><span class="author-g-ijx5rjfb8o6e7gv5 url"><a href="http://hg.mozilla.org/build/mozharness/file/bb6851655e8d/configs/talos/windows_config.py#l26">http://hg.mozilla.org/build/mozharness/file/bb6851655e8d/configs/talos/windows_config.py#l26</a></span>
        <ul>
          <li><span class="author-g-ijx5rjfb8o6e7gv5">Essentially, windows security frowns on running anything with 'install' in the name.&nbsp; Calling 'python' with mozinstall or easy_install as an argument works around this issue.</span></li>
        </ul>
      </li>
    </ul>
  </li>
  <li><span class="author-g-ijx5rjfb8o6e7gv5">The build pool is probably somewhat different; we should update this list when we solve virtualenv on the build pool.</span></li>
</ul>
<div id="magicdomid192">
  &nbsp;</div>
<div id="magicdomid193">
  <span class="author-g-ijx5rjfb8o6e7gv5">If your production script will be running a different set of actions than developers:</span></div>
<ul>
  <li id="magicdomid194"><span class="author-g-ijx5rjfb8o6e7gv5">Define a superset of all_actions</span></li>
  <li id="magicdomid195"><span class="author-g-ijx5rjfb8o6e7gv5">Define your in-script default_actions as the set that someone would use on the command line (developer centric)</span></li>
  <li id="magicdomid196"><span class="author-g-ijx5rjfb8o6e7gv5">Define a set of production config files that overrides default_actions for the set of actions pertinent to production.</span></li>
</ul>
<div id="magicdomid197">
  &nbsp;</div>
<div id="magicdomid198">
  <span class="author-g-ijx5rjfb8o6e7gv5">If you're running on the test pool or build pool:</span></div>
<ul>
  <li id="magicdomid199"><span class="author-g-ijx5rjfb8o6e7gv5">There are large differences between the test pool and build pool (and try pool), as the machines are configured differently.</span></li>
  <li id="magicdomid200"><span class="author-g-ijx5rjfb8o6e7gv5">We need to document these changes more to allow for easier usage.</span></li>
</ul>
<div id="magicdomid201">
  &nbsp;</div>
<div id="magicdomid202">
  <span class="author-g-ijx5rjfb8o6e7gv5">If the script needs mercurial:</span></div>
<ul>
  <li id="magicdomid203"><span class="author-g-ijx5rjfb8o6e7gv5">It needs to inherit MercurialScript</span></li>
  <li id="magicdomid204"><span class="author-g-ijx5rjfb8o6e7gv5">You can define a list of 'repos' like </span><span class="author-g-ijx5rjfb8o6e7gv5 url"><a href="http://hg.mozilla.org/build/mozharness/file/bb6851655e8d/configs/multi_locale/mozilla-central_android.json#l8">http://hg.mozilla.org/build/mozharness/file/bb6851655e8d/configs/multi_locale/mozilla-central_android.json#l8</a></span><span class="author-g-ijx5rjfb8o6e7gv5"> , which will get pulled when VCSScript.pull() is called (usually as an action).</span></li>
  <li id="magicdomid205"><span class="author-g-ijx5rjfb8o6e7gv5">If it needs to run on Windows, set your exes['hg'] to 'c:/mozilla-build/hg/hg' for windows.&nbsp; This is because hg isn't in the default path in the build and test pools.</span></li>
</ul>
<div id="magicdomid206">
  &nbsp;</div>
<div id="magicdomid207">
  <span class="author-g-ijx5rjfb8o6e7gv5">If your script will be run in ScriptFactory:</span></div>
<ul>
  <li id="magicdomid209"><span class="author-g-ijx5rjfb8o6e7gv5">it needs to inherit BuildbotMixin</span></li>
  <li><span class="author-g-ijx5rjfb8o6e7gv5">it needs to run self.read_buildbot_config() at some point before referring to revision, branch, installer_url, or test_url.&nbsp; We've been doing this by adding a non-default read-buildbot-config action.</span>
    <ul>
      <li id="magicdomid210"><span class="author-g-ijx5rjfb8o6e7gv5">This will populate self.buildbot_config, which you can refer to.</span></li>
    </ul>
  </li>
  <li id="magicdomid211"><span class="author-g-ijx5rjfb8o6e7gv5">the buildbot_json_path needs to be set to "buildprops.json".</span></li>
</ul>
<div id="magicdomid213">
  <span class="author-g-ijx5rjfb8o6e7gv5">These lists may not be complete and may change, but we can update this documentation over time.</span></div>
<div>
  &nbsp;</div>
<h4>Q. <span class="author-g-gmeqsyoivrkikiba">How do I inherit from one configuration file in another one?</span></h4>
<p><span class="author-g-gmeqsyoivrkikiba">A. </span><span class="author-g-ijx5rjfb8o6e7gv5">Currently you can't, and I'd like to discourage that approach.</span></p>
<div id="magicdomid218">
  &nbsp;</div>
<div id="magicdomid219">
  <span class="author-g-ijx5rjfb8o6e7gv5">The&nbsp; upside of flat config files is it's easy to annotate/blame/diff to see&nbsp; exactly who changed what, when.&nbsp; This is highly useful.&nbsp; The downside is&nbsp; there are more files to update when changing something that's shared&nbsp; amongst multiple config files.</span></div>
<div id="magicdomid220">
  &nbsp;</div>
<div id="magicdomid221">
  <span class="author-g-ijx5rjfb8o6e7gv5">Inheriting/including/programmatically&nbsp; creating config has the reverse upside/downside.&nbsp; However, RelEng has&nbsp; gone so far down this route that we're forever trying to figure out what&nbsp; will change if we pull *this* string or who changed what, when, that&nbsp; affected this seemingly unrelated thing.&nbsp; We're moving towards&nbsp; dynamically creating flat configs (and checking those flat configs into&nbsp; revision control), which should hopefully give us a combination of&nbsp; upsides with fewer downsides.</span></div>
<div id="magicdomid222">
  &nbsp;</div>
<div id="magicdomid223">
  <span class="author-g-ijx5rjfb8o6e7gv5">If&nbsp; we have to choose one upside/downside combination, it's my (Aki's)&nbsp; strong belief that optimizing for debugging + reading (read:&nbsp; diff/blame/annotate) is more important than optimizing for writing,&nbsp; since we're going to read these configs a lot more than we're going to&nbsp; write them.</span></div>
<div id="magicdomid224">
  &nbsp;</div>
<div id="magicdomid225">
  <span class="author-g-ijx5rjfb8o6e7gv5">However, when </span><span class="author-g-ijx5rjfb8o6e7gv5 url"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=779294">https://bugzilla.mozilla.org/show_bug.cgi?id=779294</a></span><span class="author-g-ijx5rjfb8o6e7gv5"> is resolved, you should be able to specify multiple config files on the command line, and there will be rules as to what to do with conflicting config entries.&nbsp; Hopefully this will allow us to easily override things in certain cases, without potentially duplicating large swaths of configs in dozens of files.</span></div>
<div id="magicdomid226">
  &nbsp;</div>
<div id="magicdomid227">
  <span class="author-g-ijx5rjfb8o6e7gv5">Also, I don't mind having tools to easily compare config files, or build new ones programmatically, as long as the final flat configs are checked in for easy reading, diff, annotate, blame, etc.</span></div>
<div>
  &nbsp;</div>
<h4>Q. <span class="author-g-gmeqsyoivrkikiba">Is there a 'master' production configuration file that I can use?</span></h4>
<p><span class="author-g-gmeqsyoivrkikiba">A</span>. <span class="author-g-ijx5rjfb8o6e7gv5">No, and this doesn't necessarily make sense when you consider that "production configuration file" could refer to an Android multilocale build, Android signing, an l10n repack, a b2g build, a Firefox windows desktop unittest, or any other of a number of wildly disparate tasks.</span></p>
<div id="magicdomid232">
  &nbsp;</div>
<div id="magicdomid233">
  <span class="author-g-ijx5rjfb8o6e7gv5">There isn't a lot in common between </span><span class="author-g-ijx5rjfb8o6e7gv5 url"><a href="http://hg.mozilla.org/build/mozharness/file/b8104340b600/configs/marionette/windows_config.py">http://hg.mozilla.org/build/mozharness/file/b8104340b600/configs/marionette/windows_config.py</a></span><span class="author-g-ijx5rjfb8o6e7gv5"> and </span><span class="author-g-ijx5rjfb8o6e7gv5 url"><a href="http://hg.mozilla.org/build/mozharness/file/b8104340b600/configs/multi_locale/release_mozilla-release_android.json">http://hg.mozilla.org/build/mozharness/file/b8104340b600/configs/multi_locale/release_mozilla-release_android.json</a></span><span class="author-g-ijx5rjfb8o6e7gv5"> , for example.</span></div>
<div id="magicdomid234">
  &nbsp;</div>
<div id="magicdomid235">
  <span class="author-g-ijx5rjfb8o6e7gv5">We are starting to see "families" of scripts forming that do have a significant amount of overlap, however.&nbsp; We can consolidate, or share, or logically split these up to take advantage of the multiple-config-files in bug 779294 when that's done.&nbsp; We haven't tried to do this at the outset until obvious patterns and use cases emerged, but once those are apparent people can spend time to make this easier or more logical.</span></div>
Revert to this revision