Creating and applying Firefox OS update packages

If you want to allow Firefox OS users to easily update the version of the system code on their devices, you need to create an update package for them to use. This article goes through the different types of update package available and covers building the package, hosting the updates (and how the system polls for available updates), and applying and verifying those updates.

Creating an applying an update is split into four steps:

  1. Building incremental update packages from old version(s) to a new version on a build host
  2. Finding the right update package to download on the client
  3. Downloading the update
  4. Applying the update to existing files on the device

Each of these steps are covered below.

Types of update

There are two types of updates to know about: FOTA (Firmware Over-The-Air) and Gecko/Gaia OTA (Over-The-Air). Let's look at the differences between them.

FOTA updates

We can update the entire Firfox OS system through FOTA updates, the technology behind which is shared with the Android project. The locations on the phone's hard drive that can be changed using FOTA updates include the system partition, kernel, modem baseband, recovery image used for updating, or any other file on the device.

Firefox OS does not depend on a particular FOTA client; the interface is abstracted through an API we call librecovery. However, we recommend using the GOTA recovery client (see below for more details), and the discussion here assumes FOTA is being used.

FOTA update packages mainly consist of a file called update.zip. This package consists of

  • A set of binary diffs and new files required to update the client to the newer software version
  • An "update script" that controls how the diffs and new files are loaded onto the client
  • An internal signature used to verify the update package

This format and set of files are the same as those used in normal Android updates, except that Firefox OS additionally wraps the update.zip package in a mar wrapper (MAR stands for Mozilla ARchive). This mar wrapper allows an additional level of verification, which is explained below.

Gecko/Gaia OTA updates

Alteratively we can update just the Gecko and Gaia files on a Firefox OS device, through a mechanism we call Gecko/Gaia OTA updates. All of the Gecko and Gaia files — comprising the core Gecko runtime and the device's user interface — are in the /system/b2g directory on the device. This is the only directory that OTA updates can make changes to.

Gecko/Gaia OTA updates use the same technology that's used to update the Firefox desktop web browser. Much like the FOTA update.zip packages discussed above, OTA updates consist of a MAR file containing a set of binary diffs and new files needed to update the client to a newer software version.

The Gecko client verifies the integrity of MARs that it downloads, and MARs can be signed by multiple parties.

Why have two update technologies?

OTA updates are not as comprehensive as FOTA updates, but they are a lot more user friendly and easy to apply, and will often be fine for what you need to update:

  • Gecko/Gaia OTA updates can be applied "in the background", while Firefox OS continues to run normally. This provides a much better user experience because users don't need to reboot their phone and wait while an update is applied. Instead, the update is applied while the user continues to use the phone, and when the update is finished the user only needs to agree to restart the main b2g process. This takes a matter of seconds, instead of the minutes that are usually required to apply FOTA updates.
  • Gecko/Gaia OTA update packages can sometimes be smaller than FOTA update packages, though not always; they should never be larger. This means that users can sometimes have less data to download.

Of course, if you need to update files outside Gecko/Gaia, you will have to go for the full FOTA package route.

Let's move on and examine the package building process.

Building update packages

Building updates is the process of generating the files needed to update Firefox OS clients from version X of the software to a newer version Y. The update package that's needed to update the client depends on what files have changed between version X and version Y.

  • If only files in /system/b2g have changed, we will generate a Gecko/Gaia OTA update
  • If any file in a location outside of /system/b2g changed, we will generate a FOTA update

To generate an incremental update package (for both FOTA and Gecko/Gaia OTA updates), our tools require full builds of both version X and version Y. Full build means that the package includes all the files that are needed to Flash a client. When we produce a full build for version X, we don't know which future versions we will be updating to from version X . Because of that, we build both full FOTA packages and Gecko/Gaia packages for each version. This allows us to generate either a Gecko/Gaia OTA incremental update, or a FOTA incremental update if needed, between version X and all future versions.

At a high level, the process of building an update looks like this:

  1. With software version X
    • Generate a complete Gecko/Gaia OTA MAR for the contents of /system/b2g.
    • Generate a complete FOTA update.zip and target files zip for the device partitions.
  2. With software version Y
    • Generate a complete Gecko/Gaia OTA MAR for the contents of /system/b2g.
    • Generate a complete FOTA update.zip and target files zip for the device partitions.
  3. If only files in /system/b2g have changed, generate an incremental Gecko/Gaia OTA update MAR from version X to version Y.
  4. Otherwise, generate an incremental FOTA update.zip from version X to version Y. Wrap the incremental FOTA update.zip in a MAR for delivery to the B2G client.
  5. Sign the packages as required by delivery agreements.

The subsections below describe how to use B2G's tools to implement each of these steps.

Note: the steps below assume that you have already set up a b2g build environment at the location $b2g. The commands below reference the $b2g/build.sh helper script, but make can also be used.

Generating a complete Gecko/Gaia OTA update MAR

Invoke the gecko-update-full target to generate a complete update MAR from the last successful b2g build. To place the MAR at $b2g/objdir-gecko/dist/b2g-update/b2g-gecko-update.mar, use the following commands:

$ cd $b2g
$ ./build.sh gecko-update-full
$ cp objdir-gecko/dist/b2g-update/b2g-gecko-update.mar <destination>

Generating a complete FOTA update zip and target files zip

The default target in the b2g build system will generate a FOTA update.zip / target files zip when the kernel binary has been copied to the appropriate location under vendor/. This enables boot image, recovery image, and update.zip generation.

  • The complete FOTA update.zip is generated in the location out/target/product/$DEVICE/$DEVICE-ota-$VARIANT.$USER.zip
  • The target files zip is generated to the location out/target/product/$DEVICE/obj/PACKAGING/target_files_intermediates/$DEVICE-target_files-$VARIANT.$USER.zip

The following commands will carry out this step:

$ cd $b2g
$ ./build.sh
$ cp out/target/product/$DEVICE/obj/PACKAGING/target_files_intermediates/$DEVICE-target_files-$VARIANT.$USER.zip <destination>

The variable values in the commands listed above should be filled in as follows:

Variable Meaning
$DEVICE Device name for the AOSP product
$VARIANT eng, user, or userdebug
$USER The build username

Generating an incremental OTA update MAR

In this example, we're assuming that we're generating an update from software version X to version Y. The location of the full Gecko/Gaia OTA MAR built from software version X using the instructions above will be called $MAR_X below. This might be a path on a build server like /home/build/b2g/versions/X/update.mar. Similarly, the location of the full MAR built from version Y will be called $MAR_Y.

The tool build-gecko-mar.py will generate an incremental Gecko/Gaia OTA update MAR using $MAR_X and $MAR_Y. We'll call the destination of the generated file $GENERATED_INCREMENTAL_MAR_X_Y. Use the following commands for this step:

$ cd $b2g
$ ./tools/update-tools/build-gecko-mar.py --from $MAR_X --to $MAR_Y $GENERATED_INCREMENTAL_MAR_X_Y

Generating an incremental FOTA update zip

In this example, we're assuming that we're generating an update from software version X to version Y. The location of the full FOTA target zip built from software version X using the instructions above will be called $TARGET_FILES_X below. This might be a path on a build server like /home/build/b2g/versions/X/target_files.zip. Similarly, the location of the full FOTA target zip built from version Y will be called $TARGET_FILES_Y.

The tool build/tools/releasetools/ota_from_target_files will generate an incremental FOTA update.zip using $TARGET_FILES_X and $TARGET_FILES_Y. We'll call the destination of this intermediate file $INTERMEDIATE_FOTA_UPDATE_FOTA_X_Y.

After this update.zip is generated, the last step is to wrap it in a MAR for delivery to the B2G client. The tool tools/update-tools/build-fota-mar.p does this step. We'll call the destination of this generated file $GENERATED_INCREMENTAL_FOTA_X_Y. Use the following commands to complete this step:

$ cd $b2g
$ ./build/tools/releasetools/ota_from_target_files -v \
    -i $TARGET_FILES_X \
    -p out/host/$HOST_ARCH \
    -k $FOTA_SIGNING_KEY \
    $TARGET_FILES_Y \
    $INTERMEDIATE_FOTA_UPDATE_FOTA_X_Y
$ ./tools/update-tools/build-fota-mar.py $INTERMEDIATE_FOTA_UPDATE_FOTA_X_Y --output=$GENERATED_INCREMENTAL_FOTA_X_Y

The variable values in the commands listed above should be filled in as follows:

Variable Meaning
$TARGET_FILES_X The FOTA target files zip for version X
$TARGET_FILES_Y The FOTA target files zip for version Y
$GENERATED_INCREMENTAL_FOTA_X_Y The destination incremental update zip wrapped in a MAR for delivery to clients
$HOST_ARCH The host and arch combo (i.e. linux-x86 or darwin-x86)
$FOTA_SIGNING_KEY Path to the prefix for a private key and public cert for signing the update zip. $FOTA_SIGNING_ZIP.pk8 and $FOTA_SIGNING_ZIP.x509.pem should both exist on the file system

Hosting updates and polling for updates on the client side

Firefox OS clients poll for updates by fetching and parsing an update manifest: update.xml. Firefox OS clients are configured to poll for updates on specific servers — they query a specially-constructed path on the server. HTTPS is the recommended protocol that the client uses to query the server, however HTTP is also supported. The server and path polled by clients can be changed by shipping an update to existing clients that changes the polling code.

In the examples below, we'll assume that updates are hosted on the server updates.b2g.com.

The URL polled by the client commonly contains the following parameters:

Parameter Explanation
PRODUCT_MODEL The name of the device model. This is the ro.product.model value in the B2G property database.
CHANNEL The update "channel". This is useful for testing: servers can be configured to host, for example, "nightly", "beta", and "release" channels.
VERSION The client's software version. For example, "18.0.2".
BUILD_ID A unique ID such as a timestamp, configured for a particular build.

However, there are more values that can be used to construct the queried update URL.

The Firefox client uses the value of its configured update host and these values to construct a URL to poll at runtime. An example of such a URL is

https://updates.b2g.com/release/unagi1/18.0/20121203123456/update.xml

If the server returns a "404 Not Found" in response to the client's request, then there is no update available. If the server returns a "200" and a manifest file, then there may be an update available. The manifest describes the newly available build; that is, the build the client would update to. An example manifest is:

<?xml version="1.0"?>
<updates>
  <update type="major" appVersion="19.0" version="19.0" extensionVersion="19.0" buildID="20121210123456"
          licenseURL="http://www.mozilla.com/test/sample-eula.html"
          detailsURL="http://www.mozilla.com/test/sample-details.html">
    <patch type="partial" URL="https://updates.b2g.com/release/unagi1/18.0/20121203123456/update.mar"
           hashFunction="SHA512" hashValue="5111e033875752b7d9b32b4795152dea5ef954cb8a9d4a602dd19a923b464c43521287dcb5781faf3af76e6dc5e8a3dd9c13edea18c1f2c8f3bd89e17d103d6f"
           size="41901319"/>
  </update>
</updates>

The fields in the manifest describe:

  • metadata used to show a user interface on the client
  • metadata about the newly-available version
  • the location of the update package
  • metadata used to verify the download of the update package

Note: The client device or the user may wish to decline an update.

Using the mechanisms described above, servers can host update packages to update any old client version to the newest version. Or they may host only a "linear update history" in which clients must upgrade through a single path.

The details of the interaction between build servers and the update host is currently beyond the scope of this document. It is highly dependent on the production environment.

Verifying and applying updates

After a Firefox OS client has successfully polled for an update, downloaded it, and verified the integrity of the downloaded update package, the final step is to apply the update.

The first step in applying an update is to verify the signatures embedded in the MAR packages. This is done by the Firefox OS client itself after checking the integrity of the downloaded package. The code used for this is the same for both FOTA and Gecko/Gaia OTA updates.

After signatures are verified, the process of applying an update diverges between Gecko/Gaia OTA updates and FOTA updates. Let's look at the differences between the two at this point.

Applying Gecko/Gaia OTA updates

The Firefox OS client applies these using the updater binary. This is part of the Gecko distribution and is the same code used to apply updates for desktop Firefox. As described above, the update is applied while the Firefox OS client continues to run normally. Users are able to make and receive calls, run apps, browse the web, etc. while updates are being applied.

The specific details of the updater binary are beyond the scope of this document, but it works approximately like so:

  • Making a copy of the /system/b2g files.
  • Applying binary patches, removing old files, and adding new ones as specified by the MAR file.
  • Restarting the main b2g process so that it uses the new files.

After the b2g process finishes restarting, the user is running the new version of the B2G client software.

Applying FOTA updates

The FOTA client applies these. The Gecko client "hands off" the update to be applied by calling into the librecovery API. What happens after this step is specific to each FOTA client.

In the implementation of librecovery used for the GOTA client, the downloaded update package is staged to be applied and special commands are enqueued for the recovery client. librecovery then reboots the device into recovery mode. The recovery client then runs the update script in the update.zip to update files and partitions as needed. The recovery client may need to reboot multiple times in order to update all files.

After the final reboot, the device will be running the new version of the Firefox OS client software.

Document Tags and Contributors

Contributors to this page: rbrandao, morvanabonin, chrisdavidmills
Last updated by: rbrandao,