Multimedia: Images and Video

Media, namely images and video, account for over 70% of the bytes downloaded for the average website. In terms of download performance, eliminating media, and reducing file size is the low-hanging fruit. This article looks at optimizing image and video to improve web performance.

This is a high-level introduction to optimizing multimedia delivery on the web, covering general principles and techniques.  For a more in-depth guide, see  https://images.guide.

Why optimize your multimedia?

For the average website, 51% of its bandwidth comes from imagery, followed by video at 25%, so it's safe to say it's important to address and optimize your multi-media content.

You need to be considerate of data usage. Many people are on capped data plans or even pay-as-you-go where they are literally paying by the megabyte. This isn't an emerging market problem either. As of 2018, 24% of the United Kingdom still use pay-as-you-go.

You also need to be considerate of memory as many mobile devices have limited RAM. It's important to remember that when images are downloaded, they need to be stored in memory.

Optimizing image delivery

Despite being the largest consumer of bandwidth, images load asynchronously so the visitor can see the page as they download and therefore, their impact on perceived performance is far lower than many expect. However, images are mostly used for content, so it's important that a visitor can see them as soon as possible for a good experience.

Loading strategy

One of the biggest improvements to most websites is lazy-loading images beneath-the-fold, rather than downloading them all on initial page load regardless of whether a visitor scrolls to see them or not. Many JavaScript libraries can implement this for you, such as lazysizes, and browser vendors are working on a native lazyload attribute that is currently in the experimental phase.

Beyond loading a subset of images, next you should look into the format of the images themselves:

  • Are you loading the most optimal file formats?
  • Have you compressed the images well?
  • Are you loading the correct sizes?

The most optimal format

The optimal file format typically depends on the character of the image.

The less colors an image has and the less photographic it is, the better it is to use the SVG format. This requires the source to be available as in a vector graphic format. Should such an image only exist as a bitmap, then PNG would be the fallback format to chose. Examples for these types of motifs are logos, illustrations, charts or icons (note: SVGs are far better than icon fonts!). Both formats support transparency.

PNGs can be saved with three different output combinations:

  • 24 bit color + 8 bit transparency — offer full color acurracy and smooth transparencies at the expense of size. You probably want to avoid this combination in favor for WebP (see below).
  • 8 bit color + 8 bit transparency — offer no more than 255 colors but maintain smooth transparencies. The size is not too big. Those are the PNGs you would probably want.
  • 8 bit color + 1 bit transparency — offer no more than 255 colors and just offer no or full transparency per pixel which makes the transparency borders appear hard and jagged. The size is small but the price is visual fidelity.

A good online tool for optimizing SVGs is SVGOMG. For PNGs there is ImageOptim online or Squoosh.

With photographic motifs that do not feature transparency there is a lot wider range of formats to chose from. If you want to play it safe, then you would go for well compressed Progressive JPEGs. Progressive JPEGs, in contrast to normal JPEGs, render progressively (hence the name), meaning the user sees a low-resolution version that gains clarity as the image downloads, rather than the image loading at full resolution top-to-bottom or even only rendering once completely downloaded. A good compressor for these would be MozJPEG, e.g. available to use in the online image optimization tool Squoosh. A quality setting of 75% should yield decent results.

Other formats improve on JPEG's capabilities in regards to compression, but are not available on every browser: 

  • WebP — created by Google and nowadays supported by all major browsers except Safari. Supports transparency and also animation, but not progressive display. 
  • JPEG-XR — created by Microsoft and only available in Internet Explorer and EdgeHTML based Edge. Doesn't support progressive display and the image decoding is not hardware accellerated and therefore resource intensive on the browser's main thread.
  • JPEG2000 — once to be successor to JPEG but only supported in Safari. Doesn't support progressive display either.

In the future browsers might add support for even more advanced formats like High Efficiency Image File Format (HEIF) or AV1 Still Image File Format (AVIF).

Given the narrow support for JPEG-XR and JPEG2000, and also taking decode costs into the equation, the only serious contender for JPEG today is WebP. Which is why you could consider offering your images in that flavor, too, for the browsers that support it. This can be done via the <picture> element with the help of a <source> element equipped with a type attribute.

If all of this sounds a bit complicated or feels like too much work for your team then there is also online services that you can use as image CDNs that will automate the serving of the correct image format on-the-fly, according to the type of device or browser requesting the image. The biggest ones are Cloudinary and Image Engine.

And finally, should you want to include animated images into your page, then know that Safari allows using video files within <img> and <picture> elements. These also allow you to add in an Animated WebP for all other modern browsers.

<picture>
   <source type="video/mp4" src="giphy.mp4">
   <source type="image/webp" src="giphy.webp">
   <img src="giphy.gif">
</picture>

Serving the optimal size

In image delivery the "one size fits all" approach will not yield the best results, meaning that for smaller screens you would want to serve images with smaller resolution and vis-versa for larger screens. On top of that you'd also want to serve higher resolution images to those devices that boast a high DPI screen (e.g. "Retina"). So apart from creating plenty of intermediate image variants you also need a way to serve the right file to the right browser. That's where you would need to upgrade your <picture> and <source> elements with media and/or sizes attributes. A detailed article on how to combine all of these attributes can be found here.

Two interesting effects to keep in mind regarding high dpi screens is that: 

Controlling the priority (and ordering) of downloading images

Getting the most important images in front of visitors sooner than the less important can deliver improved perceived performance.

The first thing to check is that your content images use  <img> elements and your background images are defined in CSS with background-image — images referenced in <img> elements are assigned a higher loading priority than background images.

Secondly, with the adoption of Priority Hints, you can control the priority further by adding an importance attribute to your image tags. An example use case for priority hints on images are carousels where the first image is a higher priority than the subsequent images.

Rendering strategy

As images are loaded asynchronously and continue to load after the first paint, if their dimensions aren't defined before load, they can cause reflows to the page content. For example, when text gets pushed down the page by images loading. For this reason, it's critical that you set width and height attributes so that the browser can reserve space for them in the layout.

For any background images, it's important you set a background-color value so any content overlaid is still readable before the image has downloaded.

Optimizing video delivery

To ensure you don't send needlessly large files to the user, it's best to compress all videos you do deliver, optimize <source> order, set autoplayremove audio from muted heroes, optimize the video preload, and consider streaming the video.

Compress all videos

Most video compression efforts involve comparing adjacent frames within a video and removing details that are the same in the original and subsequent frame. You want to both compress the video and export it to multiple video formats, including WebM, MPEG-4/H.264, and Ogg/Theora.

The software you used to create your video likely includes the ability to optimize the file size down. If not, there are several online tools, such as FFmpeg, discussed in a later section, that can help encode, decode, convert, and perform other forms of magic.

Optimize <source> order

Order video source from smallest to largest.  For example, given three video compressions at 10MB, 12MB, and 13MB, put the smallest first and the largest last:

<video width="400" height="300" controls="controls">
  <!-- WebM: 10 MB -->
  <source src="video.webm" type="video/webm" />
  <!-- MPEG-4/H.264: 12 MB -->
  <source src="video.mp4" type="video/mp4" />
  <!-- Ogg/Theora: 13 MB -->
  <source src="video.ogv" type="video/ogv" />
</video>

In terms of the order, the browser downloads the first video source it understands, so let it load a smaller one first. In terms of “smallest”, do make sure that your most compressed video still looks good. There are some compression algorithms that can make your video look like an animated gif. While a 128 Kb video may seem like better user experience than having your users download a 10 MB video, putting a grainy gif-like video behind your content may also negatively impact your brand.

See CanIUse.com for current browser support of video and the various media types.  

Video autoplay

To ensure that your looping background video autoplays, you need to add several attributes to the video tag: autoplay, muted, and playsinline.

<video autoplay="" loop="" muted="true" playsinline="" src="backgroundvideo.mp4">

While the loop and autoplay make sense for a looping and autoplaying video, the muted attribute is required for autoplay in mobile browsers.

Playsinline is required for mobile Safari, allowing videos to play without forcing fullscreen mode.

Remove audio from muted hero videos

If you do have a hero-video or other video without audio, remove the audio from your video file. 

<video autoplay="" loop="" muted="true" playsinline="" id="hero-video">
  <source src="banner_video.webm" 
          type='video/webm; codecs="vp8, vorbis"'>
  <source src="web_banner.mp4" type="video/mp4">
</video>

This hero-video code, common to many conference websites and corporate home pages, includes a video that is auto-playing, looping, and muted. It contains no controls, so there is no way to hear the audio. The audio is often empty, but it's still present. It is still using bandwidth. There is no reason to serve the audio along with a video that is always muted. Removing the audio can save 20% of the bandwidth, which is 2 MB if your video is 10 MB.

Depending on your video making software, you might be able to remove the audio during export and compression. If not, there is a free tool called FFmpeg that can do it for you with the following command:

ffmpeg -i original.mp4 -an -c:v copy audioFreeVersion.mp4

FFmpeg bills itself as the “complete, cross-platform solution to record, convert and stream audio and video,” which it pretty much is.

Video preload

The preload attribute has 3 available options: auto|metadata|none.  The default setting is metadata.

Changing the setting to auto tells the browser to automatically download the entire video. This should only be done when playback is extremely likely, otherwise large amounts of bandwdith is wasted.

preload="metadata" can result in up to 3% of the video being downloaded on page load.  For larger videos this can be a significant amount of data.

preload="none" results in none of the video being downloaded until playback is requested.  This can delay video startup time, but also saves large amounts of data for videos with a low probability of playback.

Consider streaming

Video streaming allows the proper video size and bandwidth (based on network speed) to be delivered to the end user.  Much like using responsive images, the correct size video is delivered to the browser, ensuring fast video startup, low buffering, and optimised playback to the end user.eason for using Progressive JPEG above-the-fold is that they render progressively (hence the name), meaning the user sees a low-resolution version that gains clarity as the image downloads, rather than the image loading at full resolution top-to-bottom or even only rendering once completely downloaded.