Link prefetching FAQ

  • Revision slug: Link_prefetching_FAQ
  • Revision title: Link prefetching FAQ
  • Revision id: 53546
  • Created:
  • Creator: Corycory
  • Is current revision? No
  • Comment /* Is there a preference to disable link prefetching? */

Revision Content

What is link prefetching?

Link prefetching is a browser mechanism, which utilizes browser idle time to download or prefetch documents that the user might visit in the near future. A web page provides a set of prefetching hints to the browser, and after the browser is finished loading the page, it begins silently prefetching specified documents and stores them in its cache. When the user visits one of the prefetched documents, it can be served up quickly out of the browser's cache.

What are the prefetching hints?

The browser looks for either an HTML <link> tag or an HTTP Link: header with a relation type of either next or prefetch. An example using the <link> tag follows:

<link rel="prefetch" href="/images/big.jpeg">

The same prefetching hint using an HTTP Link: header:

Link: </images/big.jpeg>; rel=prefetch

The Link: header can also be specified within the HTML document itself by using a HTML <meta> tag:

<meta http-equiv="Link" content="</images/big.jpeg>; rel=prefetch">

The format for the Link: header is described in RFC 2068 section 19.6.2.4.

Note: We are intentionally referencing the out-dated HTTP/1.1 specification here since the updated version of the standard RFC 2616 does not describe the Link: header. Despite Link: headers not being part of the revised standard, they are still used in practice by servers to specify CSS stylesheets, and so we make use of the same facility here.

The browser observes all of these hints and queues up each unique request to be prefetched when the browser is idle. There can be multiple hints per page, as it might make sense to prefetch multiple documents. For example, the next document might contain several large images.

Some more examples follow:

<link rel="prefetch alternate stylesheet" title="Designed for Mozilla" href="mozspecific.css">
<link rel="next" href="2.html">

Are anchor (<a>) tags prefetched?

No, only <link> tags with a relation type of next or prefetch are prefetched. However, if there is sufficient interest, we may expand link prefetching support to include prefetching <a> tags, which include a relation type of next or prefetch in the future. Doing so would probably help content providers avoid the problem of stale prefetching links.

Is link prefetching standards compliant?

Yes, link prefetching as outlined in this document does not violate any existing web standards. In fact, the HTML 4.01 specification explicitly allows for the definition of new link relation types (see Section 6.12: Link types). However, the exact mechanism employed by Mozilla is not yet standardized. An Internet-Draft is in the works.

How is browser idle time determined?

In the current implementation (Mozilla 1.2), idle time is determined using the nsIWebProgressListener API. We attach a listener to the toplevel nsIWebProgress object ("@mozilla.org/docloaderservice;1"). From this, we receive document start & stop notifications, and we approximate idle time as the period between the last document stop and the next document start. The last document stop notification occurs roughly when the onLoad handler would fire for the toplevel document. This is when we kick off prefetch requests. If a subframe contains prefetching hints, prefetching will not begin until the top-most frame and all its "child" frames finish loading.

What happens if I click on a link while something is being prefetched?

When the user clicks on a link, or initiates any kind of page load, link prefetching will stop and any prefetch hints will be discarded. If a prefetched document is partially downloaded, then the partial document will still be stored in the cache provided the server sent an "Accept-Ranges: bytes" response header. This header is typically generated by webservers when serving up static content. When the user visits a prefetched document for real, the remaining portion of the document will be fetched using a HTTP byte-range request.

What if I'm downloading something in the background? Will link prefetching compete for bandwidth?

Yes and no. If you are downloading something using Mozilla, link prefetching will be delayed until any background downloads complete. For example, if you load a bookmark group (which opens several tabs), any prefetch requests initiated by one of the bookmarked pages will not begin until all of the tabs finish loading. If you are using another application which uses the network, link prefetching in Mozilla may compete for bandwidth with the other application. This is a problem that we hope to address in the future by leveraging operating system services to monitor network idle time.

Are there any restrictions on what is prefetched?

Yes, only http:// URLs can be prefetched (https:// URLs are never prefetched for security reasons). Other protocols (such as FTP) do not provide rich enough support for client side caching. In addition to this restriction, URLs with a query string are not prefetched. This is done because such URLs often result in documents that cannot be reused out of the browser's cache, so prefetching them often has little benefit. We found that some existing sites utilize the <link rel="next"> tag with URLs containing query strings to reference the next document in a series of documents. Bugzilla is an example of such a site that does this, and it turns out that the Bugzilla bug reports are not cachable, so prefetching these URLs would nearly double the load on poor Bugzilla! It's easy to imagine other sites being designed like Bugzilla, so we explicitly do not prefetch URLs with query strings. (It might make sense to allow prefetching of these documents when the rel=prefetch relation type is specified, since this should not appear in any existing content.) There are no other restrictions on the URLs that are prefetched.

Will Mozilla prefetch documents from a different host?

Yes. There is no same-origin restriction for link prefetching. Limiting prefetching to only URLs from the the same server would not offer any increased browser security.

Do prefetched requests contain a Referer: header?

Yes, prefetched requests include a HTTP Referer: header indicating the document from which the prefetching hint was extracted.

This may impact referrer tracking that is commonly used on many sites. For this reason, link prefetching may not be appropriate for all content. However, it is possible to instruct Mozilla to validate a prefetched document when the user follows a href to the prefetched document by specifying the Cache-control: must-revalidate HTTP response header. This header enables caching, but requires an If-Modified-Since or If-None-Match validation request before the serving the document out of the browser's cache.

As a server admin, can I distinguish prefetch requests from normal requests?

Yes, we send the following header along with each prefetch request:

X-moz: prefetch

Of course, this request header is not at all standardized, and it may change in future Mozilla releases.

Is there a preference to disable link prefetching?

Yes, there is a hidden preference that you can set to disable link prefetching. Add this line to your prefs.js file located in your Mozilla profile directory:

user_pref("network.prefetch-next",false);

We are considering adding UI for this preference (see {{template.Bug(166648)}}); however, our theory is that if link prefetching needs to be disabled then there must be something wrong with the implementation. We would rather improve the implementation if it does not work correctly, than simply expect users to locate some obscure preference in the preferences UI. Heck, the Mozilla preferences UI is already crowded enough ;-)

Update: By popular demand, Mozilla 1.3+ includes a preference in the UI to disable prefetching. See Preferences->Advanced->
user_pref("network.prefetch-next",false);

What about folks who pay-per-byte for network bandwidth?

Basically, there are two ways of looking at this issue:

  1. websites can already cause things to be silently downloaded using JS/DOM hacks.
  2. prefetching is a browser feature; users should be able to disable it easily.

It is important that websites adopt <link> tag based prefetching instead of trying to roll-in silent downloading using various JS/DOM hacks. The <link> tag gives the browser the ability to know what sites are up to, and we can use this information to better prioritize document prefetching. The user preference to disable <link> tag prefetching may simply encourage websites to stick with JS/DOM hacks, and that would not be good for users. This is one reason why prefetching is enabled by default.

Which browsers support link prefetching?

Browser based on Mozilla 1.2 (or later) as well as browsers based on Mozilla 1.0.2 (or later) support prefetching. This includes Firefox and Netscape 7.01+. Camino builds as of March 2003 are based on Mozilla 1.0.1, and therefore do not support prefetching. Test your browser to see if it supports Link Prefetching.

What about...?

If you have any questions or comments about link prefetching, please feel free to send them my way :-)

See also...

Prefetching Hints

Original Document Information

  • Author(s): Darin Fisher (darin at meer dot net)
  • Last Updated Date: Updated: March 3, 2003

Revision Source

<p>
</p>
<h3 name="What_is_link_prefetching.3F">What is link prefetching?</h3>
<p>Link prefetching is a browser mechanism, which utilizes browser idle time to download or <i>prefetch</i> documents that the user might visit in the near future. A web page provides a set of prefetching hints to the browser, and after the browser is finished loading the page, it begins silently prefetching specified documents and stores them in its cache. When the user visits one of the prefetched documents, it can be served up quickly out of the browser's cache.
</p>
<h3 name="What_are_the_prefetching_hints.3F">What are the prefetching hints?</h3>
<p>The browser looks for either an HTML &lt;link&gt; tag or an HTTP <code>Link:</code> header with a relation type of either <code>next</code> or <code>prefetch</code>. An example using the &lt;link&gt; tag follows:
</p>
<pre>&lt;link rel="prefetch" href="/images/big.jpeg"&gt;</pre>
<p>The same prefetching hint using an HTTP <code>Link:</code> header:
</p>
<pre>Link: &lt;/images/big.jpeg&gt;; rel=prefetch</pre>
<p>The Link: header can also be specified within the HTML document itself by using a HTML <code>&lt;meta&gt;</code> tag:
</p>
<pre>&lt;meta http-equiv="Link" content="&lt;/images/big.jpeg&gt;; rel=prefetch"&gt;</pre>
<p>The format for the <code>Link:</code> header is described in <a class="external" href="http://tools.ietf.org/html/rfc2068" title="http://tools.ietf.org/html/rfc2068">RFC 2068</a> section 19.6.2.4.
</p>
<div class="tip">Note: We are intentionally referencing the out-dated HTTP/1.1 specification here since the updated version of the standard <a class="external" href="http://tools.ietf.org/html/rfc2616" title="http://tools.ietf.org/html/rfc2616">RFC 2616</a> does not describe the <code>Link:</code> header. Despite <code>Link:</code> headers not being part of the revised standard, they are still used in practice by servers to specify CSS stylesheets, and so we make use of the same facility here.</div>
<p>The browser observes all of these hints and queues up each unique request to be prefetched when the browser is idle. There can be multiple hints per page, as it might make sense to prefetch multiple documents. For example, the next document might contain several large images.
</p><p>Some more examples follow: 
</p>
<pre>&lt;link rel="prefetch alternate stylesheet" title="Designed for Mozilla" href="mozspecific.css"&gt;
&lt;link rel="next" href="2.html"&gt;
</pre>
<h3 name="Are_anchor_.28.3Ca.3E.29_tags_prefetched.3F">Are anchor (&lt;a&gt;) tags prefetched?</h3>
<p>No, only <code>&lt;link&gt;</code> tags with a relation type of <code>next</code> or <code>prefetch</code> are prefetched. However, if there is sufficient interest, we may expand link prefetching support to include prefetching &lt;a&gt; tags, which include a relation type of <code>next</code> or <code>prefetch</code> in the future. Doing so would probably help content providers avoid the problem of stale prefetching links.
</p>
<h3 name="Is_link_prefetching_standards_compliant.3F">Is link prefetching standards compliant?</h3>
<p>Yes, link prefetching as outlined in this document does not violate any existing web standards. In fact, the HTML 4.01 specification explicitly allows for the definition of new link relation types (<a class="external" href="http://www.w3.org/TR/html4/types.html#type-links">see Section 6.12: Link types</a>). However, the exact mechanism employed by Mozilla is not yet standardized. An Internet-Draft is in the works. 
</p>
<h3 name="How_is_browser_idle_time_determined.3F">How is browser idle time determined?</h3>
<p>In the current implementation (Mozilla 1.2), idle time is determined using the <code>nsIWebProgressListener</code> API. We attach a listener to the toplevel <code>nsIWebProgress</code> object ("@mozilla.org/docloaderservice;1"). From this, we receive document start &amp; stop notifications, and we approximate idle time as the period between the last document stop and the next document start. The last document stop notification occurs roughly when the onLoad handler would fire for the toplevel document. This is when we kick off prefetch requests. If a subframe contains prefetching hints, prefetching will not begin until the top-most frame and all its "child" frames finish loading.
</p>
<h3 name="What_happens_if_I_click_on_a_link_while_something_is_being_prefetched.3F">What happens if I click on a link while something is being prefetched?</h3>
<p>When the user clicks on a link, or initiates any kind of page load, link prefetching will stop and any prefetch hints will be discarded. If a prefetched document is partially downloaded, then the partial document will still be stored in the cache provided the server sent an "Accept-Ranges: bytes" response header. This header is typically generated by webservers when serving up static content. When the user visits a prefetched document for real, the remaining portion of the document will be fetched using a HTTP byte-range request.
</p>
<h3 name="What_if_I.27m_downloading_something_in_the_background.3F_Will_link_prefetching_compete_for_bandwidth.3F">What if I'm downloading something in the background? Will link prefetching compete for bandwidth?</h3>
<p>Yes and no. If you are downloading something using Mozilla, link prefetching will be delayed until any background downloads complete. For example, if you load a bookmark group (which opens several tabs), any prefetch requests initiated by one of the bookmarked pages will not begin until all of the tabs finish loading. If you are using another application which uses the network, link prefetching in Mozilla may compete for bandwidth with the other application. This is a problem that we hope to address in the future by leveraging operating system services to monitor network idle time.
</p>
<h3 name="Are_there_any_restrictions_on_what_is_prefetched.3F">Are there any restrictions on what is prefetched?</h3>
<p>Yes, only http:// URLs can be prefetched (https:// URLs are never prefetched for security reasons). Other protocols (such as FTP) do not provide rich enough support for client side caching. In addition to this restriction, URLs with a query string are not prefetched. This is done because such URLs often result in documents that cannot be reused out of the browser's cache, so prefetching them often has little benefit. We found that some existing sites utilize the &lt;link rel="next"&gt; tag with URLs containing query strings to reference the next document in a series of documents. Bugzilla is an example of such a site that does this, and it turns out that the Bugzilla bug reports are not cachable, so prefetching these URLs would nearly double the load on poor Bugzilla! It's easy to imagine other sites being designed like Bugzilla, so we explicitly do not prefetch URLs with query strings. (It might make sense to allow prefetching of these documents when the <code>rel=prefetch</code> relation type is specified, since this should not appear in any existing content.) There are no other restrictions on the URLs that are prefetched. 
</p>
<h3 name="Will_Mozilla_prefetch_documents_from_a_different_host.3F">Will Mozilla prefetch documents from a different host?</h3>
<p>Yes. There is no same-origin restriction for link prefetching. Limiting prefetching to only URLs from the the same server would not offer any increased browser security. 
</p>
<h3 name="Do_prefetched_requests_contain_a_Referer:_header.3F">Do prefetched requests contain a Referer: header?</h3>
<p>Yes, prefetched requests include a HTTP <code>Referer:</code> header indicating the document from which the prefetching hint was extracted.
</p><p>This may impact referrer tracking that is commonly used on many sites. For this reason, link prefetching may not be appropriate for all content. However, it is possible to instruct Mozilla to validate a prefetched document when the user follows a href to the prefetched document by specifying the <code>Cache-control: must-revalidate</code> HTTP response header. This header enables caching, but requires an <code>If-Modified-Since</code> or <code>If-None-Match</code> validation request before the serving the document out of the browser's cache.
</p>
<h3 name="As_a_server_admin.2C_can_I_distinguish_prefetch_requests_from_normal_requests.3F">As a server admin, can I distinguish prefetch requests from normal requests?</h3>
<p>Yes, we send the following header along with each prefetch request:
</p>
<pre>X-moz: prefetch</pre>
<p>Of course, this request header is not at all standardized, and it may change in future Mozilla releases. 
</p>
<h3 name="Is_there_a_preference_to_disable_link_prefetching.3F">Is there a preference to disable link prefetching?</h3>
<p>Yes, there is a hidden preference that you can set to disable link prefetching. Add this line to your prefs.js file located in your Mozilla profile directory:
</p>
<pre>user_pref("network.prefetch-next",false);</pre>
<p>We are considering adding UI for this preference (<strike>see {{template.Bug(166648)}}</strike>); however, our theory is that if link prefetching needs to be disabled then there must be something wrong with the implementation. We would rather improve the implementation if it does not work correctly, than simply expect users to locate some obscure preference in the preferences UI. Heck, the Mozilla preferences UI is already crowded enough ;-)
</p>
<div class="tip">
Update: By popular demand, Mozilla 1.3+ includes a preference in the UI to disable prefetching. See Preferences-&gt;Advanced-&gt;<pre>user_pref("network.prefetch-next",false);</pre></div>
<h3 name="What_about_folks_who_pay-per-byte_for_network_bandwidth.3F">What about folks who pay-per-byte for network bandwidth?</h3>
<p>Basically, there are two ways of looking at this issue:
</p>
<ol><li> websites can already cause things to be silently downloaded using JS/DOM hacks.
</li><li> prefetching is a browser feature; users should be able to disable it easily.
</li></ol>
<p>It is important that websites adopt <code>&lt;link&gt;</code> tag based prefetching instead of trying to roll-in silent downloading using various JS/DOM hacks. The <code>&lt;link&gt;</code> tag gives the browser the ability to know what sites are up to, and we can use this information to better prioritize document prefetching. The user preference to disable <code>&lt;link&gt;</code> tag prefetching may simply encourage websites to stick with JS/DOM hacks, and that would not be good for users. This is one reason why prefetching is enabled by default. 
</p>
<h3 name="Which_browsers_support_link_prefetching.3F">Which browsers support link prefetching?</h3>
<p>Browser based on Mozilla 1.2 (or later) as well as browsers based on Mozilla 1.0.2 (or later) support prefetching. This includes Firefox and Netscape 7.01+. Camino builds as of March 2003 are based on Mozilla 1.0.1, and therefore do not support prefetching. <a class="external" href="http://gemal.dk/browserspy/prefetch.php">Test</a> your browser to see if it supports Link Prefetching. 
</p>
<h3 name="What_about....3F">What about...?</h3>
<p>If you have any questions or comments about link prefetching, please feel free to send them my way :-)
</p>
<h4 name="See_also...">See also...</h4>
<p><a class="external" href="http://www.edochan.com/programming/pf.htm">Prefetching Hints</a>
</p>
<div class="originaldocinfo">
<h2 name="Original_Document_Information"> Original Document Information </h2>
<ul><li> Author(s): Darin Fisher (darin at meer dot net)
</li><li> Last Updated Date: Updated: March 3, 2003
</li></ul>
</div>
Revert to this revision