Creating Sandboxed HTTP Connections

  • Revision slug: Creating_Sandboxed_HTTP_Connections
  • Revision title: Creating Sandboxed HTTP Connections
  • Revision id: 105552
  • Created:
  • Creator: Doron
  • Is current revision? No
  • Comment http -> HTTP

Revision Content

Note: this is still a work in progress.

Introduction

In Gecko 1.8.1 (Firefox 2.0), a bug was fixed that finally allows one to create sandboxed HTTP connections which don't affect the user's cookies. This article will cover the basics of doing HTTP connections from XPCOM JavaScript, and should easily translate to C++ XPCOM.

Setting up an HTTP Connection

The first step in setting up an HTTP connection from an URL (stored in a string) is to create an nsIURI out of it. nsIURI is an XPCOM representation of an URI, with usefull methods to query and manipulate the URI. To create an nsIURI from an string, the nsIIOService is called and the newURI method is called:

// the IO service
var ioService = Components.classes["@mozilla.org/network/io-service;1"].getService(Components.interfaces.nsIIOService);

// create an nsIURI
var uri = ioService.newURI(myURLString, null, null);

Once the nsIURI has been created a nsIChannel can be generated out of it using nsIIOService's newChannelFromURI method:

// get a channel for that nsIURI
var channel = ioService.newChannelFromURI(uri);

To initiate the connection, the asyncOpen method is called. It takes two arguments: an listener and a context that is passed to the listener's methods.

channel.asyncOpen(listener, null);

HTTP Notifications

The above mentioned listener is a nsIStreamListener, which gets notified about events such as HTTP redirects and data availability.

  • onStartRequest - gets called when a new requestion is initiated.
  • onDataAvailable - new data is available. Since this is a stream, it could be called multiple times (depending on the size of the returned data, networking conditions, etc).
  • onStopRequest - the requestion has finished.
  • onChannelRedirect - when a redirect happens, a new nsIChannel is created, and both the old and new ones are passed in as arguments.

Since nsIStreamListener does not cover cookies, the current channel being used will need to be stored as a global, since another listener will be used for cookie notifications (covered in the next section). It is usually best to create a JavaScript object that implements all the required methods, which is generated by a function. Since usually a callback is needed once the connection has completed, a function reference is passed into the function. Below is an example:

// global channel
var gChannel;

// init the channel

// the IO service
var ioService = Components.classes["@mozilla.org/network/io-service;1"].getService(Components.interfaces.nsIIOService);

// create an nsIURI
var uri = ioService.newURI(myURLString, null, null);

// get a channel for that nsIURI
gChannel = ioService.newChannelFromURI(uri);

// get an listener
var listener = new streamListener (aCallbackFunc);

gChannel.asyncOpen(listener, null);

function streamListener = function(aCallbackFunc) {
  return ({
    mData : null,

    onStartRequest : function (aRequest, aContext) {
      this.mData = "";
    },

    onDataAvailable : function (aRequest, aContext, aStream, aSourceOffset, aLength){
      var scriptableInputStream = 
        Components.classes["@mozilla.org/scriptableinputstream;1"]
          .createInstance(Components.interfaces.nsIScriptableInputStream);
      scriptableInputStream.init(aStream);

      this.mData += scriptableInputStream.read(aLength);
    },

    onStopRequest : function (aRequest, aContext, aStatus) {
      aCallbackFunc(this.mData);
    },

    onChannelRedirect : function (aOldChannel, aNewChannel, aFlags) {
      // if redirecting, store the new channel
      gChannel = aNewChannel;
    },

    // nsIInterfaceRequestor
    getInterface: function (aIID) {
      try {
        return this.QueryInterface(aIID);
      } catch (e) {
        throw Components.results.NS_NOINTERFACE;
      }
    },

    // nsIProgressEventSink (not implementing will cause annoying exceptions)
    onProgress : function (aRequest, aContext, aProgress, aProgressMax) { },
    onStatus : function (aRequest, aContext, aStatus, aStatusArg) { },

    // we are faking an XPCOM interface, so we need to implement QI
    QueryInterface : function(aIID) {
      if (aIID.equals(Components.interfaces.nsISupports) ||
          aIID.equals(Components.interfaces.nsIInterfaceRequestor) ||
          aIID.equals(Components.interfaces.nsIChannelEventSink) || 
          aIID.equals(Components.interfaces.nsIProgressEventSink) ||
          aIID.equals(Components.interfaces.nsIHttpEventSink) ||
          aIID.equals(Components.interfaces.nsIStreamListener))
        return this;

      throw Components.results.NS_NOINTERFACE;
    }
  });
}

Handling Cookies

When sending a request, cookies that apply to the url are sent with the HTTP request. The HTTP response can also contain cookies, which the browser processes. As of Mozilla 1.8.1 (Firefox 2.0), it is now possible to intercept those two cases.

For example, this means that if the user was logged into an webmail account, another account on the same domain could be checked without changing the user's cookies.

The nsIObserverService service is used to send general notifications, including the two cookie ones. The addObserver method is used to add an observer for a certain topic and takes in three agruments:

  • an object than implements nsIObserver
  • the topic to listen for. For cookies, the two topics are:
    • http-on-modify-request - happens after the cookie data has been loaded into the request, but before the request is sent.
    • http-on-examine-response - happens after the response is received, but before the cookies are processed
  • whether to hold a weak reference to the observer argument. Use false.

Inorder to avoid memory leaks, the observer needs to be removed at one point. The removeObserver method takes in the listener object and the topic and removes it from the notification list.

As with the above stream listener, an nsIObserver implementing object is needed, which only needs to implement one method, observe. The observe method gets passed in three arguments, which for the two cookie topics are:

  • aSubject: the channel (nsIChannel) that caused this notification to happen.
  • aTopic: the notification topic.
  • aData: null for the 2 topics.

Since the nsIObserverService listeners gets notified for the registered topic for any connection, the listener needs to make sure that the notification is for the HTTP connection our code created. Since the channel that causes the notification is passed in as the first argument, comparing it to the globally stored channel (gChannel) in the previous section (which also gets updated each time a redirect happens).

// create an nsIObserver implementor
var listener = new observerListener(myCallbackFunc);

// get the observer service and register for the two coookie topics.
var observerService = Components.classes["@mozilla.org/observer-service;1"].getService(Components.interfaces.nsIObserverService);
observerService.addObserver(listener, "http-on-modify-request", false);
observerService.addObserver(listener, "http-on-examine-response", false);

function observerListener = function(aCallbackFunc) {
  return ({
    observe : function(aSubject, aTopic, aData) {
      // Make sure it is our connection first.
      if (aSubject == gChannel) {
        var httpChannel = aSubject.QueryInterface(Components.interfaces.nsIHttpChannel);
        if (aTopic == "http-on-modify-request") {
           ...
        } else if (aTopic == "http-on-examine-response") {
           ...
        }
      }
    },

    // we are faking an XPCOM interface, so we need to implement QI
    QueryInterface : function(aIID) {
      if (aIID.equals(Components.interfaces.nsISupports) ||
          aIID.equals(Components.interfaces.nsIObserver))
        return this;

      throw Components.results.NS_NOINTERFACE;
    }
  });
}

The final piece is to manipulate the cookies. In order to manipulate cookies, the nsIChannel needs to be converted into a nsIHTTPChannel by using QueryInterface (QI):

var httpChannel = aSubject.QueryInterface(Components.interfaces.nsIHttpChannel);

Cookies are actually part of the HTTP header, nsIHTTPChannel provides four methods for working with headers: two for getting/setting request headers, and two for getting/setting response headers. The cookie header for requests is called "Cookie", while for responses it is "Set-Cookie".

  • getRequestHeader(aHeader) - returns the request header value for the requested header
  • setRequestHeader(aHeader, aValue, aMerge) - sets the request header's value. If aMerge is true, the new value is appened, else the old value is overwritten.
  • getResponseHeader(aHeader) - returns the response header value for the requested header
  • setResponseHeader(aHeader, aValue, aMerge) - sets the response header's value. If aMerge is true, the new value is appened, else the old value is overwritten.

These methods provide all the required functionality needed to modify cookies before they are processed/sent, allowing for sandboxed cookie connections that don't affect the user's cookies.

HTTP Referrer

If the HTTP request needs to have a referrer set, two additional steps are needed after the nsIChannel is created, but before it is opened. First, a nsIURI needs to be generated for the referrer url. Like before, the nsIOService is used:

var referrerUri = ioService.newURI(referrerUrl, null, null);

Next, the nsIChannel is QIed to nsIHTTPChannel and the referrer member is set to the generated nsIURI:

var httpChannel = gNotifierService.channel.QueryInterface(nsIHttpChannel);

// set the referrer
httpChannel.referrer = referrerUri;

Creating HTTP POSTs

If a HTTP post is required, a few additional steps are required after the nsIChannel is created.

First, a nsIInputStream instance is created, after which the setData method is called. The first argument is the POST data as a string, while the second argument is the length of that data:

//nsIInputStream
var inputStream = Components.classes["@mozilla.org/io/string-input-stream;1"]
                  .createInstance(Components.interfaces.nsIStringInputStream);

inputStream.setData(postData, postData.length);

Next, the nsIChannel is QIed to an nsIUploadChannel. Its setUploadStream method is called, passing in the nsIInputStream and the type (in this case, "application/x-www-form-urlencoded"):

var uploadChannel = gChannel.QueryInterface(Components.interfaces.nsIUploadChannel);
uploadChannel.setUploadStream(inputStream, "application/x-www-form-urlencoded", -1);

Due to a bug, calling setUploadStream will reset the nsIHTTPChannel to be a GET request, so now the request type is set to POST:

// order important - setUploadStream resets to get/put
httpChannel.requestMethod = "POST";

Revision Source

<p>
</p><p>Note: this is still a work in progress.
</p>
<h3 name="Introduction"> Introduction </h3>
<p>In Gecko 1.8.1 (<a href="en/Firefox_2.0">Firefox 2.0</a>), a bug was fixed that finally allows one to create sandboxed HTTP connections which don't affect the user's cookies.  This article will cover the basics of doing HTTP connections from XPCOM JavaScript, and should easily translate to C++ XPCOM.
</p>
<h3 name="Setting_up_an_HTTP_Connection"> Setting up an HTTP Connection </h3>
<p>The first step in setting up an HTTP connection from an URL (stored in a string) is to create an <code>nsIURI</code> out of it.  <code>nsIURI</code> is an XPCOM representation of an URI, with usefull methods to query and manipulate the URI.  To create an <code>nsIURI</code> from an string, the <code>nsIIOService</code> is called and the <code>newURI</code> method is called:
</p>
<pre>// the IO service
var ioService = Components.classes["@mozilla.org/network/io-service;1"].getService(Components.interfaces.nsIIOService);

// create an nsIURI
var uri = ioService.newURI(myURLString, null, null);
</pre>
<p>Once the <code>nsIURI</code> has been created a <code>nsIChannel</code> can be generated out of it using <code>nsIIOService</code>'s <code>newChannelFromURI</code> method:
</p>
<pre>// get a channel for that nsIURI
var channel = ioService.newChannelFromURI(uri);
</pre>
<p>To initiate the connection, the <code>asyncOpen</code> method is called.  It takes two arguments: an listener and a context that is passed to the listener's methods.
</p>
<pre>channel.asyncOpen(listener, null);
</pre>
<h3 name="HTTP_Notifications"> HTTP Notifications </h3>
<p>The above mentioned listener is a <code>nsIStreamListener</code>, which gets notified about events such as HTTP redirects and data availability.
</p>
<ul>
  <li>onStartRequest - gets called when a new requestion is initiated.</li>
  <li>onDataAvailable - new data is available.  Since this is a stream, it could be called multiple times (depending on the size of the returned data, networking conditions, etc).</li>
  <li>onStopRequest - the requestion has finished.</li>
  <li>onChannelRedirect - when a redirect happens, a new nsIChannel is created, and both the old and new ones are passed in as arguments.</li>
</ul>
<p>Since <code>nsIStreamListener</code> does not cover cookies, the current channel being used will need to be stored as a global, since another listener will be used for cookie notifications (covered in the next section).  It is usually best to create a JavaScript object that implements all the required methods, which is generated by a function.  Since usually a callback is needed once the connection has completed, a function reference is passed into the function.  Below is an example:
</p>
<pre>// global channel
var gChannel;

// init the channel

// the IO service
var ioService = Components.classes["@mozilla.org/network/io-service;1"].getService(Components.interfaces.nsIIOService);

// create an nsIURI
var uri = ioService.newURI(myURLString, null, null);

// get a channel for that nsIURI
gChannel = ioService.newChannelFromURI(uri);

// get an listener
var listener = new streamListener (aCallbackFunc);

gChannel.asyncOpen(listener, null);

function streamListener = function(aCallbackFunc) {
  return ({
    mData : null,

    onStartRequest : function (aRequest, aContext) {
      this.mData = "";
    },

    onDataAvailable : function (aRequest, aContext, aStream, aSourceOffset, aLength){
      var scriptableInputStream = 
        Components.classes["@mozilla.org/scriptableinputstream;1"]
          .createInstance(Components.interfaces.nsIScriptableInputStream);
      scriptableInputStream.init(aStream);

      this.mData += scriptableInputStream.read(aLength);
    },

    onStopRequest : function (aRequest, aContext, aStatus) {
      aCallbackFunc(this.mData);
    },

    onChannelRedirect : function (aOldChannel, aNewChannel, aFlags) {
      // if redirecting, store the new channel
      gChannel = aNewChannel;
    },

    // nsIInterfaceRequestor
    getInterface: function (aIID) {
      try {
        return this.QueryInterface(aIID);
      } catch (e) {
        throw Components.results.NS_NOINTERFACE;
      }
    },

    // nsIProgressEventSink (not implementing will cause annoying exceptions)
    onProgress : function (aRequest, aContext, aProgress, aProgressMax) { },
    onStatus : function (aRequest, aContext, aStatus, aStatusArg) { },

    // we are faking an XPCOM interface, so we need to implement QI
    QueryInterface : function(aIID) {
      if (aIID.equals(Components.interfaces.nsISupports) ||
          aIID.equals(Components.interfaces.nsIInterfaceRequestor) ||
          aIID.equals(Components.interfaces.nsIChannelEventSink) || 
          aIID.equals(Components.interfaces.nsIProgressEventSink) ||
          aIID.equals(Components.interfaces.nsIHttpEventSink) ||
          aIID.equals(Components.interfaces.nsIStreamListener))
        return this;

      throw Components.results.NS_NOINTERFACE;
    }
  });
}
</pre>
<h3 name="Handling_Cookies"> Handling Cookies </h3>
<p>When sending a request, cookies that apply to the url are sent with the HTTP request.  The HTTP response can also contain cookies, which the browser processes.  As of Mozilla 1.8.1 (Firefox 2.0), it is now possible to intercept those two cases.
</p><p>For example, this means that if the user was logged into an webmail account, another account on the same domain could be checked without changing the user's cookies.
</p><p>The <code>nsIObserverService</code> service is used to send general notifications, including the two cookie ones.  The <code>addObserver</code> method is used to add an observer for a certain topic and takes in three agruments:
</p>
<ul>
  <li>an object than implements nsIObserver</li>
  <li>the topic to listen for.  For cookies, the two topics are:</li>
  <ul>
    <li>http-on-modify-request - happens after the cookie data has been loaded into the request, but before the request is sent.</li>
    <li>http-on-examine-response - happens after the response is received, but before the cookies are processed</li>
  </ul>
  <li>whether to hold a weak reference to the observer argument.  Use false.</li>
</ul>
<p>Inorder to <b>avoid memory leaks</b>, the observer needs to be removed at one point.  The <code>removeObserver</code> method takes in the listener object and the topic and removes it from the notification list.
</p><p>As with the above stream listener, an <code>nsIObserver</code> implementing object is needed, which only needs to implement one method, <code>observe</code>.  The <code>observe</code> method gets passed in three arguments, which for the two cookie topics are:
</p>
<ul>
  <li>aSubject: the channel (nsIChannel) that caused this notification to happen.</li>
  <li>aTopic: the notification topic.</li>
  <li>aData: null for the 2 topics.</li>
</ul>
<p>Since the <code>nsIObserverService</code> listeners gets notified for the registered topic for any connection, the listener needs to make sure that the notification is for the HTTP connection our code created.  Since the channel that causes the notification is passed in as the first argument, comparing it to the globally stored channel (gChannel) in the previous section (which also gets updated each time a redirect happens).  
</p>
<pre>// create an nsIObserver implementor
var listener = new observerListener(myCallbackFunc);

// get the observer service and register for the two coookie topics.
var observerService = Components.classes["@mozilla.org/observer-service;1"].getService(Components.interfaces.nsIObserverService);
observerService.addObserver(listener, "http-on-modify-request", false);
observerService.addObserver(listener, "http-on-examine-response", false);

function observerListener = function(aCallbackFunc) {
  return ({
    observe : function(aSubject, aTopic, aData) {
      // Make sure it is our connection first.
      if (aSubject == gChannel) {
        var httpChannel = aSubject.QueryInterface(Components.interfaces.nsIHttpChannel);
        if (aTopic == "http-on-modify-request") {
           ...
        } else if (aTopic == "http-on-examine-response") {
           ...
        }
      }
    },

    // we are faking an XPCOM interface, so we need to implement QI
    QueryInterface : function(aIID) {
      if (aIID.equals(Components.interfaces.nsISupports) ||
          aIID.equals(Components.interfaces.nsIObserver))
        return this;

      throw Components.results.NS_NOINTERFACE;
    }
  });
}
</pre>
<p>The final piece is to manipulate the cookies.  In order to manipulate cookies, the <code>nsIChannel</code> needs to be converted into a <code>nsIHTTPChannel</code> by using QueryInterface (QI):
</p>
<pre>var httpChannel = aSubject.QueryInterface(Components.interfaces.nsIHttpChannel);
</pre>
<p>Cookies are actually part of the HTTP header, <code>nsIHTTPChannel</code> provides four methods for working with headers: two for getting/setting request headers, and two for getting/setting response headers.  The cookie header for requests is called "Cookie", while for responses it is "Set-Cookie".
</p>
<ul>
  <li>getRequestHeader(aHeader) - returns the request header value for the requested header</li>
  <li>setRequestHeader(aHeader, aValue, aMerge) - sets the request header's value.  If aMerge is true, the new value is appened, else the old value is overwritten.</li>
  <li>getResponseHeader(aHeader) - returns the response header value for the requested header</li>
  <li>setResponseHeader(aHeader, aValue, aMerge) - sets the response header's value.  If aMerge is true, the new value is appened, else the old value is overwritten.</li>
</ul>
<p>These methods provide all the required functionality needed to modify cookies before they are processed/sent, allowing for sandboxed cookie connections that don't affect the user's cookies.
</p>
<h3 name="HTTP_Referrer"> HTTP Referrer </h3>
<p>If the HTTP request needs to have a referrer set, two additional steps are needed after the <code>nsIChannel</code> is created, but before it is opened.  First, a <code>nsIURI</code> needs to be generated for the referrer url.  Like before, the <code>nsIOService</code> is used:
</p>
<pre>var referrerUri = ioService.newURI(referrerUrl, null, null);
</pre>
<p>Next, the <code>nsIChannel</code> is QIed to <code>nsIHTTPChannel</code> and the <code>referrer</code> member is set to the generated <code>nsIURI</code>:
</p>
<pre>var httpChannel = gNotifierService.channel.QueryInterface(nsIHttpChannel);

// set the referrer
httpChannel.referrer = referrerUri;
</pre>
<h3 name="Creating_HTTP_POSTs"> Creating HTTP POSTs </h3>
<p>If a HTTP post is required, a few additional steps are required after the <code>nsIChannel</code> is created.
</p><p>First, a <code>nsIInputStream</code> instance is created, after which the <code>setData</code> method is called.  The first argument is the POST data as a string, while the second argument is the length of that data:
</p>
<pre>//nsIInputStream
var inputStream = Components.classes["@mozilla.org/io/string-input-stream;1"]
                  .createInstance(Components.interfaces.nsIStringInputStream);

inputStream.setData(postData, postData.length);
</pre>
<p>Next, the <code>nsIChannel</code> is QIed to an <code>nsIUploadChannel</code>.  Its <code>setUploadStream</code>
method is called, passing in the <code>nsIInputStream</code> and the type (in this case, "application/x-www-form-urlencoded"):
</p>
<pre>var uploadChannel = gChannel.QueryInterface(Components.interfaces.nsIUploadChannel);
uploadChannel.setUploadStream(inputStream, "application/x-www-form-urlencoded", -1);
</pre>
<p>Due to a bug, calling <code>setUploadStream</code> will reset the <code>nsIHTTPChannel</code> to be a GET request, so now the request type is set to POST:
</p>
<pre>// order important - setUploadStream resets to get/put
httpChannel.requestMethod = "POST";
</pre>
Revert to this revision