OS.File.DirectoryIterator for the main thread

This article needs a technical review. How you can help.

This document presents the mechanisms for walking the contents of a directory from the main thread. For other uses of OS.File and OS.File.DirectoryIterator, please see the main document on OS.File.

Using OS.File.DirectoryIterator

Example: Finding the subdirectories of a directory using an iterator (TBD)

The following snippet walks through the content of a directory, selecting subdirectories:

// Open iterator
let iterator = new OS.File.DirectoryIterator(somePath);
let subdirs = [];
// Iterate through the directory
let promise = iterator.forEach(
  function onEntry(entry) {
    if (entry.isDir) {
      subdirs.push(entry);
    }
  }
);

// Finally, close the iterator
promise.then(
  function onSuccess() {
    iterator.close();
    return subdirs;
  },
  function onFailure(reason) {
    iterator.close();
    throw reason;
  }
);

Or a variant using Task.js (or at least the subset already present on mozilla-central):

Task.spawn(function() {
  let iterator = new OS.File.DirectoryIterator(somePath);
  let subdirs = [];
  // Loop through all entries
  // (the loop ends with an exception).
  while (true) {
    let entry = yield iterator.next();
    subdirs.push(entry);
  }
}).then(
  null,
  // Finally, clean up
  function onFailure(reason) {
    iterator.close();
    if (reason != StopIteration) {
      throw reason;
    }
  }
);

Example: Sorting files by last modification date

The following takes advantage of feature detection to minimize file I/O:

let iterator = new OS.File.DirectoryIterator(somePath);
let entries = [];
let promise = iterator.forEach(
  function onEntry(entry) {
    if ("winLastWriteDate" in entry) {
      // Under Windows, additional information allows us to sort files immediately
      // without having to perform additional I/O.
      entries.push({entry: entry, creationDate: entry.winCreationDate});
    } else {
      // Under other OSes, we need to call OS.File.stat
      return OS.File.stat(entry.path).then(
        function onSuccess(info) {
          entries.push({entry:entry, creationDate: info.creationDate});
        }
      );
    }
  }
);
promise.then(
  function onSuccess() {
    // Close the iterator, sort the array, return it
    iterator.close();
    return entries.sort(function compare(a, b) {
      return a.creationDate - b.creationDate;
    });
  },
  function onFailure(reason) {
     // Close the iterator, propagate any error
    iterator.close();
    throw reason;
  }
);

Or a variant using Task.js (or at least the subset already present on mozilla-central):

let iterator = new OS.File.DirectoryIterator(somePath);
let entries = [];
Task.spawn(function(){
  while (true) {
    let entry = yield iterator.next();
    if ("winLastWriteDate" in entry) {
      // Under Windows, additional information allows us to sort files immediately
      // without having to perform additional I/O.
      entries.push({entry: entry, creationDate: entry.winCreationDate});
    } else {
      // Under other OSes, we need to call OS.File.stat
      let info = yield OS.File.stat(entry.path);
      entries.push({entry:entry, creationDate: info.creationDate});
    }
  }
}).then(
  null,
  // Clean up and return
  function onFailure(reason) {
    iterator.close();
    if (reason != StopIteration) {
      throw reason;
    }
    return entries.sort(function compare(a, b) {
      return a.creationDate - b.creationDate;
    });
  }
);

Example: Iterating through all child entries of directory

This example uses the Deferred function from here: Deferred. Add-SDK devs should use the core/promise defer. For example var deferred_enumChildEntries = new Deferred(); should be replaced with var deferred-enumChildEntries = require('core/promise').defer(), see here: core/promise :: defer. Also, the Promise.all for Addon-SDK developers should also be replaced with core/promise :: all, for example, the code below of var promiseAll_itrSubdirs = Promise.all(promiseArr_itrSubdirs); should be rewritten as var promiseAll_itrSubdirs = require('sdk/core/promise').all(promiseArr_itrSubdirs);.

The following function, enumChildEntries takes in a parameter of directory path and a callback known as delegate. It will go throgh all subdirectories and files contained till it reaches max_depth. The depth argument is for use by the function to track recursion. If the delegate returns true, it stops enumerating children.

function enumChildEntries(pathToDir, delegate, max_depth, runDelegateOnRoot, depth) {
	// IMPORTANT: as dev calling this functiopn `depth` arg must ALWAYS be undefined (dont even set it to 0 or null, must completly omit setting it, or set it to undefined). this arg is meant for internal use for iteration
	// `delegate` is required
	// pathToDir is required, it is string
	// max_depth should be set to null/undefined if you want to enumerate till every last bit is enumerated. paths will be iterated to including max_depth.
	// if runDelegateOnRoot, then delegate runs on the root path with depth arg of -1
	// this function iterates all elements at depth i, then after all done then it iterates all at depth i + 1, and then so on
	// if arg of `runDelegateOnRoot` is true then minimum depth is -1 (and is of the root), otherwise min depth starts at 0, contents of root
	// if delegate returns true, it will stop iteration
	// if set max_depth to 0, it will just iterate immediate children of pathToDir, unlesss you set runDelegateOnRoot to true, then it will just run delegate on the root
	var deferredMain_enumChildEntries = new Deferred();
	
	if (depth === undefined) {
		// at root pathDir
		depth = 0;
		if (runDelegateOnRoot) {
			var entry = {
				isDir: true,
				name: OS.Path.basename(pathToDir),
				path: pathToDir
			};
			var rez_delegate = delegate(entry, -1);
			if (rez_delegate) {
				deferredMain_enumChildEntries.resolve(entry);
				return deferredMain_enumChildEntries.promise; // to break out of this func, as if i dont break here it will go on to iterate through this dir
			}
		}
	} else {
		depth++;
	}
	
	if ((max_depth === null || max_depth === undefined) || (depth <= max_depth)) {
		var iterrator = new OS.File.DirectoryIterator(pathToDir);
		var subdirs = [];
		var promise_batch = iterrator.nextBatch();
		// :TODO: iterrator.close() somewhere!! maybe here as i dont use iterrator anymore after .nextBatch()
		promise_batch.then(
			function(aVal) {
				iterrator.close();
				console.log('Fullfilled - promise_batch - ', aVal);
				// start - do stuff here - promise_batch
				for (var i = 0; i < aVal.length; i++) {
					if (aVal[i].isDir) {
						subdirs.push(aVal[i]);
					}
					var rez_delegate_on_child = delegate(aVal[i], depth);
					if (rez_delegate_on_child) {
						deferredMain_enumChildEntries.resolve(aVal[i]);
						return/* deferredMain_enumChildEntries.promise -- im pretty sure i dont need this, as of 040115*/; //to break out of this if loop i cant use break, because it will get into the subdir digging, so it will not see the `return deferredMain_enumChildEntries.promise` after this if block so i have to return deferredMain_enumChildEntries.promise here
					}
				}
				// finished running delegate on all items at this depth and delegate never returned true

				if (subdirs.length > 0) {
					var promiseArr_itrSubdirs = [];
					for (var i = 0; i < subdirs.length; i++) {
						promiseArr_itrSubdirs.push(enumChildEntries(subdirs[i].path, delegate, max_depth, null, depth)); //the runDelegateOnRoot arg doesnt matter here anymore as depth arg is specified
					}
					var promiseAll_itrSubdirs = Promise.all(promiseArr_itrSubdirs);
					promiseAll_itrSubdirs.then(
						function(aVal) {
							console.log('Fullfilled - promiseAll_itrSubdirs - ', aVal);
							// start - do stuff here - promiseAll_itrSubdirs
							deferredMain_enumChildEntries.resolve('done iterating all - including subdirs iteration is done - in pathToDir of: ' + pathToDir);
							// end - do stuff here - promiseAll_itrSubdirs
						},
						function(aReason) {
							var rejObj = {name:'promiseAll_itrSubdirs', aReason:aReason};
							rejObj.aExtra = 'meaning finished iterating all entries INCLUDING subitering subdirs in dir of pathToDir';
							rejobj.pathToDir = pathToDir;
							console.error('Rejected - promiseAll_itrSubdirs - ', rejObj);
							deferredMain_enumChildEntries.reject(rejObj);
						}
					).catch(
						function(aCaught) {
							var rejObj = {name:'promiseAll_itrSubdirs', aCaught:aCaught};
							console.error('Caught - promiseAll_itrSubdirs - ', rejObj);
							deferredMain_enumChildEntries.reject(rejObj);
						}
					);
				} else {
					deferredMain_enumChildEntries.resolve('done iterating all - no subdirs - in pathToDir of: ' + pathToDir);
				}
				// end - do stuff here - promise_batch
			},
			function(aReason) {
				iterrator.close();
				var rejObj = {name:'promise_batch', aReason:aReason};
				if (aReason.winLastError == 2) {
					rejObj.probableReason = 'directory at pathToDir doesnt exist';
				}
				console.error('Rejected - promise_batch - ', rejObj);
				deferredMain_enumChildEntries.reject(rejObj);
			}
		).catch(
			function(aCaught) {
				iterrator.close();
				var rejObj = {name:'promise_batch', aCaught:aCaught};
				console.error('Caught - promise_batch - ', rejObj);
				deferredMain_enumChildEntries.reject(rejObj);
			}
		);
	} else {
		deferredMain_enumChildEntries.resolve('max depth exceeded, so will not do it, at pathToDir of: ' + pathToDir);
	}

	return deferredMain_enumChildEntries.promise;
}

//start - helper function
function Deferred() {
	if (Promise.defer) {
		//need import of Promise.jsm for example: Cu.import('resource:/gree/modules/Promise.jsm');
		return Promise.defer();
	} else if (PromiseUtils.defer) {
		//need import of PromiseUtils.jsm for example: Cu.import('resource:/gree/modules/PromiseUtils.jsm');
		return PromiseUtils.defer();
	} else {
		/* A method to resolve the associated Promise with the value passed.
		 * If the promise is already settled it does nothing.
		 *
		 * @param {anything} value : This value is used to resolve the promise
		 * If the value is a Promise then the associated promise assumes the state
		 * of Promise passed as value.
		 */
		this.resolve = null;

		/* A method to reject the assocaited Promise with the value passed.
		 * If the promise is already settled it does nothing.
		 *
		 * @param {anything} reason: The reason for the rejection of the Promise.
		 * Generally its an Error object. If however a Promise is passed, then the Promise
		 * itself will be the reason for rejection no matter the state of the Promise.
		 */
		this.reject = null;

		/* A newly created Pomise object.
		 * Initially in pending state.
		 */
		this.promise = new Promise(function(resolve, reject) {
			this.resolve = resolve;
			this.reject = reject;
		}.bind(this));
		Object.freeze(this);
	}
}
// end - helper function
 Example of enumChildEntries
/************ start usage **************/
var totalEntriesEnummed = 0; //meaning total number of entries ran delegate on, includes root dir
function delegate_handleEntry(entry) {
  // return true to make enumeration stop
  totalEntriesEnummed++;
  console.info('entry:', entry);
}

var pathToTarget = OS.Constants.Path.desktopDir;
var promise_enumEntries = enumChildEntries(pathToTarget, delegate_handleEntry, null /*to get all*/, false);
promise_enumEntries.then(
  function(aVal) {
    // on resolve, aVal is undefined if it went through all possible entries
    console.log('Fullfilled - promise_enumEntries - ', aVal);
    console.info('totalEntriesEnummed:', totalEntriesEnummed)
  },
  function(aReason) {
    console.error('Rejected - promise_enumEntries - ', aReason);
    throw {rejectorOf_promiseName:'promise_enumEntries', aReason:aReason};
  }
).catch(
  function(aCatch) {
    console.error('Caught - promise_enumEntries - ', aCatch);
    throw aCatch;
  }  
);
Another Example
var delegate = function(aEntry, aDepth) {
  console.info('aDepth:', aDepth, 'aEntry:', aEntry.name);
}

enumChildEntries(OS.Path.join(OS.Constants.Path.desktopDir, 'p'), delegate, null, true).then(
  x => console.log('x:', x),
  y => console.error('y:', y)
).catch(
  z => console.error('z:', z)
);

Example: Make a copy of a directory (uses enumChildEntries from above)

This example uses the Deferred function from here: Deferred. Add-SDK devs should use the core/promise defer. For example var deferred_enumChildEntries = new Deferred(); should be replaced with var deferred-enumChildEntries = require('core/promise').defer(), see here: core/promise :: defer. Also, the Promise.all for Addon-SDK developers should also be replaced with core/promise :: all, for example, the code below of var promiseAll_itrSubdirs = Promise.all(promiseArr_itrSubdirs); should be rewritten as var promiseAll_itrSubdirs = require('sdk/core/promise').all(promiseArr_itrSubdirs);.

The following function, duplicateDirAndContents uses the above function of enumChildEntries to duplicate a directory, can set how deep the copy should go.

// start - helper function for duplicateDirAndContents
function escapeRegExp(text) {
    if (!arguments.callee.sRE) {
        var specials = ['/', '.', '*', '+', '?', '|', '(', ')', '[', ']', '{', '}', '\\'];
        arguments.callee.sRE = new RegExp('(\\' + specials.join('|\\') + ')', 'g');
    }
    return text.replace(arguments.callee.sRE, '\\$1');
}
// end - helper function for duplicateDirAndContents

function duplicateDirAndContents(pathToSrcDir, pathToDestDir, max_depth, targetDirExists) {
	// returns promise
	// copies all stuff at depth i, then does depth i + 1, then i + 2 depth, so on // does not start at depth i and if subdir found it doesnt start copying into that right away, it completes depth levels first, i should make this change in future though as enhancement
	// if targetDirExists mark as true, else, set to false. if you set to true when it does not exist, then promise will reject due to failing to copy to non-existant dir. if it does exist, and you set it to false, then you are just wasting a couple extra function calls, function will complete succesfully though, as it tries to make the dir but it will not overwrite if already found

	var deferred_duplicateDirAndContents = new Deferred();
	var promise_duplicateDirAndContents = deferred_duplicateDirAndContents.promise;

	var stuffToMakeAtDepth = [];
	var smallestDepth = 0;
	var largestDepth = 0;

	var delegate_handleEntry = function(entry, depth) {
		// return true to make enumeration stop
		if (depth < smallestDepth) {
			smallestDepth = depth;
		}
		if (depth > largestDepth) {
			largestDepth = depth;
		}
		stuffToMakeAtDepth.push({
			depth: depth,
			isDir: entry.isDir,
			path: entry.path
		});
	};

	var promise_collectAllPathsInSrcDir = enumChildEntries(pathToSrcDir, delegate_handleEntry, max_depth, !targetDirExists);
	
	promise_collectAllPathsInSrcDir.then(
		function(aVal) {
			console.log('Fullfilled - promise_collectAllPathsInSrcDir - ', aVal);
			// start - do stuff here - promise_collectAllPathsInSrcDir
			// start - promise generator func
			var curDepth = smallestDepth;
			var makeStuffsFor_CurDepth = function() {
				var promiseAllArr_madeForCurDepth = [];
				for (var i = 0; i < stuffToMakeAtDepth.length; i++) {
					if (stuffToMakeAtDepth[i].depth == curDepth) {
						var copyToPath = stuffToMakeAtDepth[i].path.replace(new RegExp(escapeRegExp(pathToSrcDir), 'i'), pathToDestDir);
						promiseAllArr_madeForCurDepth.push(
							stuffToMakeAtDepth[i].isDir // if (stuffToMakeAtDepth[i].isDir) {
							?
								OS.File.makeDir(copyToPath)
							: // } else {
								OS.File.unixSymLink(stuffToMakeAtDepth[i].path, stuffToMakeAtDepth[i].path.replace(new RegExp(escapeRegExp(pathToSrcDir), 'i'), pathToDestDir))
								//OS.File.copy(stuffToMakeAtDepth[i].path, copyToPath)
							// }
						);
					}
				}
				var promiseAll_madeForCurDepth = Promise.all(promiseAllArr_madeForCurDepth);
				promiseAll_madeForCurDepth.then(
					function(aVal) {
						//console.log('Fullfilled - promiseAll_madeForCurDepth - ', aVal);
						// start - do stuff here - promiseAll_madeForCurDepth
						if (curDepth < largestDepth) {
							curDepth++;
							makeStuffsFor_CurDepth();
						} else {
							deferred_duplicateDirAndContents.resolve('all depths made up to and including:' + largestDepth);
						}
						// end - do stuff here - promiseAll_madeForCurDepth
					},
					function(aReason) {
						var rejObj = {name:'promiseAll_madeForCurDepth', aReason:aReason};
						console.error('Rejected - promiseAll_madeForCurDepth - ', rejObj);
						deferred_duplicateDirAndContents.reject(rejObj);
					}
				).catch(
					function(aCaught) {
						var rejObj = {name:'promiseAll_madeForCurDepth', aCaught:aCaught};
						console.error('Caught - promiseAll_madeForCurDepth - ', rejObj);
						deferred_duplicateDirAndContents.reject(rejObj);
					}
				);
			};
			// end - promise generator func
			makeStuffsFor_CurDepth();
			// end - do stuff here - promise_collectAllPathsInSrcDir
		},
		function(aReason) {
			var rejObj = {name:'promise_collectAllPathsInSrcDir', aReason:aReason};
			console.error('Rejected - promise_collectAllPathsInSrcDir - ', rejObj);
			deferred_duplicateDirAndContents.reject(rejObj);
		}
	).catch(
		function(aCaught) {
			var rejObj = {name:'promise_collectAllPathsInSrcDir', aCaught:aCaught};
			console.error('Caught - promise_collectAllPathsInSrcDir - ', rejObj);
			deferred_duplicateDirAndContents.reject(rejObj);
		}
	);

	return promise_duplicateDirAndContents;
}
Example Usage of duplicateDirAndContents
var pathToTarget = OS.Path.join(OS.Constants.Path.desktopDir, 'trgt folder');
var pathToCreate = OS.Path.join(OS.Constants.Path.desktopDir, 'deep copied dir'); // does not have to exist, but if it doesnt, then make sure to pass last argument of duplicateDirAndContents of `targetDirExists` as false or null or undefined. dont pass true, otherwise you lied to it and it will reject as the destintation dir doesnt exist

var promise_dupeTrgFol = duplicateDirAndContents(pathToTarget, pathToCreate, 0, false);
promise_dupeTrgFol.then(
    function(aVal) {
        console.log('Fullfilled - promise_dupeTrgFol - ', aVal);
        return 'promise for enumChildEntries completed succesfully';
    },
    function(aReason) {
        var rejObj = {
            promiseName: 'promise_dupeTrgFol',
            aReason: aReason
        };
        console.error('Rejected - ' + rejObj.promiseName + ' - ', rejObj);
    }
).catch(
    function(aCaught) {
        console.error('Caught - promise_dupeTrgFol - ', aCaught);
    }
);

Instances of OS.File.DirectoryIterator

General remark

Instances of OS.File.DirectoryIterator use valuable system resources – typically the same resources as files. There is a limit to the total number of files and directory iterators that can be open at any given time. Therefore, you should make sure that these resources are released as early as possible. To do this, you should use method close().

Constructor

OS.File.DirectoryIterator(
  in string path,
  [optional] in object options
) throws OS.File.Error
Arguments
path
The complete path to a directory.
options
An object that may contain the following fields:
winPattern
(ignored on non-Windows platforms) This option accepts a pattern such as "*.exe" or "*.txt". The iterator will only display items whose name matches this pattern.

Method overview

void close()
promise<void> forEach(in function callback)
promise<Array> nextBatch([optional] in number length)
promise<Entry> next()

Methods

close()

Close the iterator, releasing the system resources it uses.

void close()

You should always call this method once you are done with an iterator. Calling this method several times is harmless. Calling this method while iterating through the directory is harmless but will stop the iteration.

forEach()

Walk through the iterator

promise<void> forEach(
  in function callback
)
Arguments
Promise resolves to

Nothing - once the loop is complete

Promise rejects to

 

Promise-based loop

Iteration takes place sequentially: the callback is called with the first file, then once the callback is complete with the second file, etc.

It is often quite useful to be able to return a promise. If the callback returns a promise, the loop continues only once the callback is resolved.

If any of the callbacks throws an error or rejects a promise, the loop is stopped and rejects with the same error.
callback
A function. It will be applied to each entry in the directory successively, with the following arguments:
entry
An instance of OS.File.DirectoryIterator.Entry.
index
The index of the entry in the enumeration.
iterator
The iterator itself. You may stop the iteration by calling iterator.close().
OS.File.Error
In case of I/O error.
Anything else
In case one of the callbacks throws an error or rejects.

nextBatch()

Return several entries at once.

promise<array> nextBatch(
  [optional] in number entries
)
Arguments
entries
The number of entries to return at once. If unspecified, all entries.
Promise resolves to

An array containing entries entries, or less if there are not enough entries left in the directory. Once iteration is complete, calls to nextBatch return the empty array.

Performance note

In some circumstances, you may wish to use this method instead of forEach or next for performance reasons, as it limits the number of synchronizations between threads.

next()

Return the next entry in the directory.

promise<Entry> next()
Promise resolves to

The next entry in the directory.

Promise rejects to
OS.File.Error
In case of file error.
StopIteration
If the method is called after the last entry has been returned. Note that this is the constant StopIteration, not a constructor.

Document Tags and Contributors

 Contributors to this page: Noitidart, teoli, arai, kscarfone, Yoric
 Last updated by: Noitidart,