Skip to main content

Visit Mozilla.org

Compiled-code automated tests

From MDC


Compiled-code tests written in C++ are capable of testing anything in Mozilla, making them perhaps the most powerful testing method available. However this power comes with a price. These tests can be difficult to write correctly. For web-visible testing, you are nearly always better off with either a Mochitest or a reftest; for unit testing, you should write an xpcshell test if the functionality being tested is available through scriptable methods and interfaces.

If the functionality you wish to test is only available to compiled code, however, a compiled-code test is the right choice.

Contents

[edit] The structure of a compiled-code test

Using XPCOM requires including a number of headers and writing some tedious initialization code, under normal circumstances. For a test, however, most of this nonsense can be eliminated by including the test-support header xpcom/tests/TestHarness.h. This header provides a scoped-XPCOM class, the smart pointer classes, do_CreateInstance, and stdio.h/stdlib.h. With that, the basic structure of a compiled-code test is reduced to the following: 

// standard Mozilla tri-license

#include "TestHarness.h"

nsresult TestFoo()
{
  // do your stuff and either fail in there or fall through
  printf("TestFoo PASSED!\n");
  return NS_OK;
}

nsresult TestOtherFoo()
{
  printf("TestOtherFoo PASSED!\n");
  return NS_OK;
}

int main(int argc, char** argv)
{
  ScopedXPCOM xpcom("FooTests"); // name for tests being run
  if (xpcom.failed())
    return 1;

  // XPCOM is now started up, and you can access components/services/etc.

  int rv = 0;

  if (NS_FAILED(TestFoo()))
    rv = 1;
  if (NS_FAILED(TestOtherFoo()))
    rv = 1;
  // ...

  return rv;
}

If you're fixing bugs in compiled code, you can probably figure out how to fill out the rest of this skeleton.

There are a couple things to note that aren't demonstrated by this skeleton. First, note that a test failure doesn't cause the test to immediately fail. While it's possible to write tests that fail as early as possible, usually it's possible to run multiple tests and only report failure at the end, which can save the frustrated developer who breaks your test the trouble of fixing each failure individually. Second, a test which fails should do so by printing an error message and returning a failure code. When possible, the error should be printed as early as possible and propagated up the call stack without printing further errors (note how in the example the TestFoo() doesn't have a corresponding printed message, because TestFoo's definition should do so before returning a failure code), but if an error must be printed multiple times, that's fine. Error messages should begin with the string "FAIL " (one trailing space) to enable better future processing of errors in, for example, tinderbox logs. Third, it's fine to print out pass messages such as the one at the end of TestFoo(), but if you're printing more than 10-15 you should consider being more selective in what you print. The intent is to give information about test progress but not to overwhelm the screen or tinderbox logs.

[edit] Making the test compile and link

Once you've written your test, you need to hook it up to the build system. You'll first need to make sure xpcom/tests/TestHarness.h is found correctly. You can do this by adding the following lines to the Makefile.in in the directory containing your test: 

CPP_UNIT_TESTS = TestFoo.cpp

LOCAL_INCLUDES += -I$(topsrcdir)/xpcom/tests
CPPSRCS += $(CPP_UNIT_TESTS)
SIMPLE_PROGRAMS += $(CPP_UNIT_TESTS:.cpp=$(BIN_SUFFIX))
REQUIRES += xpcom
LIBS += $(XPCOM_LIBS) $(NSPR_LIBS)

You can hook up any other headers you need through the standard methods like REQUIRES. Note that because your test will be run in non-debug trees and tinderboxen, you should ensure your code builds in both debug and optimized configurations so that you don't rely on non-exported symbols libxul hides from you.

[edit] Hooking your test up to run automatically

Now that you have the test, it's time to make sure it works. We may eventually have Makefile magic to run the tests automatically, but for now, just add the following lines to the Makefile.in: 

# to the check:: target in the file, making sure to preserve the tab-indents...
check::
	@$(EXIT_ON_ERROR) \
	for f in $(subst .cpp,,$(CPP_UNIT_TESTS)); do \
	  XPCOM_DEBUG_BREAK=stack-and-abort $(RUN_TEST_PROGRAM) $(DIST)/bin/$$f; \
	done

Again, there are a couple things to note here. First, the EXIT_ON_ERROR causes a non-zero return value to break make check; without it the failure is ignored. Second, the tests are run with XPCOM_DEBUG_BREAK set so that assertions are fatal. You must ensure any tests you commit run correctly, without generating any assertions, so that your tests don't break the build. (Currently the tests aren't run in debug builds where the assertions would be tested, but this will hopefully change sometime soon.)

[edit] Functionality provided by TestHarness.h

Currently the only thing beyond useful includes is the ScopedXPCOM class; the functionality it provides is minimal enough (and is fully demonstrated above) that just reading the header is easier than describing it. More functionality for standardizing logging and error messages may be added at some later time.

[edit] Running the tests

Just run them using the standard make -C $OBJDIR/path/to/containing/folder check command.

Retrieved from "http://developer.mozilla.org/en/docs/Compiled-code_automated_tests"