Try not to use Mochitest
Yes, really. For many things Mochitest is overkill. In general you should always try to use one of the lighterweight testing frameworks. For example, if you only want to test a single XPCOM component then you should use xpcshell. On the other hand there are some things that Mochitest cannot do, or isn't designed to do. For example, for visual output tests you should try to use the reftest framework. For more information on the different types of automated testing frameworks see Mozilla automated testing.
The Mozilla tinderboxes now run Mochitest as part of the build process, so anyone committing changes to the source will know pretty quickly if they've broken something. Nevertheless, it is a good idea to run the Mochitest before committing new code. This is especially true if the code changes are large or would be difficult to back out.
To run Mochitest you need to build Mozilla with your changes, change directory to $(OBJDIR)/_tests/testing/mochitest, and run the command:
> perl runtests.pl
This will open your new build with a document containing a "Run Tests" link at the top. To run the tests simply click this link and watch the results being generated. If any tests have failed this will be reported.
Writing new Mochitests
A basic document on Writing MochiTest-based unit tests has already been started. (Maybe the basic stuff from that document should be moved here and that document should give the matter more detailed treatment?)
Adding new Mochitests
Once you've written a new test you need to add it to the Mozilla source tree and tell the build system about it so that the Mozilla tinderboxes will run it automatically.
New Mochitest tests should go somewhere close to the code they are testing. For example, if you create a new test for some HTML feature, you probably want to put the test in the directory mozilla/content/html/content/test or mozilla/content/document/content/test. If a test directory does not exist near the code you are testing you can add a new test directory as the patch in bug 368531 demonstrates.
To tell the build system about your new test you need to add the name of your test file to _TEST_FILES in the test directory's Makefile.in.
Before committing your new test and the Makefile.in changes, do run Mochitest in an up to date trunk build to check that you will not unexpectedly turn the tree orange.
For answers to frequently asked Mochitest questions see the Mochitest FAQ. (Information between these two documents could really do with a lot of reorganization.)