So, you find that you need to develop a test. Perhaps you want to write tests. Perhaps you are not so enthusiastic and, really, you want to get back to writing "real" code. That is ok. You are probably being asked to write tests. But really this is not about you. It is about, or should be about, changes that other people are going to make to your code later. You should want to write a test so that, if anyone tries to change your code in the future and screws it up, they and you and others will know about it as soon as possible.
There are several test harnesses that you can use to write a test, depending on what you are testing:
- reftest - create two different html files that should look the same in browser. Do they?
- stand-alone - using XPCOM and command-line? These are for you.
Tests can be written from different parts of the Mozilla system. Different kinds of tests need to be developed to test different things. As a developer, you should ask yourself what kinds of things make the most sense to you and where you are most comfortable experimenting. Developing tests may be a way to learn how to develop new functionality in Mozilla, but it is something else also. If you are thinking about testing, you should not think about only the obvious ways to do things. We need to stress the common ways that things are done, but we need to walk the lesser trodden paths as well.
There are different approaches to take in learning about the Mozilla environment. The main "DevMo" site, http://developer.mozilla.org, is your most important resource. Make sure that, as your vocabulary increases, as you learn the terms used to describe parts of the system, you remember to search for terms again. You may have found some resources in the site, but you have not found all of the pages that will help you.
It is important to remember to keep looking around to see what is available. Remember to check the "Recent Changes" page above. It will list the things that are being changed right now, which will tell you what issues people are concerned about and it will point you to people who may be willing to discuss an issue and may want to help you make an impact on an issue that interests you.
Bugs are an important source of information. Some people believe that bugs are the primary documentation of all Mozilla functionality. But it is not always easy to read bugs and make sense of the problem. There are links on the front of the Bugzilla site to "Hot Bugs" or "Bugs Filed Today". Keep in mind that the bugs one sees in these reports may not be the easiest to understand.
The "Describe Keywords" link above may be valuable. Looking at bugs may not help find the "big picture" problems, but the keywords point to some of the cross-cutting issues. Once one finds a bug or bugs that one is interested in, one can look at other bugs in that Component. As you read bugs, some will make more sense than others. Read other bugs originated by the people who write bugs that make sense to you. These people may be able to answer e-mails and may be available on the irc channels on the IRC site listed above.
It may help to change the set of columns in the bug lists. One does this by going to the bottom of a list of bugs and hitting the "Change Columns" link. The default set of columns may not be the most helpful for new users of the system. Try including 'Component' and 'Changed', which will help show which bugs are recently edited, to see the list in a different way. The default list of columns works for some people and not for others.
There are a few ways to find HTML and web pages that cause problems. The reports in the "Reporter" site list web pages for which people have reported problems. Also, one can search for bugs that have html in their attachments. It may not be very obvious how one does this. Go to Bugzilla, hit the Search link and switch to the "Advanced Search" pane. All the way at the bottom of the page is section titled "Advanced Searching Using Boolean Charts". Either select "Attachment is a URL" from the first pull-down menu, or select from all three to say "Attachment mime type" "is equal to" "text/html".
If you are a visual person, a visual thinker, or if your primary strength is using HTML and CSS, you may want to focus on developing tests with reftest. In reftest, one can test a fragment of HTML/CSS by putting it in a "test" page, and then creating other HTML/CSS that is, presumably, simpler and produces the same visual effect. This should go into a "reference" page, and the reftest harness verifies that these two pages render the same. This approach may seem odd at first, but if you think about HTML, there is always more than one way to achieve a given visual effect. This is even more true with HTML and CSS.
Just to give an extremely simple example, what are the different kinds of HTML that will produce a single space in a web page? Looking at the HTML 4.01 spec, one can see 4 different space characters and one can see that any arbitrary series of these four characters collapses to one visible space character. There are many test pages that can be written to check only against a reference page that contains nothing but a character, a space, and then another character.
An extremely simple reftest would consist of two html pages.
A line would then be included in one of the existing "reftest.list" files that would say:
== space-collapses-test.html space-collapses-ref.html
When the reftests are then run, the two files would be rendered and it would be confirmed that two spaces do indeed collapse to the same visual display as one non-breaking space.
See the 'Automated Testing' category link in devmo above to find other documents that describe reftest.
Mochitest and xpcshell tests
If you are already comfortable with working in the lower layers of the system and are going to work in xpcshell, you may want to skip ahead to some of the documentation referenced above.
One can write something as a reftest:
<!DOCTYPE html> <html><head><title>Color Check</title></head> <body> <p id="gray">I am some color. Which? I do not know!</p> <script> var p = document.getElementById("gray"); p.style.color="gray"; </script> </body>
<!DOCTYPE html> <html><head><title>Color Check</title></head> <body> <p id="gray">I am some color. Which? I do not know!</p> <script> var p = document.getElementById("gray"); p.style.color="#808080"; </script> </body>
But the fact that it was possible to write this with a script and not just with a static style element should give a hint to the lesson here. It is also possible to do this much more efficiently with MochiTest. Indeed, I was able to test all the colors specified in the SVG specification, checking that the color names metched with the expected hex values and rgb values also, all in just one script:
It is important to note that MochiTest cannot be used for everything. The test above is checking that the logical values of the colors assigned match what they should. This is not completely guaranteeing that the colors display correctly. But a test targeted at just the color and that the color displays correctly would be very small and there would only need to be one test that verified that. Without a bit of discipline, many tests could waste time re-checking this fact.
Test-only JS modules are declared in the Makefile by defining the
TESTING_JS_MODULES and optionally the
TESTING_JS_MODULES_DIR variable. Here is an example:
DEPTH = ../.. topsrcdir = @top_srcdir@ srcdir = @srcdir@ include $(DEPTH)/config/autoconf.mk TESTING_JS_MODULES = mockserver.js common.js TESTING_JS_MODULES_DIR = foobar
Makefile will result in the files
common.js from the directory containing the Makefile to be copied into the
foobar directory of the test-only modules destination. If
TESTING_JS_MODULES_DIR is not defined, files will be installed to the root directory.
Once you have installed test-only modules, they can be imported via the
resource://testing-common/ URL prefix. For example:
Still To Come....
- XULRunner application testing
- Extension testing
If you want to learn how to develop in the Mozilla environment and are developing tests as a means to that end, that is a good thing. It is a good way to learn. But you should be aware that test code is sometimes not the easiest code to figure out. It often does things in an unusual way. This is sometimes the point.
So, look at the code of the other tests. Go to http://dxr.mozilla.org/ and search for the word "test". Look at the code you are trying to test and look for other code, findable around the Internet, that does things you want to try to do. Go to http://dxr.mozilla.org/ and browse. Click on the links that take you from one source to an identifier in another source, and another, and another. Be patient.
Remember that things may look odd, or just wrong, but this system works and things are usually the way they are for some reason. The reason may not apply any longer, but you will not know that at first. There is a great deal of stored "tribal" knowledge in the Mozilla community. There are a lot of things that people with experience just know, and they may not realize that these things need to be explained. Go to http://developer.mozilla.org and read the documentation. If you find something that does not make sense, please edit the page until it does.