Running Mochitests on the N800

For the past month or so I’ve been running the mochitest system for Firefox on builds of Fennec (code name of Firefox for mobile project). The mochitest system was written nicely enough to make this transition pretty smooth, requiring just one extra command line argument to get them running on Linux desktop builds (Mac builds force some hacking of the test system). Here are the instructions for building Fennec with tests and the tracking bug for the failures.

Running the same tests on a Nokia N800 device is a bit more troublesome. The memory usage for Fennec hovers right around the 128 MB the device normally has (there is some cool work going on right now to detect low memory and disable features based on this). Top that off with a small test server and mad Javascript libraries and you get a crashing browser.

The mochitest system is a python script that 1) spawns an HTTP server to serve the test files and 2) launches the browser with a fresh profile and special preferences and privileges. It points the browser to a specialized index page with a listing of the test files. Basically, the mochitest system is a webpage (test harness) that loads other webpages (tests) into an iframe one-by-one. These webpages go through different actions, popping up windows, ensuring certain handlers fire, etc. using assertions to communicate with the test harness. when all this is done and the page finishes loading, the main page gets the results of the tests and loads the next page into the iframe (unless you specify that the harness wait for you to signal that the test is finished) . There are thousands of these test webpages in the tree, each of which lives in the same directory as the code it is testing. When Firefox is built, these files are picked up and put in a directory structure where the testing code lives. You can run the entire tree of tests or just one folder.

Running the whole suite at once on the device is not possible right now. As a kind of work-around, I wrote a script to run the entire suite in chunks of small directories. It runs the tests for one directory, logs the results, then kills the browser and moves on to the next one, Finally coalescing the log files into one big log file at the end. This is necessary because some directories use of too much memory and crash the browser or reboot the phone, and others have tests that cause the test system to lose focus on the main test page and thus hang forever. You can kill the script and when it’s run again it will skip the directories that have already been run.

The mochitest system is an important part of keeping Firefox bug-free. When a test case is first written it’s used to verify that a certain feature works as it should or to show others how a bug plays out, but once it’s in the tree it becomes a regression test. The mochitests are run every time the Firefox trunk is built, so as soon as something regresses a bug, we know about it. We need this for Fennec. Regular builds of Fennec are starting to be set up, so hopefully this regression testing will follow behind closely.

I was in particular interested in what tests failed on the phone versus the desktop build and I’ll be analyzing this soon, probably in the tracking bug. Next up: running the chrome and browser-chrome mochitests.

note: the script is my first python script, so I probably wrote it all crappy and perl-like, let me know if you have any suggestions.