Test Results
This section covers test results for the second, third, and fourth deliveries.
Results for the Second Delivery
These results were gathered on 21 December 2002. They reflect the state of the tests and code at that time. As mentioned in the release notes (relnotes.txt), three bugs remain following testing for this delivery. Many other bugs were fixed during formal and informal unit testing.
The tests at the time of these results are mainly functional, but do not fully cover the all the features of the current code. The primary functional unit testing is performed using the JUnit test framework for Java. JUnit testing is lightweight and mostly easy to write, so it has been used for acceptance testing. Acceptance testing in this context means running a suite of lightweight non-regression tests during development every time code is compiled. The theory is that with acceptance testing, you catch many bugs right as they are coded. While the acceptance tests used are not perfect, they have helped prevent slipping back behind where the code is now. JUnit tests are relatively easy to manage, so it was also easy to comment out tests when locating bugs.
The results of the acceptance tests are not very visibly thrilling when things go well. For example, the code currently passes the acceptance tests.
[mcraig@localhost java]$ java org.mcraig.cs445.refentry.AcceptanceTest ...... Time: 9.725 OK (6 tests) [mcraig@localhost java]$ |
The acceptance tests check most of the exposed methods in the org.mcraig.cs445.refentry package. The release notes explain why some methods are not checked. If you are wondering why there are only six tests in the acceptance suite, it is due to how they are packaged. Refer to the code listings in the appendix for details.
In addition to the acceptance tests, a shell script has been used to test the Format tool.
[mcraig@localhost test]$ ./Format.sh
~/cvs/cs445/java ~/cvs/cs445/test
*****************************************************************
Testing simple but valid input and output...
The input is test.xml:
<?xml version="1.0"?>
<!DOCTYPE refentry SYSTEM "../refentry.dtd">
<refentry>
<refmeta>
<refentrytitle>test</refentrytitle>
<manvolnum>test</manvolnum>
</refmeta>
<refnamediv>
<refname>test</refname>
<refpurpose>test RefEntry servlet code</refpurpose>
</refnamediv>
<refsynopsisdiv>
<title>SYNOPSIS</title>
<synopsis>A synopsis.</synopsis>
</refsynopsisdiv>
<refsect1>
<title>REFSECT1</title>
<para>A reference section.</para>
</refsect1>
</refentry>
The output is as follows:
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>test(test) - test RefEntry servlet code</title>
</head>
<body bgcolor="white">
<form>
<input name="SrchString" size="60" value="" />
<input type="submit" value="Search" />
</form>
<hr />
<div>
<table width="100%" border="0">
<tr>
<td align="left">test(test)</td>
<td align="center">Section unknown (see refentry.xslt)</td>
<td align="right">test(test)</td>
</tr>
</table>
<h1>NAME</h1>
<blockquote>
<tt>test</tt> - test RefEntry servlet code</blockquote>
<h1>SYNOPSIS</h1>
<blockquote>
<p>
<tt>A synopsis.</tt>
</p>
</blockquote>
<h1>REFSECT1</h1>
<blockquote>
<p>A reference section.</p>
</blockquote>
</div>
<table border="0" width="100%">
<tr>
<td align="left">Generated [Sat Dec 21 21:53:17 CET 2002]</td>
<td align="right">Server [CLI format]</td>
</tr>
</table>
</body>
</html>
*****************************************************************
This delivery does not support validation inside Format or Build.
You must perform validation yourself using software such as the
Sun Multi-Schema XML Validator written by Kohsuke Kawaguchi.
*****************************************************************
Testing some garbage input...
The input is this test file...
#!/bin/bash
JAVADIR='../java'
COMMAND='java org.mcraig.cs445.refentry.Format'
TESTDIR=`pwd`
pushd $JAVADIR
echo "*****************************************************************"
echo "Testing simple but valid input and output..."
echo "The input is test.xml:"
cat $TESTDIR/test.xml
echo
echo "The output is as follows:"
$COMMAND -infile $TESTDIR/test.xml
echo
echo "*****************************************************************"
echo
echo
echo This delivery does not support validation inside Format or Build.
echo You must perform validation yourself using software such as the
echo Sun Multi-Schema XML Validator written by Kohsuke Kawaguchi.
echo
echo "*****************************************************************"
echo "Testing some garbage input..."
echo "The input is this test file..."
cat $TESTDIR/Format.sh
echo
echo "The output is as follows:"
$COMMAND -infile $TESTDIR/Format.sh
echo
echo "*****************************************************************"
The output is as follows:
javax.xml.transform.TransformerException: org.xml.sax.SAXParseException:\
Content is not allowed in prolog.
java.lang.NullPointerException
javax.xml.transform.TransformerException: org.xml.sax.SAXParseException:\
Content is not allowed in prolog.
java.lang.NullPointerException
javax.xml.transform.TransformerException: org.xml.sax.SAXParseException:\
Content is not allowed in prolog.
java.lang.NullPointerException
javax.xml.transform.TransformerException: org.xml.sax.SAXParseException:\
Content is not allowed in prolog.
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>null(null) - null</title>
</head>
<body bgcolor="white">
<form>
<input name="SrchString" size="60" value="" />
<input type="submit" value="Search" />
</form>
<hr />
<table border="0" width="100%">
<tr>
<td align="left">Generated [Sat Dec 21 21:53:21 CET 2002]</td>
<td align="right">Server [CLI format]</td>
</tr>
</table>
</body>
</html>
*****************************************************************
[mcraig@localhost test]$ |
As mentioned in the output, the Format tool does not at this time perform validation on the input, so the tool does not handle garbage gracefully. Instead, users must currently find their own validator, and ensure their files comply with the DTD before sending them through Format. The parser exception text is not easy to decipher either for a user unfamiliar with the XML specification.
Part of the testing done now is checking the documentation against the code. In other words, checking essentially that the RefEntry documents in section 3 match the javadoc results. Such testing was done using recent javadoc output, and comparing with the contents of this documentation. Bugs were fixed during the testing process.
The final testing done at this time concerns the appearance of the output in various browsers. This testing amounted to inspecting the results in various browsers. The output file used was bigtest.xml, a compendium of nearly everything supported by refentry.dtd.
The following snapshots show some results.
Clearly, the text to be generated in the XRef links did not work at the time of these tests. That should be fixed in a subsequent delivery.
Results for the Third Delivery
These results were gathered on 24 February 2003.
The servlet architecture was refined from delivery two to delivery three. The acceptance tests, functional unit tests that work in the JUnit framework, now run as a build task every time the servlet is built.
The file layout also changed considerably after I read suggestions in the documentation for Apache Cactus (although I ended up not using Cactus). I now deliver all the sources inside the servlet .war file, and the layout reproduces itself that way. The current layout is supposed to reflect a more "standard" layout for web application projects.
This round of system tests and documentation centers on five items:
Documentation accurately and completely explains what you need to do to use the servlet (but not to install it, yet).
The format utility handles garbage gracefully and generates useful diagnostics when problems are encountered.
The build process generates usable diagnostics when things go awry.
The servlet works for a variety of browsers, including the search string in the output.
The servlet handles garbage requests gracefully, complies with servlet standards for portability, and allows multiple, concurrent searches without performance degradation.
The following section explain what was done for each of these items.
Testing Documentation
I wrote the first design documentation at the beginning of the project. Since then, I have refined that design documentation several times to reflect changes made to work around some issue I was not aware of at the outset, or to implement some improvement learned about along the way. The user documentation I wrote at the end, however, working with the servlet in its current state. I have gone back and checked this documentation to verify that the procedures do work.
One step depends mightily on the user, however. At one point, I leave it up to the user to handle making the DTD available to the user's editor, if necessary. I'm not sure how else to explain that, other than to restrict the user to a single editor such as Emacs with PSGML. Users on Windows systems may be comfortable with other XML editors, such as XML Spy from Altera, however. I do not know whether Altera products support use of an SGML_CATALOG_FILES variable or if they handle DTDs some other way.
Testing the format Utility
For delivery two, the format utility was not ready for testing with both valid and invalid documents. For this release it can handle both, so I wrote an acceptance test to check. An example of how it handles both is shown in the user documentation.
It seems to me unnecessary to test this command heavily with a series of separate tests as most of the work is done with the RefEntryTransformer it uses, and additionally by the SAXParser, which throws the errors and warnings needed to debug RefEntry documents. Instead, I run the acceptance test to verify nothing is seriously broken at build time.
Testing the Build Process
In the original design and first draft implementation, I wrote Java code directly to build the servlet. Then I discovered Ant. Ant is a sort of reimplementation of make that looks easier to use and is native to Java. Had I known about it at the outset, I would not have tried to write my own servlet builder. I was trying to avoid make because you often cannot find it on Windows systems.
Ant takes care of the build error messages and many other issues in building the servlet. Once I learned some Ant, it was much easier to break my build process down into bite-sized pieces and to fix small issues I found in the layout. I was also able to integrate the acceptance tests into the build process to run them iteratively as a target, accept, on which the build depends.
accept:
[java] ......
[java] Time: 9.064
[java] OK (6 tests) |
After building the servlet, I install it in the servlet container. I have used Tomcat 4.1.18. Next, I run a few manual tests on the servlet to check that backend, common words, and links work. For example, open the servlet to the default page using the URL, http://localhost:8080/refentry/, then I search for build. This results in a page of links to other entries. Searching for the build does not change that list. From the list, I click the link for Build(3), and check that the link inside that page to build(1) works correctly.
The last of these manual tests caught a bug that I had in my refentry.xslt stylesheet. It turns out I was creating full, rather than relative links. In other words, I was generating static links to http://mcraig.org/refentry/ rather than to pages inside the servlet. Easy to fix, but I might not have seen that without relatively systematic testing.
Finally, I also run post-integration tests to check that the servlet handles both garbage and a couple of acceptable requests without difficulty. These tests are written using the JUnit and HttpUnit test frameworks.
$ java TestGarbageRequests . Time: 4.221 OK (1 test) $ java TestOkayRequests . Time: 3.106 OK (1 test) $ |
If all goes well with those tests, I feel satisfied that the servlet I built basically works.
Testing a Variety of Browsers
To set this up, I installed the servlet on my workstation at work and ran the basic manual tests on Internet Explorer, Netscape Navigator, Mozilla, links, and Galeon. (Galeon is a GNOME browser based on Mozilla.)
Everything seems to function, although I admit my XHTML is ugly. Maybe I need to learn something about attractive page layout. I solemnly swear to do that as soon as I have learned to stop taking my T-shirts of the top of the pile.
Testing Performance
To simulate multiple, simultaneous client access, I put the servlet on my workstation and ran the TestClient.java thread 100 times, more or less in parallel, using TestThreads.java test case from a 10-processor server in our QA lab. I had to modify TestThreads.java slightly to change localhost strings to celadon.france.sun.com, but otherwise I ran the test as is.
The results show that response time degrades as the load increases. I examined the test output by first using UNIX tools to massage it into comma-separated variable format, then using a spreadsheet program to analyze the results. I found the following times:
Table 1. Response Times For 100 "Simultaneous" Threads
| Value | Number of Seconds |
|---|---|
| Average | 21.58 |
| Median | 23.72 |
| Maximum | 26.21 |
| Minimum | 0.34 |
In other words, with 100 "simultaneous" users, the servlet response is more than 10 times too slow. I am not sure what can be done about this, but it is definitely not the required behavior. For now, this problem is being logged as a bug to fix in the release notes. Perhaps it would suffice to change the way Tomcat is configured to handle the servlet. Perhaps implementing SingleThreadModel simply does not scale for many client requests. In that case, improving performance might involve moving the locking to a smaller section of the servlet code.
Results for the Fourth Delivery
The test campaign for this delivery aims to validate as much of the planned functionality as possible, including that checked by functional, system, and installation tests specified above. No new automated test cases were written for this campaign; instead, existing tests were reused, and other conditions checked by hand.
Possibly the most serious failure of the project with respect to initial expectations is failure of the "installation procedures [to be] clear and easy to follow." On Windows systems, users must set environment variables, for example. To my surprise, I found that even some colleagues working day in and day out on Solaris felt uncomfortable setting environment variables on UNIX systems, not to mention Windows systems.
Additionally, auxiliary configuration that could almost be considered installation, such as configuring an XML-aware text editor to use a DTD -- a task I have not described in detail -- proved to cause confusion even for highly qualified software architects, which perhaps explains why so few people edit documents directly in SGML and XML.
Despite those failings, the project software essentially fulfills initial expectations, as documented in the list below.
Final Test Results
- Servlet public and friendly methods respond as documented.
These methods are documented using Javadoc, but also as reference entries. The Javadoc and RefEntry documentation have therefore been compared, and the result has been compared with the design documentation. Where necessary, the documentation was aligned with the implementation, which is different in some aspects from the original specifications (provided with the first delivery).
The methods are unit tested as part of the build process.
- RefEntry document elements are formatted as described in the documentation.
Verifications are indicated in the following table.
Table 2. RefEntry Element Formatting
Element Verified using bigtest.html? arg Yes, in the SYNOPSIS section attribution Yes, as in the quote attributed to Lao Tsu and the other attributed to Lyman Bryson blockquote Yes, as in the two quotes mentioned for the attribution element citerefentry Yes, as shown at the outset of the Paras and Lists section; no check is made to ensure that the target document exists in the servlet, however citetitle Yes, as shown at the outset of the Paras and Lists section classname Yes, as shown at the outset of the Paras and Lists section classsynopsis Yes, as appears in the SYNOPSIS section of the page classsynopsisinfo Yes, also shown in the SYNOPSIS section cmdsynopsis Yes, also shown in the SYNOPSIS section command Yes, as shown at the outset of the Paras and Lists section computeroutput Yes, as shown in the first example of the Example section; come to think of it, this element is probably useless constructorsynopsis Yes, as shown in the SYNOPSIS section destructorsynopsis Yes, also shown in the SYNOPSIS section emphasis Yes, as shown at the outset of the Paras and Lists section entry Yes, as shown in the various tables in the Tables section errorcode Yes, as shown at the outset of the Paras and Lists section errorname Yes, as shown at the outset of the Paras and Lists section errortype Yes, as shown at the outset of the Paras and Lists section example Yes, as shown in the Example section exceptionname Yes, as appears in the SYNOPSIS section fieldsynopsis Yes, also in the SYNOPSIS section funcdef Yes, also in the SYNOPSIS section funcparams Yes, also in the SYNOPSIS section funcprototype Yes, also in the SYNOPSIS section funcsynopsis Yes, also in the SYNOPSIS section funcsynopsisinfo Do not know; have not used this one yet even in bigtest.xml. Formatting as surrounding element is, however, the default behavior during the transformation so this should not be broken. function Yes, as shown in the Paras and Lists section group Yes, as shown in the SYNOPSIS section initializer Yes, as shown in the SYNOPSIS section interfacename Yes, also shown in the SYNOPSIS section itemizedlist Yes, as shown in the Paras and Lists section listitem Yes, as shown in the Paras and Lists section literal Yes, as shown in the first paragraph of the Paras and Lists section manvolnum Yes, as shown in the Paras and Lists section methodname Yes, as shown in the SYNOPSIS section methodparam Yes, also shown in the SYNOPSIS section methodsynopsis Yes, also shown in the SYNOPSIS section modifier Yes, also shown in the SYNOPSIS section msg Yes, as shown in the MsgSet section msgentry Yes, also shown in the MsgSet section msgexplan Yes, also shown in the MsgSet section msginfo Yes, also shown in the MsgSet section msglevel Yes, also shown in the MsgSet section msgmain Yes, also shown in the MsgSet section msgorig Yes, also shown in the MsgSet section msgrel Yes, also shown in the MsgSet section msgset Yes, also shown in the MsgSet section msgsub Yes, also shown in the MsgSet section msgtext Yes, also shown in the MsgSet section ooclass Yes, as shown in the SYNOPSIS section ooexception Yes, also shown in the SYNOPSIS section oointerface Yes, also shown in the SYNOPSIS section orderedlist Yes, as shown in the Paras and Lists section para Yes, as shown all over the place paramdef Yes, as shown in the SYNOPSIS section parameter Yes, also shown in the SYNOPSIS section programlisting Yes, as shown in the Example section quote Yes, as shown in the SYNOPSIS section refentry Yes, the whole thing is a RefEntry refentrytitle Yes, as shown in all the section titles refmeta Yes, as shown at the outset of the document refmiscinfo That's right, it does not appear refname Yes, as shown at the outset of the document refnamediv Yes, as shown at the outset of the document refpurpose Yes, as shown at the outset of the document refsect1 Yes, as shown for each of the top-level sections refsect2 Yes, as shown in the MsgSet and Tables sections refsynopsisdiv Yes, as shown at the top of the document replaceable Yes, as shown in the first paragraph of the Paras and Lists section row Yes, as demonstrated in the Tables sections sbr Yes, as shown in the command synopsis in the SYNOPSIS section screen Yes, as shown in the Example section synopsis Yes, as shown in the SYNOPSIS section, though the handling of whitespace at line ends has not been checked table Yes, as shown in the Tables sections tbody Yes, also shown in the Tables sections term Yes, as shown in the Paras and Lists section tgroup Yes, as shown in the Tables sections thead Yes, also shown in the Tables sections title Yes, as shown throughout the document trademark Yes, as shown in the first paragraph of the Paras and Lists section type Yes, as shown in the SYNOPSIS section ulink Yes, as shown in the first paragraph of the Paras and Lists section userinput Yes, as shown in the Example section varargs Yes, as shown in the SYNOPSIS section variablelist Yes, as shown in the Paras and Lists section varlistentry Yes, also shown in the Paras and Lists section varname Yes, as shown in the SYNOPSIS section void Yes, as shown in the SYNOPSIS section xref Yes, as shown in the first paragraph of the Paras and Lists section - Documentation covers all public features of the clients including RefEntry document development and stylesheet extension.
Originally, the design called for a build client. That functionality has been delegated to Ant, which comes with its own extensive documentation.
The servlet output webman client is documented above, as is the format command. Both are relatively simple, so their documentation is short.
- Configuration file content has the documented effect on build results.
This requirement was originally intended to refer to the Build configuration file. Since the switch to Ant, this requirement has certainly been met for the build (configuration) file, build.xml. Other configuration files, cw.props, refentrys.props, and xslt.props, were documented retrospectively according to how they impact the servlet.
- Build process results in a working servlet .war file if at all possible, otherwise providing good diagnostics for configuration problems.
Before learning about Ant, my intention was to write a Java class by hand to perform the build. The project now depends on Ant for builds. Ant provides clear messages on its own. For example:
$ mv README.txt mvREADME.txt $ ant Buildfile: build.xml init: [mkdir] Created dir: /test/tomcat/webapps/refentry/build [mkdir] Created dir: /test/tomcat/webapps/refentry/build/build [mkdir] Created dir: /test/tomcat/webapps/refentry/build/classes [mkdir] Created dir: /test/tomcat/webapps/refentry/build/copy [mkdir] Created dir: /test/tomcat/webapps/refentry/build/docs [mkdir] Created dir: /test/tomcat/webapps/refentry/build/docs/api [mkdir] Created dir: /test/tomcat/webapps/refentry/build/serial [mkdir] Created dir: /test/tomcat/webapps/refentry/dist compile: [javac] Compiling 20 source files to /test/tomcat/webapps/refentry/build/classes accept: [java] ...... [java] Time: 9.991 [java] OK (6 tests) servletunit: tool: copy: [copy] Copying 1 file to /test/tomcat/webapps/refentry/build/build BUILD FAILED file:/test/tomcat/webapps/refentry/build.xml:64: Warning: Could not find file /test/tomcat/webapps/refentry/README.txt to copy. Total time: 27 seconds- Concurrent searches does not corrupt requests or results, and the servlet returns simple search results withing 2 seconds given a reasonable number of concurrent clients.
The bug for delivery three has been fixed. It turns out that the servlet need not implement SingleThreadModel, because the servlet only reads, never writes global data when handling a client request. My earlier understanding was flawed even after having taking the ACCIS course on programming languages that included explanations of how memory is typically managed, and how it is managed for implementations of the Java language. There are probably still many things about threading I do not understand, but at least now I understand that when a thread executes the code for a method, the thread has memory specifically for local objects created in that code. Those objects are not at risk of being modified by another thread calling the same method code.
The new results for handling client requests using the same performance test as for delivery three:
- Servlet complies with standards for portability.
The web application descriptor complies with the 2.3 version of the DTD, which you can download from Sun Microsystems, Inc. at http://java.sun.com/dtd/web-app_2_3.dtd. In principle, this means none of the exposed functionality is non-standard. In practice, the servlet functionality is so basic that it hardly stretches the limits of that allowed by the 2.3 standard.
- Servlet handles garbage requests gracefully.
This is shown by the TestGarbageRequests.java test.
$ java TestGarbageRequests . Time: 3.744 OK (1 test)
- The browser-based servlet output (webman client) allows input of search strings, echoes search strings in output, and remains readable with a variety of different browsers.
This has been shown in snapshots with the tests for delivery two. I have validated it by hand for the updated servlet.
- Installation procedures are clear and easy to follow. Installation, up to installation of the default servlet, takes no more than 10-15 minutes for a novice.
Catherine Hislam, one of my colleagues, installed the project software on 15 March 2003. It took Catherine 14 minutes from the time she started downloading the installer .jar file, to the time she had the default servlet running on her workstation. Catherine made a number of remarks about the installation, comments I have integrated into the project documentation. For example, she found references to CATALINA_HOME and Tomcat did not originally indicate clearly enough that Tomcat is a servlet container, and that CATALINA_HOME designates the Tomcat installation directory.
Gordon Stretch, another colleague, installed the project software on 25 March 2003, in 13 minutes. Gordon had one remark: that I should replace the example JAVA_HOME path with a variable to prevent the temptation to copy and paste an incorrect path. I fixed this.
Stuart Clements, also a colleague, installed the software on 25 March 2003 as well. It took Stu 18 minutes, but he was interrupted in the middle of the process to look at a bug. Stu gave me feedback on the order of the steps in the installation procedure, and I have fixed that based on his feedback.
- Development and installation of modified servlets works as documented.
This was documented retrospectively, and I used the servlet software itself to generate the listings present in the documentation.
- Installing over an existing copy does not overwrite what has been changed by the user.
The installer configuration file is set to build an installer that overwrites only the top level text files, install.txt, license.txt, README.txt, and relnotes.txt.
This was verified by installing once in an Original directory, once in a Modified directory. After installation, the refentry.profile file was changed in the Modified directory. Then the servlet refentry.war file was copied to the Tomcat webapps directory, Tomcat started, and http://localhost:8080/refentry read to unpack the refentry.war file. Subsequently, a new refentry.war servlet was built after bigtest.xml was added to the list of RefEntry documents. This simulated a user creating a modified servlet. At this point, the differences were captured:
$ diff -r Original Modified > before.diff
Next, the project software was installed atop the files in Modified, the message stating that files might be overwritten was ignored. At this point, the differences were again captured:
$ diff -r Original Modified > after.diff
To demonstrate that the modified files had not been changed upon re-installation, the two difference files were checked for differences:
$ diff before.diff after.diff $
The installer therefore did not overwrite modified files.
- Customized servlets may use customized stylesheets.
The servlet code remains independent of particular stylesheets. For example, all default stylesheets take the extension .xslt:
$ find . -name "*.xslt" ./src/xslt/manvolnum.xslt ./src/xslt/stripxml.xslt ./src/xslt/refentry.xslt ./src/xslt/refname.xslt ./src/xslt/refpurpose.xslt
None of the code contains references to such files, however:
$ find . -name "*.java" -exec egrep -e '\.xslt' {} \; $As documented, the user may change refentry.xslt to support different formatting; such changes are handled transparently by the servlet.
- Improper installation configuration results in useful, intelligible diagnostic messages.
Improper configuration results in diagnostic messages. Whether the messages are useful and intelligible depends on how much the user knows about the software installed. The messages about unset JAVA_HOME and Tomcat installation location environment variables seem relatively clear:
$ unset JAVA_HOME $ unset CATALINA_HOME $ mv tomcat timcat $ . refentry.profile Java Development Kit: JAVA_HOME does not appear to be set. Set JAVA_HOME and run this script again. For example, under bash: export JAVA_HOME=/usr/java/j2sdk1.4.1 Apache Tomcat: Apache Tomcat does not appear to be installed. This project is designed to use Tomcat as the servlet container. If you have an existing installation of Tomcat you want to use, set the CATALINA_HOME environment variable as described in the Tomcat doc and source this file again.Should the installer skip the step of copying the refentry.war servlet file to the proper location, however, the user sees a 404 status message when browsing the servlet location: "The requested resource (/refentry) is not available." This is a Tomcat message, however. Messages from Ant and HttpUnit may be confusing as well to the inexperienced user.
Test result data was gathered through the month of March, 2003.