Processing XML using Java

1.1. What is XML?

• Originally, details of how a WWW page was displayed was left to the browser.
• HTML has evolved: author has more control over the appearance.
• So the HTML of a WWW page is a mix of data and presentation instructions.

<table border=1>
   <tr>
      <td><b>inkjet cartridges</b></td>
      <td><em>HP Deskjet print cartridge 51645A</em></td>
      <td bgcolor="yellow">19.50</td>
   </tr>
...
</table>

• XML is another markup language.
• An XML document does not usually contain presentation details.
• Instead, it is just used to describe data.
• The tags of the language are chosen by the author.
• For this reason, it is called the eXtensible Markup Language.

<consumables>
   <product>
      <category>inkjet cartridges</category>
      <item>HP Deskjet print cartridge 51645A</item>
      <price>19.50</price>
   </product>
   ...
</consumables>

1.2. Some jargon

Here is some jargon:

<price>19.50</price>

is called an element. It is introduced by a start tag. In this example, the start tag is:

<price>

And it is terminated by an end tag:

</price>

A start tag may have one or more attributes as in:

<price units="pounds" country="UK">19.50</price>

An element may be empty as in:

<price amount="19.50"></price>

and this can be abbreviated to:

<price amount="19.50"/>

Another example is:

<unknown_price></unknown_price>

and this can be abbreviated to:

<unknown_price/>

1.3. The main goals of XML

As I see it, there are three main goals of XML:

1. To provide a language that just describes some data. If it is necessary to describe how this data is to be displayed (and in some situations this is not the case), then this is described in a separate document.
2. To allow the author to use their own tags. In this way, it is possible for an author to choose tags that are appropriate to the data.
3. To insist on the language being used in a correct manner rather than being tolerant, e.g.,
   • each start tag must have an end tag;
   • the values of attributes must be quoted.

1.4. Well-formed, DTDs and valid

• WWW browsers are lenient about the quality of the HTML.
• The XML specification does not allow XML parsers to be lenient.
• A correct document is said to be well-formed.
• An XML document that is not well-formed must be reported as being in error.
• An XML document may use a document type definition (DTD).
• The DTD is a sequence of rules describing the markup language.
• Can appear in the same file as the XML document or in a separate file.
• XML that conforms to a DTD is said to be a valid document.
• Even if there is a DTD, not all parsers check that the XML conforms to the DTD.

<!ELEMENT consumables (product*)>
<!ELEMENT product (category, item, price)>
<!ELEMENT category (#PCDATA)>
<!ELEMENT item (#PCDATA)>
<!ELEMENT price (#PCDATA)>

1.5. XML schemas

More recently, XML schemas have been introduced. They can be used to define the structure of some XML (instead of using a DTD). Two of the main differences are:

1. An XML schema is written as an XML document (which means that there is less to learn and that it can be parsed by an XML parser).
2. When producing an XML schema, not only can you use predefined types, your schema can also introduce new types that help you to define the structure of the XML.

1.6. Resources about XML

1. Jon Bosak, 'XML, Java and the future of the Web', http://www.ibiblio.org/pub/sun-info/standards/xml/why/xmlapps.htm
2. Elliotte Rusty Harold, 'XML Bible', IDG Books, 1999, 0-7645-3236-7.
3. Charles Goldfarb and Paul Prescod, 'The XML Handbook', Prentice Hall, 2000, 0-13-014714-1.
4. Brian E. Travis, 'XML and SOAP Programming for BizTalk Servers', Microsoft Press, 2000, 0-7356-1126-2.
5. Some of the main WWW pages about XML are http://www.w3c.org/XML/, http://www.xml.com/, http://www.xml.org/, http://www.ibm.com/xml/, http://www.ibiblio.org/xml/, http://www.oasis-open.org/, http://www.oasis-open.org/cover/ and http://www.ucc.ie/xml/.

3.1. XSL

• Suppose some other person wants to read our XML document.
• Suppose that their XML reading program requires only part of the data ...
• ... or it requires the XML to be written in a different way.
• It would be useful to describe how one XML document can be transformed into another one.
• This is one of the roles of the eXtensible Style Language (XSL).
• This is called an XSL Transformation (XSLT).
• As HTML is close to XML, XSL is often used to transform XML into HTML.

3.2. The consumables element

<xsl:template match="consumables">
   <html>
   <body>
   <table>
      <xsl:apply-templates/>
   </table>
   </body>
   </html>
</xsl:template>

• The notation used by an XSL document is XML.

3.3. The product element

<xsl:template match="product">
   <tr>
   <td><xsl:value-of select="category"/></td>
   <td><xsl:value-of select="item"/></td>
   <td><xsl:value-of select="price"/></td>
   </tr>
</xsl:template>

3.4. Other aspects of XSL

There are many other aspects to the use of XSL to transform documents. These include the possibilities of:

1. re-ordering the processing of the elements of an XML document;
2. looping over elements;
3. deciding which elements to include;
4. sorting with respect to some element.

4.1. Approach One: using a tool to process XML

We could write a program to process an XML document. The program could be used to produce HTML which we could then make available on the WWW. More details about this are given in Section 5.

The problem with using a tool is that we have to remember to run it everytime the XML document is updated. The remaining two approaches enable the user of a browser to work directly on the XML document.

4.2. Approach Two: processing XML with a JavaServer page

If a webserver supports the running of server-side programs, then a browser visiting a page could trigger a webserver to run a program that reads an XML document and produces HTML. Although this could be done using a CGI program or by running a PHP script, Section 6 looks at how a Java program could be run by a webserver.

4.3. Approach Three: get the browser to process the XML

Increasingly, browsers are providing support for the use of XML and XSL. So Section 7 looks at how they are supported by Microsoft's Internet Explorer and Mozilla.

5.1. Creating a tool in the Java programming language

• You could write your own program to process an XML document.
• Two APIs make these programs easy to write in Java: SAX, DOM.
• SAX walks through an XML document. It generates events for each part of a document.
• In your Java program, you can provide an object to handle these events.
• DOM builds a data structure representing an XML document.
• Your Java program can wander around the data structure at will.

• Suppose we want to write a Java program to transform some XML into HTML.
• Although DOM could be used to do this, makes more sense to use SAX.
• Many software developers provide SAX.
• Code you write that uses SAX is the same except for creating an instance of the SAX parser.
• Suppose we use the Xerces parser from the Apache Software Foundation:

XMLReader tXMLReader = new org.apache.xerces.parsers.SAXParser();

• We also need to create the handler that is to handle the events generated by the SAX parser:

Handler tHandler = new Handler();

• This object then needs to be registered with the SAX parser:

tXMLReader.setContentHandler(tHandler);

• Then the parser is let loose on the document:

FileReader tFileReader = new FileReader("consumables.xml");
InputSource tInputSource = new InputSource(tFileReader);
tXMLReader.parse(tInputSource);

• It is the Handler class that is used to state what should happen when each event arises.
• It has to override the methods of the DefaultHandler class (which has methods that do nothing).
• For example, consider:

public class Handler extends DefaultHandler
{
   public void startElement(String pURI, String pLocalName,
                            String pQualifiedName, Attributes pAttributes)
   {
      System.out.println(pLocalName);
   }
}

• This Handler class would cause the name of each element to be output when the start tag of the element is parsed.

• The full code of a program that produces HTML from the consumables.xml file is given at http://www.dur.ac.uk/barry.cornelius/java/xml.processing/code/ConvertConsumables.java and http://www.dur.ac.uk/barry.cornelius/java/xml.processing/code/Handler.java.

5.2. Using a tool to create HTML from XML and XSL

• Instead of sprinkling HTML all over a program, we could use the XSL stylesheet that was produced earlier.
• Many software developers provide a tool producing the document that results from applying XSL to XML.
• The Apache Software Foundation provide Xalan.
• It can be used in the following way:

  java org.apache.xalan.xslt.Process -in consumables.xml -xsl consumables.xsl -out consumables.html
  

• Assumes the classpath includes xerces.jar and xalan.jar.

5.3. Resources about SAX, DOM and XSLT

XML parsers (including the SAX and DOM APIs) and XSLT processors are available from:

1. http://xml.apache.org/xerces-j/ and http://xml.apache.org/xalan/.
2. http://msdn.microsoft.com/xml/default.asp.
3. http://users.iclway.co.uk/mhkay/saxon/.
4. http://www.jclark.com/xml/xp/ and http://www.jclark.com/xml/xt.html.

Some articles and books about SAX, DOM and XSLT include:

1. David Megginson, 'SAX2: Quick Start', http://www.megginson.com/SAX/Java/quick-start.html
2. Mazmul Idris, 'XML and Java Tutorial Part 1', http://www.developerlife.com/xmljavatutorial1/default.htm
3. Brett McLaughlin, 'Java and XML', O'Reilly, 2000, 0-596-00016-2.
4. Brett McLaughlin, 'All about JAXP: Sun's Java API for XML Parsing', http://www.ibm.com/software/developer/library/x-jaxp/index.html
5. Dave Pawson, 'XSL Frequently Asked Questions', http://www.dpawson.co.uk/xsl/xslfaq.html
6. Ken Holman, 'What is XSLT?', http://www.xml.com/pub/a/2000/08/holman/index.html
7. Paul Sandoz, 'FOP Slide Kit', http://www.sun.com/software/xml/developers/fop/
8. Doug Tidwell, 'Tutorial: Transforming XML documents [into HTML, SVG, and PDF]', http://www.ibm.com/xml/

6.1. Servlets

• Instead of running a tool to transform an XML document into HTML, you could get a webserver to run a program that does the transformation.
• Use a CGI script, a PHP script, a servlet using SAX, or a JavaServer page.
• Later, we will do this using a tag library in a JavaServer page, but first it is useful to have a detour introducing the ideas of servlets, JavaServer pages and JavaBeans.
• A servlet is a WWW page that is written in Java.
• The author of the servlet has to compile the servlet.
• When a browser accesses the URL of the servlet, the bytecodes of the servlet will be run.
• This will done either by the webserver itself or the webserver will get some other program to arrange for the servlet to be run.
• This program just starts a new thread to run the bytecodes of the servlet. So, no new process needs to be created to run the servlet.
• The servlet does some work and generates some HTML.
• The webserver gets the HTML from the servlet and passes that to the visitor's browser.

• Suppose we want a WWW page that uses a form to get data from a visitor, and some Java source code that gets the data from the form and generates some HTML.
• Here are the HTML instructions of a WWW form:

  <HTML>
     <BODY>
        <FORM METHOD="POST" ACTION=
           "http://altair.dur.ac.uk:8080/barry.cornelius/servlet/choice">
           Please enter your choice<BR>
           <INPUT TYPE="text" NAME="choice">
           <BR>
           <INPUT TYPE="submit" VALUE="Submit choice">
           <INPUT TYPE="reset">
        </FORM>
     </BODY>
  </HTML>
  

• Here is the Java source code of a servlet:

  import javax.servlet.http. HttpServlet;
  import javax.servlet.http. HttpServletRequest;
  import javax.servlet.http. HttpServletResponse;
  import java.io.            IOException;
  import java.io.            PrintWriter;
  import javax.servlet.      ServletException;
  public class choice extends HttpServlet
  {
     public void doPost(final HttpServletRequest request,
                        final HttpServletResponse response)
           throws ServletException, IOException
     {      
        final String tChoiceString = request.getParameter("choice");
        response.setContentType("text/html");
        final PrintWriter tResponsePrintWriter = response.getWriter();
        StringBuffer tStringBuffer = new StringBuffer();
        tStringBuffer.append("<html>\n" );
        tStringBuffer.append("<title>Reply</title>\n" );
        tStringBuffer.append("Your choice was: " + tChoiceString + "\n" );
        tStringBuffer.append("</html>\n" );
        tResponsePrintWriter.println(tStringBuffer);
        tResponsePrintWriter.close();
     }
  }
  

• Here the class is derived from javax.servlet.http.HttpServlet and it overrides the doPost method of that class.
• When doPost is called, it is passed two arguments: request enables access to the values on the form, and response provides a means of passing back HTML to the webserver.
• In order for this to work, you need a webserver that understands servlets.
• One possiblity is to use tomcat, a program provided by the Apache Software Foundation, or to use Apache's webserver augmented by tomcat.
• Although I have successfully configured Apache's webserver to use tomcat, I have chosen not to put this into active use.
• This is because a badly written servlet could be cpu-bound leading to poor performance of the computer running the webserver.
• Instead, at the University of Durham, we use tomcat on its own (i.e., without Apache's webserver) on a different computer, a multi-processor machine called altair.dur.ac.uk.
• The above form can be accessed by going to: http://www.dur.ac.uk/barry.cornelius/tomcat/choice/choice.html.
• We could instead provide source code that uses the SAX API to generate HTML from an XML document.
• Could use code like that given earlier in ConvertConsumables.java and Handler.java.
• In this way, we could provide a servlet that automatically generates HTML from XML.

6.2. JavaServer pages

• A servlet is a mix of Java source code that does some work and Java source code that generates HTML instructions.
• Because of this, the code is not easy to read.
• A JavaServer page is a WWW document written in a mix of HTML and elements that are dynamically generated by executing Java code.
• The webserver will either itself transform, or instead arrange for another program to transform, the JavaServer page into Java source code (a servlet).
• This is then executed and its output is then passed to the webserver to give to the browser.

• Suppose we want a WWW page that uses a WWW form to get a maximum price from a visitor. Suppose we then want to run some Java source code that queries a database generating some HTML giving all the products that are not greater than this price.
• Here are the HTML instructions of a WWW form:

  <HTML>
     <BODY>
        <FORM METHOD="POST" ACTION=
           "http://altair.dur.ac.uk:8080/barry.cornelius/prices/prices.jsp">
           What is your choice of price?<BR>
           <INPUT TYPE="text" NAME="testprice">
           <BR>
           <INPUT TYPE="submit" VALUE="Submit price">
           <INPUT TYPE="reset">
        </FORM>
     </BODY>
  </HTML>
  

• Here is the code of a JavaServer page:

  <%@page language="java" import="java.sql.*" %>
  <%
     String tMaxPriceString = "1000000000000.0";
     if (request.getParameter("testprice") != null) 
     {
        tMaxPriceString = (String)request.getParameter("testprice");
     }
     final Driver tDriver = 
           (Driver)Class.forName("org.gjt.mm.mysql.Driver").newInstance();
     final Connection tConnection = DriverManager.getConnection(
           "jdbc:mysql://www.dur.ac.uk/Pdcl0bjc_prices", "", "");
     final PreparedStatement tPreparedStatement = 
           tConnection.prepareStatement(
                 "SELECT * FROM consum WHERE price < " 
                 + tMaxPriceString + " ORDER by price");
     tPreparedStatement.setQueryTimeout(0);
     final ResultSet tResultSet = tPreparedStatement.executeQuery();
  %> 
  <html>
  <head>
  <title>Access to the prices database</title>
  </head>
  <body bgcolor="#FFFFFF">
  <table>
  <%
     while (tResultSet.next())
     {
  %>
  <tr>
  <td>
     <%= ((tResultSet.getObject("price")!=null)?tResultSet.getObject("price"):"") %>
  </td>
  <td>
     <%= ((tResultSet.getObject("goods")!=null)?tResultSet.getObject("goods"):"") %>
  </tr>
  <%
     }
  %>
  </table>
  </body>
  </html>
  <%
     tResultSet.close();
     tConnection.close();
  %>
  

• The above form can be accessed by going to: http://www.dur.ac.uk/barry.cornelius/tomcat/prices/prices.html.
• A JSP is a page of HTML with Java source code inserted between

  <%
  

  and

  %>
  

  brackets.
• So the HTML is a lot easier to read if a JSP is used instead of a servlet.

6.3. Using a JavaBean from a JSP

• An object whose class provides appropriate get and set methods is called a JavaBean.
• A JavaServer page can easily create a JavaBean and access its get and set methods.
• This is done using the tags jsp:useBean, jsp:setProperty and jsp:getProperty.
• Here is a JavaServer page that uses a JavaBean called tAdder:

  <html><body>
  <p>start</p>
  
  <jsp:useBean id="tAdder" scope="page" class="addition.Adder" />
  
  <p>effectively this applies setFirst(27) to tAdder<p>
  <jsp:setProperty name="tAdder" property="first" value="27" />
  
  <p>effectively this applies setSecond(42) to tAdder<p>
  <jsp:setProperty name="tAdder" property="second" value="42" />
   
  <p> now apply getSum to tAdder </p>
  <p>sum is <jsp:getProperty name="tAdder" property="sum" /> </p>
  
  <p>effectively this applies setFirst(5) to tAdder<p>
  <jsp:setProperty name="tAdder" property="first" value="5" />
  
  <p> now apply getSum to tAdder again </p>
  <p>sum is <jsp:getProperty name="tAdder" property="sum" /> </p>
  
  <p>finish</p>
  </body></html>
  

• Here is the code of the Adder class:

  package addition;
  public class Adder
  {
     private String iX = "0";
     private String iY = "0";
     public void setFirst(String pFirst)
     {
        iX = pFirst;
     }
     public void setSecond(String pSecond)
     {
        iY = pSecond;
     }
     public String getSecond()
     {
        return iY;
     }
     public String getSum()
     {
        int tResult = Integer.parseInt(iX) + Integer.parseInt(iY);
        return "" + tResult;
     }
  }
  

• The above texts are in the files http://www.dur.ac.uk/barry.cornelius/java/xml.processing/code/addition.jsp and http://www.dur.ac.uk/barry.cornelius/java/xml.processing/code/Adder.java.
• The JSP can be executed by going to: http://altair.dur.ac.uk:8080/barry.cornelius/addition/addition.jsp.
• The use of JavaBeans can help to provide a better separation of the HTML from the Java source code.
• There are three other standard tags: jsp:include, jsp:forward, jsp:plugin.

6.4. Using a tag library in a JSP

• Besides the six standard tags, in a JavaServer page we can also use tags from a tag library.
• The Apache Software Foundation has developed a number of tag libraries: this is the jakarta-taglibs project.
• Of interest to us is a tag library containing tags for processing XML input and applying XSL transformations.
• Here is a JavaServer page that uses these tags:

  <%@page language="java" %>
  <%@taglib uri="http://jakarta.apache.org/taglibs/xsl-1.0"
            prefix="xsltlib" %>
  <html>
  <head>
  <title>Consumables example</title>
  </head>
  <body>
  <xsltlib:apply xml="/xml.processing/consumables.xml" 
                 xsl="/xml.processing/consumables.xsl" />
  </body>
  </html>
  

• The above text is in the file http://www.dur.ac.uk/barry.cornelius/java/xml.processing/code/Consumables.jsp.
• It can be executed by going to: http://altair.dur.ac.uk:8080/barry.cornelius/xml.processing/jsp/Consumables.jsp.

• Three files are needed to get this to work with tomcat:

  • web.xml is used to associate http://jakarta.apache.org/taglibs/xsl-1.0 with the location of xsl.tld.
  • xsl.tld contains a definition of the tag library together with names of the Java classes that implement the tags.
  • xsl.jar contains the .class files of these classes.

6.5. Resources about JSPs

1. Sun's main page about JavaServer pages is at: http://java.sun.com/products/jsp/
2. Jakarta's main page about tomcat is at: http://jakarta.apache.org/tomcat/
3. 'JavaServer Pages Tag Libraries', http://java.sun.com/products/jsp/taglibraries.html
4. 'Developing XML Solutions with JavaServer Pages Technology', http://java.sun.com/products/jsp/html/JSPXML.html
5. Duane K. Fields and Mark A. Kolb, 'Web Development with JavaServer Pages', Manning Publications, 2000, 1-884777-99-6.

7.1. Use of CSS1 with XML

Both IE5.5 and M18XSLT provide support for XML with CSS1 (version 1 of Cascading Style Sheet). To use CSS, you need to augment an XML document with a xml-stylesheet processing instruction. This results in the text that is in the file http://www.dur.ac.uk/barry.cornelius/java/xml.processing/code/css1.xml.

<?xml version="1.0" standalone="no"?>
<?xml-stylesheet type="text/css" href="css1.css"?>
<!DOCTYPE consumables SYSTEM "consumables.dtd">
<consumables>
...
</consumables>

The type attribute of this xml-stylesheet processing instruction says that this XML document is to be transformed using a CSS stylesheet and the href attribute gives the URL of the file containing the actual CSS instructions.

The file css1.css might contain:

consumables { display: block }
product     { display: block }
category    { font-size: x-large; 
              color: black }
item        { font-size: large; 
              color: red }
price       { background-color: yellow; 
              color: blue }

7.2. Use of CSS2 with XML

CSS2 introduces some new aspects like the handling of tables. So instead we could use the file http://www.dur.ac.uk/barry.cornelius/java/xml.processing/code/css2.xml.

<?xml version="1.0" standalone="no"?>
<?xml-stylesheet type="text/css" href="css2.css"?>
<!DOCTYPE consumables SYSTEM "consumables.dtd">
<consumables>
...
</consumables>

where the file css2.css contains:

consumables { display: table }
product     { display: table-row }
category    { display: table-cell; font-size: x-large; 
              color: black }
item        { display: table-cell; font-size: large; 
              color: red; text-indent: 0.1in }
price       { display: table-cell; background-color: yellow; 
              color: blue; text-align: right; text-indent: 0.1in }

Although these instructions are obeyed correctly by M18XSLT, they are not understood by IE5.5.

7.3. Use of XSL with XML

Both IE5.5 and M18XSLT can use XSL to process an XML document. You need to insert the following xml-stylesheet processing instruction in your XML document:

<?xml-stylesheet type="text/xsl" href="consumables.xsl"?>

Inconveniently, there are two important differences in the XSL that is understood by IE5.5 and M18XSLT:

1. IE5.5 requires a rule in the XSL stylesheet for processing the outermost level of an XML document. So IE5.5 needs the extra rule:

   <xsl:template match="/">
      <xsl:apply-templates/>
   </xsl:template>
   

2. IE5.5 requires the following element to identify the namespace being used:

   <xsl:stylesheet xmlns:xsl="http://www.w3.org/TR/WD-xsl">
   

   whereas an M18XSLT XSL document requires:

   <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
   <xsl:output method="html"/>
   

This is because IE5.5 has an old version of Microsoft's XML Parser (MSXML). The WWW page http://www.netcrucible.com/xslt/msxml-faq.htm gives details of how you can obtain Version 3.0 of MSXML (November 2000) and get IE5.5 to use it.

The files http://www.dur.ac.uk/barry.cornelius/java/xml.processing/code/xsl.xml and http://www.dur.ac.uk/barry.cornelius/java/xml.processing/code/consumables.xsl contain the texts that can be used by both M18XSLT and the modified version of IE5.5.

9.1. Computer Science projects during 1999-2000

• 2nd year Computer Science students engage in a Software Engineering Group project.
• During the life cycle, many files containing documentation and code are produced.
• Unfortunately, they are managed in an ad-hoc fashion.
• During 1999-2000, two final year Computer Science students undertook projects investigating how the documentation and code could be stored in an XML document.
• They also looked at how this XML could be viewed in different ways, ...
• ... or processed to extract a particular piece of source code or a particular test case.
• Isabel Delacour chose to utilise the XML support provided by IE5.0.
• Michael Creasy chose to process the XML using servlets (using JServ and an Apache webserver).

9.2. Engineering projects during 2000-2001

• During 2000-2001, two final year Engineering students are involved with developing XML for Geotechniques.
• Alex Cubitt is providing XML markup for structures.
• He will also be looking at displaying the XML using Java's 2D/3D graphics.
• Rob Shields is providing XML markup for boreholes.
• He will be looking at converting between AGS and his XML (and back again).
• He will also be producing graphical output from the XML.