Blog Host: Daniel Cazzulino - senior architect, developer and co-founder, Clarius Consulting S.A. READ ENTIRE BIO

As an XML geek, it's hard to even think that this is generally the case. But reality and productivity are pretty much irrefutable. The good thing is that .NET allows you to work either at the most hardcore low-level (XmlReader, a bare XML parser) right up to the most OO-friendly abstractions (XmlSerializer). In the middle, you have XmlDocument and XPathNavigator.

I cannot end this series of blog posts without mentioning my frustration when I knew that .NET v2.0 would not ship with an XQuery implementation. Some in the blame-MS-for-everything club think that MS is the one to blame. However, I'm aware that this is not the first time the W3C is to blame instead.

XQuery had the potential to transform many kinds of applications, which now are tedious, hard to maintain or poorly performing in .NET because they have to resort to either XmlReader/XmlWriter or XmlDocument/XPathNavigator APIs; either of these options requires a lot of work and is very difficult to evolve over time (especially a short time). XQuery could have rendered all that XmlDocument manipulation or XmlReader/XmlWriter manipulation unnecessary.

XQuery will probably become the first widely used domain-specific language, where the domain is XML manipulation. It has some key differences with XSLT -- another very powerful and popular language to manipulate XML -- the most important of which is probably its non-XML syntax. It's a language more familiar to the average programmer, with that slight feel of T-SQL or traditional procedural programming, which I think will make for the killer language for XML manipulation.

I was just about to start my self-training (as usual) in XQuery when I got the news. Unfortunately this means that story of XQuery in .NET is probably delayed by at least 2-3 years, so I'll have to find some other programming area to keep my spare time occupied. WS-* no longer entices me as I know that sooner or later (read Indigo) it will become part of the protocol stack, much like TCP/IP, HTTP or COM+ transactions did in the past. You won't have to care about it when it becomes just a .NET attribute on a class. You learn the attribute, when to apply it and forget about the goo of SOAP, protocols, the W3C or OASIS.

This may sound discouraging, but SQL Server 2005 will bring many XQuery features that you can take advantage of. There will be no client API, but it will let you get your hands dirty with XQuery if you can afford starting/moving a project to Yukon as soon as it comes out.

In the meantime, I'll keep working on interesting projects from MS Patterns & Practices and making my company grow to meet new demands. My focus for now is switching to long-time growing interest in design patterns and OO framework techniques.

I hope you enjoyed these two weeks of XML topics. I know I did!
Posted by kzu On XML performance and XPath expressions and execution
27 JAN 2005 19:56 EST (00:56, GMT)
Manipulating XML is something that not many people take seriously, as it seems almost like string manipulation to them: How much can be optimized on that, after all?

It turns out that there's quite a lot to look for when squeezing the CPU for maximum performance, and XML handling is no exception. Your first stop should be the document from the Patterns and Practices guys: Improving .NET Application Performance and Scalability. I'm glad I could contribute a couple ideas to the Improving XML Performance chapter.

One of the areas that can yield better results is XPath expression precompilation. I discuss what's involved in expression compilation in my Weblog.

The P&P article has a section named "Compile Both Dynamic and Static XPath Expressions" section, but doesn't go into depth on compiling dynamic expressions. The aforementioned Weblog post was my initial approach to it, and it can already provide between 1.5x and 2xperformance increase to your applications. However, the API wasn't exactly the most friendly one.

So I moved on to implement the XPathCache object for the Mvp.Xml project, which providesone-line-of-code, high-performance, precompiled and parameterized expressions execution. Typical code looks like the following:

XPathNodeIterator it = XPathCache.Select( "/customer/order[id=$id]/item",  navigator,  new XPathVariable("id", idvalue));

1. Check if the given expression has been previously compiled. If not, it will compile it at this time.
2. Build a dynamic context for the expression to execute, so that variables can be passed to it.
3. Receive the variable values and resolve them as the query is executed.

XPathNodeIterator it = navigator.Select( "/customer/order[id='" + idvalue + "']/item");

The XPathCache even provides overloads to execute queries and get the results ordered automatically, get the results as an XmlNodeList instead of an XPathNodeIterator (if the query is run against an XmlDocument) and even pass prefix/namespace mappings in a single line without having to even create an XmlNamespaceManager at all. An example of a sorted and prefixed expression execution is as follows:

XPathNodeIterator cached = XPathCache.SelectSorted( /* expression to execute */ "mvp:books",  /* xml source */ navigator, /* expression to sort by - will also be precompiled and cached!!! */ "mvp:price",  /* sorting criteria, look for the documentation of XPathExpression.AddSort */ XmlSortOrder.Ascending, XmlCaseOrder.LowerFirst, String.Empty,XmlDataType.Number, /* inline prefix-namespace mapping */ new XmlPrefix("mvp", "http://mvp-xml.sf.net", navigator.NameTable));

Still not convinced to download the project source code or binaries from SourceForge?
Posted by kzu XML performance
26 JAN 2005 08:51 EST (13:51, GMT)
Today I'll make a pause on my series titled "To DOM or not to DOM, is that the question?" and give you a break on XML APIs on .NET to focus on another very interesting aspect that many don't pay attention enough: performance. You have to be especially careful about it when using XML as it's usually not as efficient as other representations and the added features (transformations, XPath and so on) can be a major bottleneck and resource hog if not used properly. I've put together a list of interesting tricks to use when you want to boost your XML manipulation routines: Roadmap to high performance XML.

And just to leave you thinking: Did you know that the new xsd.exe tool in .NET v2 generates partial classes (besides properties instead of fields, of course)? Will let you imagine the scenarios that become possible (hint: The return of the king OO and the end of the plain data transfer object pattern).

Stay tuned!
Posted by kzu To DOM or not to DOM, is that the question?, part 3
25 JAN 2005 06:06 EST (11:06, GMT)
In parts 1 and 2 of this series, I've talked about the alternatives to the DOM in .NET for manipulating XML. Today I'll discuss a third alternative, the XmlReader API.

The XmlReader is very compelling because it's a forward-only, read-only, non-cached cursor over an XML stream. Hence, it's very performant and well-suited to handing large documents. However, programming against this API makes for very long and hard-to-maintain and -evolve blocks of code. Typically, you will have to keep track of where you are in the document and switch on the current reader information to determine whether you have to process something or not. The code usually looks as follows:

XmlReader reader; // Initialized afterwards to point to the XML.
...
object ns = reader.NameTable.Add("expected-namespace");
while (reader.Read())
{
 switch (reader.LocalName)
 {
  case "Customer":
   if (ns.Equals(reader.NamespaceURI)) continue;
   // process customer
   break;
  case "Order":
   if (ns.Equals(reader.NamespaceURI)) continue;
   // process order
   break;
  // Other elements that need to be processed. Typically check
 reader.Depth too.
 }

Even so, the XmlReader has its place. In the future, I'd like to have access to strongly typed readers generated from XML schemas, hence bringing the benefit of compiler checks for your code (and, therefore, the ease of prototyping/evolution), while keeping the high performance and low resource consumption the XmlReader approach has.

Whenever you need to process a subtree of the entire reader, the way you can guarantee that the consuming method will not move the reader past the node he is supposed to process is by reading the subtree into another stream and pass only that. This is required if you have pluggable components that process the XML based on some kind of matching information, as otherwise you risk one component breaking the processing performed by others. This happens because the XmlReader is forward-only, and once someone moves it too far, there's no way of getting back. The way you would read a subtree would be:

XmlReader reader; // Initialized afterwards to point to the XML.
... // Move to root of subtree to process

StringWriter sw = new StringWriter();
XmlTextWriter tw = new XmlTextWriter(sw);
// Write the entire subtree to the underlying writer.
tw.WriteNode(reader);
tw.Close()

// Call the processing function with subtree.
ProcessNode(new XmlTextReader(new StringReader(sw.ToString())));

Version 2 of .NET XmlReader brings several improvements for these and other scenarios. Among the new features you will find particularly useful are:

1. Reading subtrees in a single step, avoiding double parsing. The ReadSubtree method does it automatically.

2. Moving directly to certain nodes instead of looping. ReadToDescendant and ReadToNextSibling allow you to specify the name and namespace of the item to move to and avoid the loop altogether.

3. Read typed values directly from the stream. ReadAsBoolean, ReadAsDateTime and so on provide conversion automatically.

4. Read of an object deserialized with XmlSerializer. This is a very cool one and indeed one that will make the approach I explained in my previous post an obvious choice even for large documents. The new method is called ReadAsObject, and you pass a Type to it, which will be used to create an instance of the XmlSerializer to deserialize the current subtree into an instance of that type and its entire object model. If you combine this with 2-, you can split a large document in manageable pieces, do the main processing with strongly typed deserialized objects and have a fairly simple main loop of reading that will load these objects as they appear on the XML stream. This will most probably be my choice in the future.

As an XML geek, I've sadly discovered that most developers don't want to see angle brackets if there's any way of avoiding it. The XmlDocument approach is too low-level for them, too generic, too XML-ish. The XPathNavigator doesn't make things any better for them, as they are still in the realm of attributes, elements, siblings and so on. Not their usual familiar object models. For these guys, there is a very compelling alternative in the XmlSerializer, which can turn XML into a nice deserialized object model you can manipulate much the same way you manipulate your own custom models.

The way it works is ingenious: You create an XML schema that describes what the XML will look like. Then the xsd.exe utility (provided with .NET) or some other similar tools (some of them even better than the former) will generate .NET classes in either C# or VB tagged with XML Serialization attributes that will tell the XmlSerializer class how to serialize/deserialize to and from that XML format. If you do enough experimentation, you can design your schema in a way that will give you pretty much the object model you want.

I've found this approach very compelling. I can build a prototype that requires configuration or some other form of XML processing very fast, by avoiding tons of XmlDocument or XmlReader processing that is later hard to both evolve and maintain. Yeah, I know... I said prototype... but you know that some prototypes live way longer than you expected, and sometimes they even become products. Having an XSD -> Classes step makes evolving the prototype very easy. You need a new attribute? rename an element? remove another? relocate something? Just go change the XSD, try to recompile, and now all the places that fail are the ones to fix, as they are expecting classes where they no longer are. So you just fix it so it compiles again, and you can rest assured that nothing will break at runtime because the input XML has changed. You can layer XML Schema validation on before deserializing, and you're bullet-proof.

Not only is this approach very interesting for highly iterative projects and prototypes, but it is also very performant and is much more familiar to the average .NET developer.

There are a couple well-known issue in v1.x with the XmlSerializer. It will generate a custom assembly on the fly at runtime, which will know how to deserialize your model. This is a very time-consuming task that can hinder your application's startup time. The other issue is that it's very hard to participate in the deserialization process, and, therefore, you cannot perform additional processing you may need at that time. This means that you usually have to post-process the deserialized model using potentially very long FOREACH statements that traverse the entire graph. Both problems have a solution right now, however. The MVP XML Project provides a Design package that comes with a custom tool that can generate at design time a custom XmlSerializer class that is strongly typed for a particular object model. It therefore avoids the first problem. But it also adds deserialization events for each object in the model you can attach to, therefore removing the second problem, too.

I'll continue to discuss alternative to XML processing in .NET in the following days. Stay tuned!
Posted by kzu To DOM or not to DOM, is that the question?, part 1
21 JAN 2005 23:57 EST (04:57, GMT)
Traditionally, as a developer dealing with XML, you had two opposite choices: Use the DOM, which was the de facto standard way of handling in-memory representations of an XML document (therefore, the document must be loaded entirely in memory) or use SAX, which was the de facto standard API for streaming manipulation of XML (therefore, it was required when very large documents need to be processed, as it doesn't load the entire document in memory).

So the question was pretty much whether you were going to use the DOM or not. The DOM is a very well-understood model that most developers know, which is fairly easy to learn and which is also used in another ubiquitous environment: client-side Web programming.

With the advent of .NET, there are new options, and the decision to use one over the other is no longer so easy.

You can still use the familiar DOM through the System.Xml.XmlDocument class. This is generally the preferred one for newcomers to .NET, as it's the natural upgrade path for users of MSXML DOM. However, it's also the worst performing in the general case, too. The DOM itself turned out to be a model that exposes too much information about the underlying internal structures used to create it and, therefore, leaves almost no room for optimizations. This problem became evident when the XML team realized that the performance of XSLT transformations over this model was really bad, and there was almost nothing they could do about it. Hence, a new beast entered the scene: the XPathNavigator.

The XPathNavigator is a cursor-based class (with methods like MoveToAttribute, MoveToNext and so on), that is abstract and therefore independent of any particular implementation of an in-memory store for XML. It was quite revolutionary, and now we've started seeing Java implementations of such an approach. The key benefit of the XPathNavigator is that any object can implement the IXPathNavigable interface, to expose its CreateNavigator method and to return a navigator that allows traversal of an XML view over the object. Of course, such a view over an XmlDocument store (which implements IXPathNavigable) is fairly easy to imagine, as it's just an alternate way of navigating over nodes, their attributes and their children.

But there's another XML store that implements the IXPathNavigable interface: XPathDocument. This is an optimized store that excels in the processing of XPath queries, which is at the core of XSLT processing. So, whenever you are doing XPath queries (either standalone or in the context of XSLT processing), you should definitely be using an XPathDocument to load the data.

Of course, nothing forbids other object models to expose an XML view by implementing the IXPathNavigable interface or XPathNavigator themselves. There is actually one such example in the ObjectXPathNavigator, which exposes an arbitrary object model as XML by making each object and property an element. It's just a sample of what's possible, as it's not synchronized with XML Serialization attributes, for example, but it clearly shows the idea.

Over the following days, I'll delve into the alternatives to XmlDocument, the migration path for those using it now and the improvements you can expect from .NET v2.
Posted by kzu A world-class XML editor coming with Whidbey?
20 JAN 2005 21:37 EST (02:37, GMT)
I've been playing with the new XML editor coming with the latest Whidbey Community Technology Preview of December 2004. I've got to say I'm impressed. Just to name a few features you'll enjoy:

1. Automatic validation against schemas: there's a schema folder at %prog.files%\Microsoft Visual Studio 8\Xml\Schemas where you can put schemas. They will be automatically picked for the current edited document based on its namespace. This works not only for an entire document with a single namespace, but also for mixed elements with different namespaces! The validation gives you the familiar red squigglies to point you right on the offending element/attribute. This feature even integrates with the standard .NET Errors window.

2. Support for a schema catalog file in the same location, where you can specify not only local but also remote (http:// !!!) schemas and they will be automatically downloaded and cached, too.

3. Built-in schemas for a LOT of XML vocabularies, including MSBuild, SOAP 1.1 and 1.2, WSDL, XHTML, XSD, XSLT and so on. Notably missing still is the newly standardized XInclude specification I talked about yesterday :( ... Shouldn't be long until it makes its appearance, though.

4. Ability to locate schemas from the cache by instance document/element namespace, by xsi:schemaLocation and xsi:noNamespaceSchemaLocation. Schemas are also automatically picked from the current solution, no matter where they are :) :). And if none of these location probings work, you can even assign the schemas to use with the file manually.

5. Collapsing of elements!

6. XML code snippets!!! This is a really fantastic feature. Say you're under the root element of an instance document in a known namespace (that is, the editor knows about the schema ruling it), and the required child element is <Author ...>, you can just type <Author and press TAB, and you'll get a skeleton of the minimum requirements for such an element to be valid. Way cool. And it's apparently using the same snippet engine from the other languages like C#, so the user experience is very familiar, where each of the required values you will need to type can be accessed just by pressing TAB.

7. Fully customizable: coloring, indenting, auto completion, formatting and so on.

8. "View Data Grid..." designer that allows you to easily enter data in the document if it's high regular and tabular-like. (I know, this one is already in Everett ;)).

9. Ability to run XSLT transformations. You can just assign an XSLT file to the Stylesheet property and the target file to generate in the Output one, and you're all set! You can run the transformation and view the results without ever leaving VS.

10. If you specify an XSLT, you can even start the transformation in debug mode and step through it!!!

11. Documentation tooltips. If you properly document your schema by using the xs:documentation tags, you'll even get a tooltip on elements and attributes. I'd like to hear the excuses now for poorly documented schemas....

12. Commenting. You can select a chunk of XML (combine this with collapsing, and it's really useful) and click the Comment or Uncomment buttons on the Text Editor toolbar. Just as it does // for C# code, it will do <!-- ... --> for the selected chunk.

13. Infer schema from document. This one was previously available through GDN and now has been integrated in the editor itself. So you just click a button and get a good starting point from which you can move on by refining and polishing the schema.

Big kudos to the XML team! We're all sad that XQuery didn't make it into v2, but this certainly shows you have been working hard on other areas as well.

The only small missing thing is the "tag navigator" at the bottom of the editor like the one in the HTML editor, which shows the current element and all its parents so you can easily go up from where you are....
Posted by kzu XML modularization and a new kid on the block: XInclude!
19 JAN 2005 23:49 EST (04:49, GMT)
Ever since XML was born, people realized some way to split files was necessary. Configuration files is the most common case, but validation schemas (be it XML schemas or Relax NG or ISO Schematron) and custom scenarios (such as XHTML rendering using common sections, XML composition for transformation and so on) are all too common.

So far, almost everyone has come to custom solutions, mostly in the form of some "keyword" element (<include>, <import> and so on), which signals an XML parser that it should go and look for an external document to read as if it were part of the current document. Sometimes the solution was at a higher level than the parser, such as in XML schemas import and include elements, where processing of these elements has richer semantics than just pulling external XML into the containing document.

In .NET, and particularly in configuration files, this has been a major area of pseudo-innovation. Framework and library authors, in an effort to somehow simplify increasingly big configuration files, would pick arbitrary and rigid points where the configuration could be split, mainly based on their idea of what the target use of the framework/library would be. In my experience, in most cases this turns into a pretty annoying and inflexible model for the users of such products.

But maybe I believe it's so, because I've been waiting several years for a generic inclusion mechanism that would allow me to split XML files as I see fit (almost as many years as the spec itself, which dates back to 1999). Now the wait is over, as XInclude is already a W3C Recommendation, meaning that major players (i.e., MS and Sun or Apache) will soon provide built-in implementations in .NET and Java SDKs.

In .NET, there's already one such implementation, even open source, as part of a project I'm working on: the MVP XML library. MS is already working on this implementation, as well as the future implementation, I'm sure, for .NET v3, is based on the XmlReader paradigm. As such, it's transparent to the developer, as you can just rely on the reader to perform inclusions as necessary. This completely removes the need to think about the potential problems of big configuration files and how to split them for better manageability; it also enables a lot of scenarios that would require custom readers or post-loading processing.

Now I only wish all those alternative mechanisms, especially the ones developed for standard XML vocabularies, just go away and rely on this single generic mechanism.
Posted by kzu XML schemas: Is it really too late to fix them?
18 JAN 2005 19:23 EST (00:23, GMT)
The W3C XML Schema specification (XSD) used to be the most complex spec ever released by the W3C (until XQuery and the Semantic Web specs arrived). It was the result of merging several in-use languages for defining XML data and several often-contradictory interests from DB and OO lovers alike.

The net result is a huge mess that is fully understood by a very small percentage of developers, and that causes great pain to implementers almost four years after its release. Even after the amazing job done by Priscilla Walmsley (Definite XML Schema) in trying to explain some really weird features supported by it, it's hard to imagine why something so apparently simple as defining an XML structure is so difficult.

But most developers fortunately never have to deal with the more complex features of XSDs, such as substitution groups, redefinitions and so on. And that's one more reason to wonder why it has to be so complex. Most implementers argue it's already too late to fix it. Big ones such as Microsoft, with big investments on the technology already (think Office support for XML and custom schemas, SQL Server 2005 and its new support for typed XML data type and so on), have completely surrendered under its weight.

I've heard DBox say that if SOAP (the core technology driving Web services today) died completely, the most significant suspect of its murder would be XSDs. Interesting thing is that SOAP doesn't depend on XSDs at all.

So at this point there are two options:

1. We try to get rid of it and adopt alternatives such as RelaxNG
2. We simplify it, take the best parts of it and get rid of the worst, cryptic, underused ones.

Simplification of SGML gave birth to the fantastic XML. So there's nothing intrinsically wrong in simplifying things. Let's hope simplification of XSD arrives and gives birth to something much more fun to learn and use than the current mess.
Posted by kzu Let the games begin
17 JAN 2005 05:48 EST (10:48, GMT)
During the next two weeks, I will be sharing some experiences, thoughts and trends in the XML space, especially with regards to C# and the .NET Framework.

Even if I'm much more attracted to the XML core technologies such as XML validation, parsing, APIs to manipulate it and so on, it's impossible to ignore the important advances happening in so-called WS-* protocols (XML-based specifications and standards that layer on top of SOAP to provide enterprise-grade features to Web services, such as transactions, reliable messaging, addressing of messages and so on). The entry point for information on that subject is Microsoft WebServices DevCenter.

As a Microsoft MVP on XML technologies, a long-time (as long as XML's life, at least ;)) fan of XML and avid researcher on the way XML is shaping the application development space as well as future trends, I hope you'll enjoy the following days as much as I surely will.

Stay tuned!
Posted by kzu