Waltzing with WSDL

One of my immediate goals for MockingBird is to have the “Configurator” UI done where you can just pick a WSDL on file or point to a URL and have the tool shred it and generate sample request and response messages into the correct folders (and generate the Handler Configuration) after which the user can then do any extra XPath based configurations to make the system more dynamic. Sounds easy enough, doesnt it? Well, thats what i thought too, until i delved into the intricacies of WSDL Parsing.

The class most people use is ServiceDescription and to be fair, it gives some information about the WSDL but navigating it is not easy at all and its not all that intuitive after you get to the point of listing all Operations inside the PortTypeCollection. There is also another big issue with trying to use ServiceDescription alone and that is the fact that WCF by default gives you multipart WSDLs. So getting hold of the WSDL from a WCF endpoint is not straightforward but one class that comes to the rescue is DiscoveryClientProtocol as Mike Hadlow points out in his article. This behavior can be changed as pointed out by Tomas Restrepo in hs post on “Inline XSD in WSDL with WCF” and Christian Weyer takes it up a notch in his article on Flat WSDL in WCF.

Anyway after pondering this a while, I decided it would be in my best interests to factor the code into a WSDLRetriever and a WSDLDescriptor. The Descriptor takes a dependency on WSDLRetriever (an interface of course, so its easy to mock) and expects  a single WSDL XDocument which it can work with. Now with this approach I dont care if the endpoint (if the WSDL is not on file) is a WCF endpoint or not, the Retriever does all that work. So the Descriptor can do its work and with that sorted i wrote up the first few tests for the Descriptor using a mock Retriever. So far so good.

Now, along with the requirement stated above, I also want to build a nice UI, where, you can not only do a single click to generate messages, but you can exercise finer control over the message content before storing them in the appropriate folders. So imagine if you will, a tree view which shows all the operations and the messages in them and immediately under the message name, the name of the complex type and additionally the body of the type (not in the tree node, but in an associated panel).

Of course, I’m not trying to write a completely generic WSDL parser, but something thats immediately usable by MockingBird and for other tools which can use a WSDL as input.

I had a look around at some blog posts and samples but none seemed to go as far as I wanted. I also took a look through the code for WSCF Blue  but couldnt make immediate use of that API. (I’ve since chatted with Buddhike about that so there may be something i could do with WSCF libraries in future). I looked at WebServiceStudio as the license allows reuse of the code there, but the code structure is simply awful. It works as is but everything is so deeply tied into the UI that it will take a feat of coding to extend it or pull out any reusable logic from it. I tried re-factoring a local copy but gave up pretty soon. So I had to work through the requirement myself.

Anyway, parsing WSDL is not easy at all. Once you get to the operation and message, you then have to wak through all schemas and their associated object tables to pick out the correct complex type corresponding to the message. This took me the best part of a few hours. Given that the user will not supply any namespaces, it gets very hard indeed. Thankfully the Schema Object Model is very decent. I used LINQ to XML for little bits such as extracting endpoint information (which also required me to resort to XPath extension methods because LINQ insisted i give it qualified names but I didnt know what the namespaces were !!) but for the schemas in WSDL, i think the SOM is better than LINQ. But I’m happy to be proved wrong!!

So I’ve got it to the point where it brings back the complex types. Now to generate sample instances from that for which I’ve got some resources. Once thats done and a simple Retriever implemented it wont be hard to put a UI on that because none of this is tied to the UI.

Thats it for now, a sort of ‘progress’ post for those waiting for the GUI for MockingBird and also to point out some of the dragons that await those who venture into the WSDL arena. Talk about ‘design by committee.. its a minefield!!

Couple of really good articles are Understanding WSDL by Aaron Skonnard (an old one, but an excellent read) and Walking the SOM by Stan Kitsis

Till next time

Introducing ‘Mockingbird’

In my previous post, I briefly mentioned a tool that helped with the testing of web-service consumers. I’m happy to announce that this is now publicly available on CodePlex under the moniker “MockingBird’. Here are some details about the tool – where it came from, how it helps and where it is going. (Most of this info is on the CodePlex space, but i’ll reproduce it anyway).

The Scenarios for MockingBird

(1) Imagine you are given the WSDL for a third-party web-service but no functioning system is available yet (it may be a brand new service or perhaps dev/site licenses are being negotiated). You need to get on with development right now. What do you code against? You maybe a ‘TDD & Mock Objects’ savvy dev which will help in many cases, but what if you ‘don’t do’ TDD & Mocks? Or it may be that you are maintaining/enhancing an existing system that wasn’t coded against interfaces etc. Or what if you are a BizTalk developer? You cannot mock/inject dependencies into your orchestrations and other components (well, except for pipeline components, but thats another story)!!

(2) Next, imagine that you are setting up a build server and multiple environments (DEV, TEST, UAT etc). But now the vendor says you can only have one license for their software. Now how do you run DEV, TEST, UAT in parallel with different data sets? Or you may have more than one license, but what if that service has maintenance schedules that clash with your build? Your build server is then completely exposed to something you don’t control.

I’ve expanded on these scenarios in the tool documentation here, but essentially these are the two main scenarios that MockingBird targets.

The Origins of MockingBird

MockingBird started life as MockWeb an internal tool that my former colleagues (Senthil Sai and Will Struthers) and I developed. All credit must go to them as Senthil first came up with the concept and Will then contributed a lot to the code-base.

It started when we got rather fed up with having to set up multiple instances of a third party service and build data-sets just to help with testing our BizTalk orchestrations reliably. It grew rapidly and organically. While we felt that this concept and tool would be useful to the .NET dev community at large (not just BizTalk teams), the structure of the codebase (at the time) would not lend itself to easy extension and needed to be refactored before making it publicly available. Its taken a while , but now I’ve completely rewritten MockWeb and as I have mentioned in the roadmap, I intend this to go beyond HTTP Web Services and hence the new name.

Whats with the name – ‘MockingBird’ ? Sounds daft !!

Ok, so its not a tool that sits there and laughs at you when you try to test !! . Thats not what the name is intended to convey. Wikipedia says that “They (mockingbirds) are best known for the habit of some species mimicking the songs of insect and amphibian sounds as well as other bird songs,[1] often loudly and in rapid succession” .

So the intent is to mimic web-services and I quite like the name ‘MockingBird anyway!!

The current implementation

There are two main elements to the current implementation

• An ASP.NET HttpHandler which is configured to return pre-set responses
• A WinForms GUI to setup the handler and associated configuration for a given WSDL.

The Roadmap

• The first thing I intend to do is revise the implementation to be fully WCF based as i want to be able to intercept calls irrespective of transport protocol. This is 0ne of the reasons i rewrote the code-base and i hope there wont be too much churn as i extend it
• The second thing is to make it a sort of platform to test webservices, generate BizUnit test steps, maybe C# test fixtures, possibly to code-gen Biztalk artifacts from given WSDLs and schemas.
• Thirdly to take it into the realm of mock Biztalk adapters. WCF is key to that. I envisage this as being a pair of custom WCF send and receive adapters that can dynamically be configured.
• Beyond that, lets see .

So, check it out. There’s an alpha release currently available and your feedback will be appreciated. If you have any thoughts on how the mock Biztalk adapters could be done, I will be glad to hear and of course, if you want to join the project, you will be welcome.

On the ‘Opacity’ of Orchestrations

Richard Hallgren recently sent me a link to a question that was posted on StackOverflow dealing with “Mocking Web Services consumed  by Biztalk Request Response ports” and asked for my thoughts on the subject as BizUnit and BizUnitExtensions were both mentioned. There were a couple of posts there at the time, one questioning why bother, another directing him to use SoapUI and another really good reply by David Hall on the topic of mock BizTalk adapters (which I want to talk about in some more depth later). David’s points helped me crystallize some of my thoughts on the subject of testing orchestrations and I replied there and figured that it was a good subject to blog about in more detail.

David raised the point about there not being real ‘unit tests’ in BizTalk (at least not unit tests in the purist sense of the word) and that the ‘opaque’ nature of orchestrations made it very difficult to actually ‘unit’ test them. He did also raise the point about BizUnit being a bit of a misnomer (the ‘Unit’ part being somewhat misleading to someone new to BizTalk). I do agree with the points he raises there. Here are some of the things I wrote there and which I will expand upon to some extent in this post and some extra points.

One approach which I have recently got sold on is the use of mock object frameworks (such as MoQ, RhinoMocks, TypeMock and NMock2). I used NMock2 in my last ASP.NET project and have recently started with MoQ which i think is much better and i’m really getting into the lambdas.. they make expressing intent so much more concise. I’ll write a bit more about how I’m using them in another post, but suffice to say here that, although I havent tried it myself, several Biztalk folk have successfully used mock objects to test pipeline components.

Orchestrations unfortunately, in this respect (of being able to completely control what their dependencies are and ‘faking’ the behavior of the dependency) can be considered opaque. But there are good reasons for that.

a) Because of the huge subscription system in the message box – that orchestrations use when being activated etc, it is not possible to fire up some “virtual” process to host the orchestration (I think Tomas’s Pipeline Testing Library does something along these lines for pipeline components, but I havent had the opportunity to use that so far although i have heard good things about it).

(b) Now for arguments sake, lets assume you could write such a virtual host. Now how would this virtual process handle persistence and dehydration?. I’d wager that people using WF would have the same problem in trying to test the workflow fully. (Soon after writing this I came across Matt Milners MSDN article on unit testing workflows, but workflows are just CLR code)

(c) Although you can see the C# equivalent of the ODX, we dont develop with the C# equivalent directly, so there is no way we can “inject” a mock interface into the orchestration code. Orchestrations are at a higher level of abstraction than ‘normal’ code so while you may declare variables and use them in expression shapes etc, its not the same as writing getters and setters in regular C#.

(d) An orchestration is not really a “unit”.  It is a composite element. The units are the messages going to and from the message box and the external components called through expression shapes.So even if you could inject a mock webservice interface you cannot inject mock message boxes and correlation sets and other things.

Of course, it may be possible to get more testability for orchestrations in future versions of the product (beyond v.2009), but this is the situation now. If there are other approaches you use to dealing with the innards of orchestrations then do let me know.

In terms of getting more info on the orchestration itself, one thing that can be done (and i’ve been considering an addition to the BizUnitExtensions library to do this) is to link in with the OrchestrationProfiler tool a(s that tool gives a pretty detailed report of all the shapes ). Now from the BizUnit test we can verify that individual steps were executed and perhaps also check the time it took for execution. This is a bit like setting expectations for mock objects. This could go quite far in making the orchestration a bit more of a white box.Also considering that the orchestration debugger shows a lot of the variable values, surely it must be possible to get that info via an API (one hopes!!) to show what the values of variables were at a given point for a given instance.

One thing which a lot of people get hung up on is terminology (unit tests vs. functional tests). I dont know why its hard for some folk to accept that the granularity of a unit is entirely subjective and depends a lot on the system. For a system like Biztalk, especially with orchestrations, in my opinion, the unit of testing includes the pickup of the message from the source, the processing and its dispatch. A functional test will expand on that to include things like getting the data into the source and an integration test will span the entire system, and of course, since the solutions built with Biztalk are quite different this definition of whats involved in each category of testing needs to be ‘adapted’ for each project.

Back to Richard’s question though, my previous dev team had a solution. Basically what we did was to write a generic configurable HttpHandler that parsed incoming service requests and returned pre-set responses. The response sent back was configurable based on conditions such as XPath. In the BUILD and DEV binding files, the webservice end point was the mock. This worked brilliantly in isolating the BUILD and DEV environments from the actual third party webservices. This also helped in a “contract first” approach where we built the mock endpoint and the orch developer used it while the webservice author went ahead and built the actual service behind the endpoint. In my next post, I’ll describe this in more detail as I have now put it on CodePlex.

As my previous team wasnt into traditional mock objects, we used this quite successfully for both Biztalk consumers and non Biztalk consumers. If you are not dealing with Biztalk , and if you have control over the consumer (especially if you are writing it from scratch), then i would recommend a mock object framework.