BizTalk Documentor Update

Darren has made available a new release (v.3.2.1) of the BizTalk Documentor. It fixes some long standing issues and also some issues that came up when used with BizTalk 2009. Check it out and let us know if you run into any problems.

We are planning some re-work of this tool as well as the venerable Orchestration Profiler  with a view to combining these tools and introducing new output types. We are also looking at fleshing out an API so that they could be used with other tools or in a programmatic way as well. For instance, in an earlier post,  I had voiced a wish to be able to link OrchProf with BizUnit so we could verify step execution etc directly from a test and make the test a bit more “white box”  (ok, so it wont be as whitebox as BizMonade but it could add more reach for BizUnit)   but at present there isnt an API that is surfaced to allow such a link. But this will change soon. 

I have some ideas for other features as well which I will write about soon and there are a couple of requests on the forum to be able to link the documentor with BizTalkMapDoc which I will look into.

So if you have any feature requests, feedback and other ideas on how these tools could work together in your projects do send them along, preferably on the CodePlex workspace forums or feel free to contact me directly through the blog and let me know.

BizTalk, WCF and the Bad Gateway

We ran into an interesting problem a couple of days ago and I decided it would be worth posting the solution in case anyone else runs into it.

The scenario involves a Windows Service communicating with a BizTalk WCF endpoint which is hosted using WCF-CustomIsolated and some fairly complex custom bindings, notably using X.509 Mutual Certificates. This is the second half of the communication, the first being where BizTalk sends a request to a vanilla WCF endpoint and gets an ACK, the service does some processing and the WinService then does a “call back” with the “real response” / outcome of the processing.  Both communication routes are secured via the same bindings. We didnt have any trouble with the transmission from BizTalk to the WCF service but the WinService initiated communication always failed with a HTTP 502 : Bad Gateway message. This had us stumped for a while as some articles we read seemed to suggest that there was a problem on the server. Now if you have looked into custom bindings and certs in BizTalk, you will know that there is some really heavy duty and scary stuff there so we wondered whether the way we had setup the certs and credentials info was playing havoc with the communication.

So I used the trusty WSCF.blue and mocked up another WCF service to mimic the BizTalk endpoint replete with the bindings (no, WSCF Blue doesnt support custom bindings yet, but will sometime soon) and swapped the URI in the WinService and it worked just fine. Still puzzled, I then created a console app to mimic the winservice and it was able to communicate with both my mock BizTalk service as well as the real endpoint. Curiouser and curiouser. At that time the only difference was in the App Pool between the BizTalk service and mock service, but i used the same app pool and still the problem only manifested between the Winservice and Biztalk.

Anyway, with some guidance from an expert colleague (who, unfortunately wasnt around when I was writing those mock clients and services) It turned out that the problem was with a setting “UseDefaultWebProxy”  in the Win Service. The difference between the WinService and the console client was that the console was running under my credentials and in the IE settings I had turned off the proxy for the specified URL so although I didnt specify the setting as “false” in the client it automatically picked up my settings. But the WinService was running under another user account, and so the system didnt care about my proxy settings and we had to explicitly set the value to “false”. 

That was it. A little setting buried deep in the bindings that caused us the grief. After sorting this out I stumbled across this post from Kenny Wolf  which points to the offending setting. Wish I had found that earlier, but then I wouldnt have realized that aspect of the console client where it picked those settings. So, hope this helps someone in a similar situation.

To DAL or not to DAL

Do BizTalk consultants need to care about Data Access Layers? Does a BizTalk solution really need a DAL?  These are the questions that I’ve been mulling over in the past few weeks. Let me explain.

There are a couple of places where a BizTalk solution encounters a DAL. The first is where the DAL acts as an integration enabler. Here the endpoint of the LOB application we are integrating with happens to be a database. The second is where the DAL acts as a process enabler. Here the DAL provides the underpinning of the business process (that is, as part of the business process, it is frequently necessary to update a database with the state of the business document being operated on).

In my current gig, we are using both BizTalk and SSIS. SSIS is great for the ETL and various data related actions. BizTalk then takes over and passes the data to an LOB application doing various business processes as part of that communication. The nature of the processes is such that there is a significant DAL. Early on in the project we went through the usual debate on whether a custom DAL was necessary or if we should just use the requisite database adapters. Isnt the database adapter an obvious choice?  Maybe, or maybe not. In an earlier post , i talked about just such a situation a few years ago where we had choose whether to link directly to the DB or wrap the system in a web-service first and as i explained, things didn’t turn out the way they were expected to.

So, what are the considerations?

1. Firstly, (as I explained in the post and the follow up posts) one of the key issues is the level of abstraction you are given. Especially when dealing with the scenario of integration enablers, a database endpoint is very rarely coarse grained enough to support a service oriented approach. Its more likely that you will be provided with CRUD level interfaces. Even if you decide to direct all communication via an orchestration that wraps all this, how does the orchestration actually call the backend system? Go via the adapter or use a DAL?
2. For the scenario of process enablers, abstraction comes into play again. You don’t want to be cluttering up your orchestrations with bits and pieces of database schema related stuff. You could choose to wrap the database calls in a coarser stored proc but this leads to the next key point which is
3. Performance. If you have a number of send ports (for all these stored procs) in the middle of your orchestrations, there is a cost associated with all those persistence points. If your transaction handling requirements permit, you could think about wrapping some of those calls in atomic scopes, but you have to be  very careful with this. If you do encounter an issue and everything gets rolled back, are your processes really designed to start at the right place all over again without compromising data integrity?
4. If your DAL is designed well, your orchestrations will benefit from having to call methods on business level entities and, just from a persistence point consideration, will, in my opinion, be better off.
5. Transaction Bridging : There were a few situations where we had to bridge the transaction across the database and the segment of the business process. Fortunately, the DAL being of extremely high quality (courtesy of an expert colleague) made this very easy to do.

But, having said all this, a DAL doesn’t come free. You have to write code. Sometimes lots of it. The more code you write, the higher the probable bug density. If the functionality can be satisfied with a code-generator then that will reduce the code you have to write, but it DOES NOT reduce the amount of code you have to MAINTAIN. I think many developers forget about this last point. I’m all in favour of code-gen, but don’t forget the maintenance cost.  (Further, if the functionality in the middle of your processes can be satisfied with boiler plate code, perhaps it’s an opportunity to question what it’s doing there in the first place. Can it be pushed to a later stage and componentized? )

I must confess, at one point, when wading through a sea of DAL code early on in the project, I was quite tempted to throw it all away and go for the adapters, but the considerations above outweighed the pain at that point. Now much later, with everything having stabilized, we know just where to go to make any changes and the productivity is quite high.

But I’ve seen cases where BizTalk developers didn’t care about the SQL they wrote and they ended up in a mess with locking and poor performance. And it takes a really good developer to write a first class DAL and having interviewed and worked with a number of devs I can say that its hard to find good skills in this area. Pop quiz: Do you know how to use System.Transactions yet ? 

There is always the option of using something like NHibernate. If you use some coarse grained stored procs and some business entities, you could kill all the “goo” in the middle by letting NH take care of the persistence. That, i wager would reduce the bug count in that area. But watch out for the maintenance times and the bug fixing. When there’s a component in the middle that you don’t know the internals of, it can make life very hard when trying to track down bugs.

That leads me on to the point of making choices based on knowledge and not ignorance. If you want to adopt “persistence ignorance”, don’t do it because you cant write proper DAL code yourself. Do it for the right reasons.

So I hope the points above have given some food for thought. Custom code is not always bad as long as it is approached and implemented correctly. Whether you choose to use a DAL or not, do it with careful thought on issues like the ones above. As always, your feedback is welcome.

Technorati Tags: BizTalk,DAL,NHibernate

MockingBird@SBUG

The UK SOA BPM User Group (SBUG) is holding a series of mini-meetings over the next few months. These are short live meetings / webcasts. I’m delighted to be presenting my web-service mocking tool , MockingBird at the next mini-meeting to be held on the 29th July.

I will be discussing how MockingBird assists and simplifies integration testing with web services, how it is used in Biztalk and non Biztalk scenarios, how the system works, how it can be extended and the roadmap.

Please check the event link on the SBUG site for details.  All are welcome to join (no membership needed). Hope to see a good turnout

BizUnitExtensions_v3.0 Beta-1

Finally made some progress on BizUnitExtensions and have released Beta-1 for the new version which now complies with BizUnit 3.0’s object model. This release specifically centres around the main Extensions and the Extensions.BizTalk  libraries.

Main changes

• The Altova library has now been deprecated as there is much better xml validation support in the .NET Framework so the altova library is redundant.
• The steps for manipulating applications, receive locations and send ports have now been completely overhauled, so there is a separate ApplicationConductorStep, ApplicationGetStatusStep and ApplicationVerifyStatus step and likewise for ReceiveLocations and Send Ports. The GetStatus steps will load the current status into the context which can be useful when we need to use that info in subsequent steps. VerfyStatus will check the status against a given value.
• The ReceivePortConductorEx step in the old code-base has been transitioned to the above mentioned set of ReceiveLocation steps as it really was dealing with locations rather than the ports. There is a new ReceivePortConductor step which allows the bulk enabling and disabling of all the locations in that port.
• Updates to the Orchestration Conductor allowing a STOP action on an orchestration.
• The namespaces and assembly names have all been revised (for example, using the Biztalk steps requires the assembly name to be BizUnit.Extensions.BizTalk.dll) so your existing test cases will need to be updated.

Forthcoming Updates and Releases

• Bram Veldhoen has contributed a fair amount of code including steps to link in with the Pipeline Testing Library. I will incorporate that into Beta-2 as I need to work through all the steps and the associated helpers and utility classes.
• I have updated the Oracle and EntLib libraries as well, but haven’t had any time to test them. As I now have Oracle XE, I should be able to make some progress on that pretty soon. With EntLib, its a little more complicated. As I didnt write those steps myself, I am not too familiar with the code, so it will take a while to update. So for the present, its still using EntLib 3.1. 

Workspace Updates

• The codeplex workspace has also been cleaned up and slimmed down drastically. A lot of the pointers to tutorials etc are all largely redundant anyway.
• The Roadmap page has been completely rewritten. The framework has come a long way since that material was written. The only stuff left there is a list of probable new steps and some ideas for tooling around BizUnit which may make it into some future version of the framework.

Hope you find the updates useful. Let me know if you encounter any bugs or if you have any suggestions.

MockingBird – Beta

Whew! I’ve finally managed to get the first beta release of MockingBird out the door, almost 3 months after the alpha. The past 3 months have gone by in a blur with my first major project in MCS and the associated tight deadlines and long hours. Technically its been a good learning experience, but it doesnt leave any time at all for side projects like this one, but I’ve managed to steal the odd weekend here and there and get this done.

The key elements of the beta release are

• The branding and naming is now beginning to work out quite well. As I had written earlier, it started life as MockWeb, then became MockingBird and now I have decided that I like the name “MockingBird” so much its worth promoting to the ‘brand’ or the name of the toolset. The constituent applications now have their own names. The web application with the HttpHandler is now called the “Service Simulator” and the main WinForms application is now called the “Service Studio” which will hold many modules and plugins.
• The Service Simulator has been completely revised using Unity IoC. I feel a lot happier with the design now and i think its more loosely coupled and the DI /IoC has a good side effect in increasing the test surface. The Service Studio doesnt use the Unity configuration yet, but will do so by RTM.
• The little ‘Driver’ GUI of the alpha has now been rolled into the “Service Studio” and is officially called the “Service Invoker”. The layout has changed and now uses tabs so occupies less screen real estate.
• There is a new “WSDL Browser” module in the service studio that allows you to load up a WSDL and it will parse the WSDL for you and list all the operations and allow you to generate sample request and response messages.
• The WSDL  Browser in turn makes use of an Xml Instance Generator module that provides a more usable facade over a sample from the MSDN archives. The XIG module can stand on its own and will grow in future releases.

The design principle of the Service Studio is that every piece of functionality must be accessible independently and not depend on a UI. This is one of the reasons it took so long to get this done. I came across quite a few samples of WSDL handling where the authors were willing to share and/or the license permitted extension, but the code was simply too deeply hooked into the UI and did not have any “API” to speak of. In the end I wrote the whole thing myself. I’m still looking for ways to plug it into related tools such as WSCF which I also participate in developing.

There are some limitations in the beta especially with regard to WSDL Parsing.

• The first limitation is that it cannot handle WCF multi-part WSDLs. This will be done by around RTM time or along with the V2 release as V2 should be fully WCF based.
• While developing i tested it with different types of WSDLs but in the final push I focussed on testing it with a local monolithic WSDL and so it may fall over with complex WSDL documents. The WSDL spec allows for a wide variation in the actual WSDLs so its not possible for me to cater to all possible kinds of WSDLs. I hope that if you come across an issue, then the source is easy to fix or that you can share the WSDL with me and I will try and sort out the parsing.
• I also only finalised the testing with a “local” WSDL on the file system. The “SimpleWSDLRetriever” class that I have implemented does not use DiscoveryClientProtocol (although i shall plug in a new class soon that does this) so it cant handle connecting to a non local service and downloading the contract and associated metadata. (As you can see, I know how to address the limitation – just havent had the time to write it   ).

Anyway, those limitations aside, I’m happy with the code base so far and it really is quite stable and if you need to extend and use the source, i think the current structure is easy to understand. On the codeplex space I have written more documentation on how the internals of the system work so hopefully that will be useful to you. If you do end up extending it, I hope you will let me know what you are doing with it and maybe I can incorporate some of those extensions if it will make the product more generic.

As always, please send me feedback, especially if you run into any bugs. Hope you like the new release.

Look Ma ! An evil middleware product!

I couldn’t resist commenting on this issue. I was just doing some final prep for my VBUG talk tomorrow and came across Richard Hallgren’s oddly titled post – Does BizTalk have man-boobs?. Richard writes about a QCon webcast of a session done by Martin Fowler and Jim Webber and writes

“Their main point is that we use much to bloated middleware (BizTalk is mentioned as an example here) and that we should have a more agile approach to implement SOA and ESBs. They’ve used all the agile ideas (testing, isolated testable functionality, deliver small and often, continuous builds etc, etc) and applied them to integration..”.

Richard goes on to make some good points which I totally agree with. Check out the post for details. I commented on the post and decided I would make those points again here and add a couple more, which is also quite good timing (for me) considering I’m speaking on BizTalk tomorrow.

So, I agree that BizTalk is totally unsuitable for small situations and if it is naively (read – most often) used without any performance tuning whatsoever, then its latency can be bad. But if its tuned, it can totally rock (and I know this from MCS colleagues who’ve experienced it in very large BizTalk projects).

BizTalk is big. Yes, totally. But it has to be because it addresses a vast set of use cases. Same goes for its competitors like TIBCO, webMethods etc. If you dont want all the features, dont use them. They are in the box anyway and they wont slow you down if you dont use them. You want content based routing – its in the box; you want long running business processes – check, you want ‘aspect oriented’ interception of messages for tracking , you’ve got BAM. You want monitoring tools – check. You don’t want tracking – turn it off. You dont want business rules – ignore the BRE. The list goes on.

There’s also a case to be made for using something like the ESB toolkit to give you a decent jumpstart on your routing infrastructure.

If you only have a couple of systems to integrate (and you are very sure there wont be more) then go ahead and custom code it. Its not worth buying a product for that.

The problem I have is with some “agilists” who, it would seem, want to custom code every darn thing under the sun. As I commented on Richard’s post, the irony is that the very same folk will then stress the importance of having good, robust pre-built frameworks and good tools to help with the “agile” approach. But wait, those tools have to be on the “acceptable” short list. Take NHibernate for example. An excellent tool, no doubt. And for many folk, since its open source, so it must be sent from above, totally divine, but not BizTalk. That’s way too big. And besides, its from Microsoft. Gasp! horror! It must be evil !!

Some take the view that its about having confidence in your code. I can understand that. Having been exposed to a fair amount of TDD etc, I can attest to the feeling of security when your edge cases have all been tested and you see all those “green signals”. But in an integration scenario where use cases are similar , how many times will you write a test, write the code, refactor, refactor, blah.. till the code comes out of your ears. Pretty soon customers are going to wonder how many times they have to pay for something thats already been written.

If Ican have confidence in code I write, I can have the same amount and more in a product thats been tested out in hundreds of more scenarios than I could imagine. Do these products (such as Biztalk and its competitors – I’m not selling anything here ) have bugs? Well of course they do. But in many cases, you can be just as confident in commercial closed source as in open source. Besides, if pre-built tools werent a good idea, there wouldnt be any open or closed source tools would there?

I’m not partial. Go in for the Neuron ESB if you a want a pure WCF ESB, or go in for something thats gaining a good reputation like NServiceBus. Just don’t give me that story about having to write a 100 tests first and inherit a dozen interfaces before I can deliver anything of value. Bah! Humbug!

Ok, so enough of the rant.. Got to go and write some custom components now 

Speaking at VBUG (Bracknell)

I will be presenting an introductory session to Biztalk 2006 R2 (and spending some time on the WCF adapters) at the local chapter of VBUG in Bracknell. Of course, we will also spend a little while discussing the impact of Oslo and Dublin and the factors to keep in mind when choosing your integration technologies.

Despite the fact that it has been around for ages and has IMO a lot of visibility, it seems that BizTalk Server still remains a sort of niche product with a lot of developers totally unfamiliar with it and what it could give them. The last time I presented on Biztalk,WCF and WF at the VBUG conference  in 2008, we found that a significant number of the audience didn’t know anything at all about BizTalk and from discussing with contacts in other countries as well, it seems that its a familar situation . Well, anyway, I hope I can do something in my own small way towards rectifying that.

So if you’re in the area and Biztalk is unfamiliar territory why not check it out?

BSF meets Deployment Framework

This post is just to call out some work that Bram Veldhoen has done in linking the Biztalk Software Factory to the venerable Deployment Framework. One thing to note is that the integration was done to the NAnt version of the framework prior to the latest MSBuild version of the framework (5.x). I guess linking into the new MSBUILD version is next on the cards and I’ve chatted to Bram about the possibility of doing something jointly (as if i didnt have enough irons in the fire with MockingBird and BizUnitExtensions ). Check it out and post feedback on the BSF codeplex space.

Been there, done that, got the certificate!

Well, I finally overcame my continued procrastination in this area and sat the MCTS: BizTalk exam today. I always knew I was ‘certifiable’ and now I got the paper to prove it   I cleared it with a score of 899 out of 1000. I’m quite chuffed with it. It revealed a couple of areas I need to work on a bit more but it was a good experience overall.

I know a lot of people debate the merit of certifications in general (and there has been lots of debates on this particular one as well), but I’m not going to re-hash them here. My aim in writing this was to give me a spur to read through areas that I havent had the opportunity to work in and also in a depth that may not have been necessary in the past. As Saravana points out , if you seriously study for this, you get a good feel for the breadth and depth of the product.

Studying can never take the place of actual hands-on experience, but with any large framework or product its common to find yourself working in a ‘niche’ area and never having the chance to explore all that it offers. Biztalk being a huge product, it’s quite ‘normal’ to find developers who have never had to use things like TPM/RoleLinks or the BRE and even BAM (actually, i think BAM is criminally underused, but thats a topic for another post). The study process can open up areas for you to look into and improve your productivity which you may not even attempt otherwise.

I have been quite selective in my approach to certifications thus far, the only one I bothered to write was 70-320 (Building Web Services and  Server Components), that too 4 years ago, but at the end of that study process I was totally amazed at the breadth and depth of the .NET stack and was far more confident of my grasp on Web Services, Enterprise Services, Remoting etc than I had been till that point. I think that going forward that (selective approach) will continue. The R2 exam is now available so that’s next on the agenda and working on the WCF exam should be a useful process as well (assuming I ever get off my lazy backside and take them :-) )

I thought that Id’ just point out a few resources that were of great help.

• Saravana Kumar’s awesome 70-235 Exam Preparation Diary.
• Professional Biztalk - the best explanation of BAM, ever . Period.
• The BAM Training Kit - specifically the hands on exercises. The training material itself is pretty complex in some areas and I personally didnt get much out of those explanations, but the practice was good.
• For BRE, I honestly couldn’t find any single resource which allowed me to get to grips with it. There were some nuggets scattered here and there. I ended up writing my own study paper where I combined notes from all the books and whitepapers into something I could revise easily. The highlights of all the resources were the awesome BRE Tips and Tricks – MSDN Webcast – by Jeff Seagard and the MSDN BRE Walkthroughs . The MSDN exercises were exceptionally good. Saravana also points out (in his post) another techNet webcast on the BRE, but i didnt get a chance to watch that.
• Biztalk 2006 Recipes – I used this for a quick final review of BAM and BRE, Orchestrations etc.

So, if you are considering writing the exam, I’d say go for it. Working on the numerous areas involved can only make you a better developer (or at least a more ‘aware’ developer). And when you are done with it, look for areas that you can feed back the learning gained into your daily development tasks.IMO, the real payback is there. Good luck.!!