To DAL or not to DAL

Do BizTalk consultants need to care about Data Access Layers? Does a BizTalk solution really need a DAL?  These are the questions that I’ve been mulling over in the past few weeks. Let me explain.

There are a couple of places where a BizTalk solution encounters a DAL. The first is where the DAL acts as an integration enabler. Here the endpoint of the LOB application we are integrating with happens to be a database. The second is where the DAL acts as a process enabler. Here the DAL provides the underpinning of the business process (that is, as part of the business process, it is frequently necessary to update a database with the state of the business document being operated on).

In my current gig, we are using both BizTalk and SSIS. SSIS is great for the ETL and various data related actions. BizTalk then takes over and passes the data to an LOB application doing various business processes as part of that communication. The nature of the processes is such that there is a significant DAL. Early on in the project we went through the usual debate on whether a custom DAL was necessary or if we should just use the requisite database adapters. Isnt the database adapter an obvious choice?  Maybe, or maybe not. In an earlier post , i talked about just such a situation a few years ago where we had choose whether to link directly to the DB or wrap the system in a web-service first and as i explained, things didn’t turn out the way they were expected to.

So, what are the considerations?

1. Firstly, (as I explained in the post and the follow up posts) one of the key issues is the level of abstraction you are given. Especially when dealing with the scenario of integration enablers, a database endpoint is very rarely coarse grained enough to support a service oriented approach. Its more likely that you will be provided with CRUD level interfaces. Even if you decide to direct all communication via an orchestration that wraps all this, how does the orchestration actually call the backend system? Go via the adapter or use a DAL?
2. For the scenario of process enablers, abstraction comes into play again. You don’t want to be cluttering up your orchestrations with bits and pieces of database schema related stuff. You could choose to wrap the database calls in a coarser stored proc but this leads to the next key point which is
3. Performance. If you have a number of send ports (for all these stored procs) in the middle of your orchestrations, there is a cost associated with all those persistence points. If your transaction handling requirements permit, you could think about wrapping some of those calls in atomic scopes, but you have to be  very careful with this. If you do encounter an issue and everything gets rolled back, are your processes really designed to start at the right place all over again without compromising data integrity?
4. If your DAL is designed well, your orchestrations will benefit from having to call methods on business level entities and, just from a persistence point consideration, will, in my opinion, be better off.
5. Transaction Bridging : There were a few situations where we had to bridge the transaction across the database and the segment of the business process. Fortunately, the DAL being of extremely high quality (courtesy of an expert colleague) made this very easy to do.

But, having said all this, a DAL doesn’t come free. You have to write code. Sometimes lots of it. The more code you write, the higher the probable bug density. If the functionality can be satisfied with a code-generator then that will reduce the code you have to write, but it DOES NOT reduce the amount of code you have to MAINTAIN. I think many developers forget about this last point. I’m all in favour of code-gen, but don’t forget the maintenance cost.  (Further, if the functionality in the middle of your processes can be satisfied with boiler plate code, perhaps it’s an opportunity to question what it’s doing there in the first place. Can it be pushed to a later stage and componentized? )

I must confess, at one point, when wading through a sea of DAL code early on in the project, I was quite tempted to throw it all away and go for the adapters, but the considerations above outweighed the pain at that point. Now much later, with everything having stabilized, we know just where to go to make any changes and the productivity is quite high.

But I’ve seen cases where BizTalk developers didn’t care about the SQL they wrote and they ended up in a mess with locking and poor performance. And it takes a really good developer to write a first class DAL and having interviewed and worked with a number of devs I can say that its hard to find good skills in this area. Pop quiz: Do you know how to use System.Transactions yet ? 

There is always the option of using something like NHibernate. If you use some coarse grained stored procs and some business entities, you could kill all the “goo” in the middle by letting NH take care of the persistence. That, i wager would reduce the bug count in that area. But watch out for the maintenance times and the bug fixing. When there’s a component in the middle that you don’t know the internals of, it can make life very hard when trying to track down bugs.

That leads me on to the point of making choices based on knowledge and not ignorance. If you want to adopt “persistence ignorance”, don’t do it because you cant write proper DAL code yourself. Do it for the right reasons.

So I hope the points above have given some food for thought. Custom code is not always bad as long as it is approached and implemented correctly. Whether you choose to use a DAL or not, do it with careful thought on issues like the ones above. As always, your feedback is welcome.

Technorati Tags: BizTalk,DAL,NHibernate

MockingBird@SBUG

The UK SOA BPM User Group (SBUG) is holding a series of mini-meetings over the next few months. These are short live meetings / webcasts. I’m delighted to be presenting my web-service mocking tool , MockingBird at the next mini-meeting to be held on the 29th July.

I will be discussing how MockingBird assists and simplifies integration testing with web services, how it is used in Biztalk and non Biztalk scenarios, how the system works, how it can be extended and the roadmap.

Please check the event link on the SBUG site for details.  All are welcome to join (no membership needed). Hope to see a good turnout

BizUnitExtensions_v3.0 Beta-1

Finally made some progress on BizUnitExtensions and have released Beta-1 for the new version which now complies with BizUnit 3.0’s object model. This release specifically centres around the main Extensions and the Extensions.BizTalk  libraries.

Main changes

• The Altova library has now been deprecated as there is much better xml validation support in the .NET Framework so the altova library is redundant.
• The steps for manipulating applications, receive locations and send ports have now been completely overhauled, so there is a separate ApplicationConductorStep, ApplicationGetStatusStep and ApplicationVerifyStatus step and likewise for ReceiveLocations and Send Ports. The GetStatus steps will load the current status into the context which can be useful when we need to use that info in subsequent steps. VerfyStatus will check the status against a given value.
• The ReceivePortConductorEx step in the old code-base has been transitioned to the above mentioned set of ReceiveLocation steps as it really was dealing with locations rather than the ports. There is a new ReceivePortConductor step which allows the bulk enabling and disabling of all the locations in that port.
• Updates to the Orchestration Conductor allowing a STOP action on an orchestration.
• The namespaces and assembly names have all been revised (for example, using the Biztalk steps requires the assembly name to be BizUnit.Extensions.BizTalk.dll) so your existing test cases will need to be updated.

Forthcoming Updates and Releases

• Bram Veldhoen has contributed a fair amount of code including steps to link in with the Pipeline Testing Library. I will incorporate that into Beta-2 as I need to work through all the steps and the associated helpers and utility classes.
• I have updated the Oracle and EntLib libraries as well, but haven’t had any time to test them. As I now have Oracle XE, I should be able to make some progress on that pretty soon. With EntLib, its a little more complicated. As I didnt write those steps myself, I am not too familiar with the code, so it will take a while to update. So for the present, its still using EntLib 3.1. 

Workspace Updates

• The codeplex workspace has also been cleaned up and slimmed down drastically. A lot of the pointers to tutorials etc are all largely redundant anyway.
• The Roadmap page has been completely rewritten. The framework has come a long way since that material was written. The only stuff left there is a list of probable new steps and some ideas for tooling around BizUnit which may make it into some future version of the framework.

Hope you find the updates useful. Let me know if you encounter any bugs or if you have any suggestions.

MockingBird – Beta

Whew! I’ve finally managed to get the first beta release of MockingBird out the door, almost 3 months after the alpha. The past 3 months have gone by in a blur with my first major project in MCS and the associated tight deadlines and long hours. Technically its been a good learning experience, but it doesnt leave any time at all for side projects like this one, but I’ve managed to steal the odd weekend here and there and get this done.

The key elements of the beta release are

• The branding and naming is now beginning to work out quite well. As I had written earlier, it started life as MockWeb, then became MockingBird and now I have decided that I like the name “MockingBird” so much its worth promoting to the ‘brand’ or the name of the toolset. The constituent applications now have their own names. The web application with the HttpHandler is now called the “Service Simulator” and the main WinForms application is now called the “Service Studio” which will hold many modules and plugins.
• The Service Simulator has been completely revised using Unity IoC. I feel a lot happier with the design now and i think its more loosely coupled and the DI /IoC has a good side effect in increasing the test surface. The Service Studio doesnt use the Unity configuration yet, but will do so by RTM.
• The little ‘Driver’ GUI of the alpha has now been rolled into the “Service Studio” and is officially called the “Service Invoker”. The layout has changed and now uses tabs so occupies less screen real estate.
• There is a new “WSDL Browser” module in the service studio that allows you to load up a WSDL and it will parse the WSDL for you and list all the operations and allow you to generate sample request and response messages.
• The WSDL  Browser in turn makes use of an Xml Instance Generator module that provides a more usable facade over a sample from the MSDN archives. The XIG module can stand on its own and will grow in future releases.

The design principle of the Service Studio is that every piece of functionality must be accessible independently and not depend on a UI. This is one of the reasons it took so long to get this done. I came across quite a few samples of WSDL handling where the authors were willing to share and/or the license permitted extension, but the code was simply too deeply hooked into the UI and did not have any “API” to speak of. In the end I wrote the whole thing myself. I’m still looking for ways to plug it into related tools such as WSCF which I also participate in developing.

There are some limitations in the beta especially with regard to WSDL Parsing.

• The first limitation is that it cannot handle WCF multi-part WSDLs. This will be done by around RTM time or along with the V2 release as V2 should be fully WCF based.
• While developing i tested it with different types of WSDLs but in the final push I focussed on testing it with a local monolithic WSDL and so it may fall over with complex WSDL documents. The WSDL spec allows for a wide variation in the actual WSDLs so its not possible for me to cater to all possible kinds of WSDLs. I hope that if you come across an issue, then the source is easy to fix or that you can share the WSDL with me and I will try and sort out the parsing.
• I also only finalised the testing with a “local” WSDL on the file system. The “SimpleWSDLRetriever” class that I have implemented does not use DiscoveryClientProtocol (although i shall plug in a new class soon that does this) so it cant handle connecting to a non local service and downloading the contract and associated metadata. (As you can see, I know how to address the limitation – just havent had the time to write it   ).

Anyway, those limitations aside, I’m happy with the code base so far and it really is quite stable and if you need to extend and use the source, i think the current structure is easy to understand. On the codeplex space I have written more documentation on how the internals of the system work so hopefully that will be useful to you. If you do end up extending it, I hope you will let me know what you are doing with it and maybe I can incorporate some of those extensions if it will make the product more generic.

As always, please send me feedback, especially if you run into any bugs. Hope you like the new release.

Look Ma ! An evil middleware product!

I couldn’t resist commenting on this issue. I was just doing some final prep for my VBUG talk tomorrow and came across Richard Hallgren’s oddly titled post – Does BizTalk have man-boobs?. Richard writes about a QCon webcast of a session done by Martin Fowler and Jim Webber and writes

“Their main point is that we use much to bloated middleware (BizTalk is mentioned as an example here) and that we should have a more agile approach to implement SOA and ESBs. They’ve used all the agile ideas (testing, isolated testable functionality, deliver small and often, continuous builds etc, etc) and applied them to integration..”.

Richard goes on to make some good points which I totally agree with. Check out the post for details. I commented on the post and decided I would make those points again here and add a couple more, which is also quite good timing (for me) considering I’m speaking on BizTalk tomorrow.

So, I agree that BizTalk is totally unsuitable for small situations and if it is naively (read – most often) used without any performance tuning whatsoever, then its latency can be bad. But if its tuned, it can totally rock (and I know this from MCS colleagues who’ve experienced it in very large BizTalk projects).

BizTalk is big. Yes, totally. But it has to be because it addresses a vast set of use cases. Same goes for its competitors like TIBCO, webMethods etc. If you dont want all the features, dont use them. They are in the box anyway and they wont slow you down if you dont use them. You want content based routing – its in the box; you want long running business processes – check, you want ‘aspect oriented’ interception of messages for tracking , you’ve got BAM. You want monitoring tools – check. You don’t want tracking – turn it off. You dont want business rules – ignore the BRE. The list goes on.

There’s also a case to be made for using something like the ESB toolkit to give you a decent jumpstart on your routing infrastructure.

If you only have a couple of systems to integrate (and you are very sure there wont be more) then go ahead and custom code it. Its not worth buying a product for that.

The problem I have is with some “agilists” who, it would seem, want to custom code every darn thing under the sun. As I commented on Richard’s post, the irony is that the very same folk will then stress the importance of having good, robust pre-built frameworks and good tools to help with the “agile” approach. But wait, those tools have to be on the “acceptable” short list. Take NHibernate for example. An excellent tool, no doubt. And for many folk, since its open source, so it must be sent from above, totally divine, but not BizTalk. That’s way too big. And besides, its from Microsoft. Gasp! horror! It must be evil !!

Some take the view that its about having confidence in your code. I can understand that. Having been exposed to a fair amount of TDD etc, I can attest to the feeling of security when your edge cases have all been tested and you see all those “green signals”. But in an integration scenario where use cases are similar , how many times will you write a test, write the code, refactor, refactor, blah.. till the code comes out of your ears. Pretty soon customers are going to wonder how many times they have to pay for something thats already been written.

If Ican have confidence in code I write, I can have the same amount and more in a product thats been tested out in hundreds of more scenarios than I could imagine. Do these products (such as Biztalk and its competitors – I’m not selling anything here ) have bugs? Well of course they do. But in many cases, you can be just as confident in commercial closed source as in open source. Besides, if pre-built tools werent a good idea, there wouldnt be any open or closed source tools would there?

I’m not partial. Go in for the Neuron ESB if you a want a pure WCF ESB, or go in for something thats gaining a good reputation like NServiceBus. Just don’t give me that story about having to write a 100 tests first and inherit a dozen interfaces before I can deliver anything of value. Bah! Humbug!

Ok, so enough of the rant.. Got to go and write some custom components now 

Speaking at VBUG (Bracknell)

I will be presenting an introductory session to Biztalk 2006 R2 (and spending some time on the WCF adapters) at the local chapter of VBUG in Bracknell. Of course, we will also spend a little while discussing the impact of Oslo and Dublin and the factors to keep in mind when choosing your integration technologies.

Despite the fact that it has been around for ages and has IMO a lot of visibility, it seems that BizTalk Server still remains a sort of niche product with a lot of developers totally unfamiliar with it and what it could give them. The last time I presented on Biztalk,WCF and WF at the VBUG conference  in 2008, we found that a significant number of the audience didn’t know anything at all about BizTalk and from discussing with contacts in other countries as well, it seems that its a familar situation . Well, anyway, I hope I can do something in my own small way towards rectifying that.

So if you’re in the area and Biztalk is unfamiliar territory why not check it out?

BSF meets Deployment Framework

This post is just to call out some work that Bram Veldhoen has done in linking the Biztalk Software Factory to the venerable Deployment Framework. One thing to note is that the integration was done to the NAnt version of the framework prior to the latest MSBuild version of the framework (5.x). I guess linking into the new MSBUILD version is next on the cards and I’ve chatted to Bram about the possibility of doing something jointly (as if i didnt have enough irons in the fire with MockingBird and BizUnitExtensions ). Check it out and post feedback on the BSF codeplex space.

Been there, done that, got the certificate!

Well, I finally overcame my continued procrastination in this area and sat the MCTS: BizTalk exam today. I always knew I was ‘certifiable’ and now I got the paper to prove it   I cleared it with a score of 899 out of 1000. I’m quite chuffed with it. It revealed a couple of areas I need to work on a bit more but it was a good experience overall.

I know a lot of people debate the merit of certifications in general (and there has been lots of debates on this particular one as well), but I’m not going to re-hash them here. My aim in writing this was to give me a spur to read through areas that I havent had the opportunity to work in and also in a depth that may not have been necessary in the past. As Saravana points out , if you seriously study for this, you get a good feel for the breadth and depth of the product.

Studying can never take the place of actual hands-on experience, but with any large framework or product its common to find yourself working in a ‘niche’ area and never having the chance to explore all that it offers. Biztalk being a huge product, it’s quite ‘normal’ to find developers who have never had to use things like TPM/RoleLinks or the BRE and even BAM (actually, i think BAM is criminally underused, but thats a topic for another post). The study process can open up areas for you to look into and improve your productivity which you may not even attempt otherwise.

I have been quite selective in my approach to certifications thus far, the only one I bothered to write was 70-320 (Building Web Services and  Server Components), that too 4 years ago, but at the end of that study process I was totally amazed at the breadth and depth of the .NET stack and was far more confident of my grasp on Web Services, Enterprise Services, Remoting etc than I had been till that point. I think that going forward that (selective approach) will continue. The R2 exam is now available so that’s next on the agenda and working on the WCF exam should be a useful process as well (assuming I ever get off my lazy backside and take them :-) )

I thought that Id’ just point out a few resources that were of great help.

• Saravana Kumar’s awesome 70-235 Exam Preparation Diary.
• Professional Biztalk - the best explanation of BAM, ever . Period.
• The BAM Training Kit - specifically the hands on exercises. The training material itself is pretty complex in some areas and I personally didnt get much out of those explanations, but the practice was good.
• For BRE, I honestly couldn’t find any single resource which allowed me to get to grips with it. There were some nuggets scattered here and there. I ended up writing my own study paper where I combined notes from all the books and whitepapers into something I could revise easily. The highlights of all the resources were the awesome BRE Tips and Tricks – MSDN Webcast – by Jeff Seagard and the MSDN BRE Walkthroughs . The MSDN exercises were exceptionally good. Saravana also points out (in his post) another techNet webcast on the BRE, but i didnt get a chance to watch that.
• Biztalk 2006 Recipes – I used this for a quick final review of BAM and BRE, Orchestrations etc.

So, if you are considering writing the exam, I’d say go for it. Working on the numerous areas involved can only make you a better developer (or at least a more ‘aware’ developer). And when you are done with it, look for areas that you can feed back the learning gained into your daily development tasks.IMO, the real payback is there. Good luck.!!

BizTalk Testing and Mocks

In an earlier article , I had briefly mentioned that some folk had used mocks with Biztalk, notably to test pipeline components. Since I didnt have the bookmarks at hand then I didn’t provide the links, but I have since found the links again so here they are (and I can also now use this as a note to self if I want to refer to them again or expand on any of the material they have written).

• Testing Pipeline Components – The Pipeline Testing Library – Nick Heppleston
• Unit Testing Pipeline components with Mocks - Rido
• BizTalk Testing Series – Testing Pipeline Components – Mike Stephenson
• TDD and BizTalk Orchestration Expression Code – Matt Podwysocki
• Rhino Mocks and custom BizTalk code revisited   – Matt Podwysocki
• Biztalk TDD – Pipeline Components - Arun Gopalam

While the  blog posts pointing to the Pipeline Testing Library are useful, if you want to go straight to the source, check out the WIKI page that Tomas has set up on GitHub. That page has more samples on how to use the API.

I’m going to have a play around with MoQ and pipeline components in the next couple of days . I think MoQ’s approach is a bit more elegant than Rhino (particularly, the absence of record and replay). I’m also going to link into Tomas’s excellent pipeline testing library from BizUnitExtensions. This has been a long overdue item on my roadmap.

UPDATE: Bram Veldhoen has already done some work on linking the Pipeline Testing Library into BizUnit and has very graciously contributed his code to be put into BizUnitExtensions so that will be released soon with Extensions 3.0.

Enjoy the links and if you find others of a similar ilk that are also useful feel free to put them in the feedback section here and I will update the post.

Introducing ‘Mockingbird’

In my previous post, I briefly mentioned a tool that helped with the testing of web-service consumers. I’m happy to announce that this is now publicly available on CodePlex under the moniker “MockingBird’. Here are some details about the tool – where it came from, how it helps and where it is going. (Most of this info is on the CodePlex space, but i’ll reproduce it anyway).

The Scenarios for MockingBird

(1) Imagine you are given the WSDL for a third-party web-service but no functioning system is available yet (it may be a brand new service or perhaps dev/site licenses are being negotiated). You need to get on with development right now. What do you code against? You maybe a ‘TDD & Mock Objects’ savvy dev which will help in many cases, but what if you ‘don’t do’ TDD & Mocks? Or it may be that you are maintaining/enhancing an existing system that wasn’t coded against interfaces etc. Or what if you are a BizTalk developer? You cannot mock/inject dependencies into your orchestrations and other components (well, except for pipeline components, but thats another story)!!

(2) Next, imagine that you are setting up a build server and multiple environments (DEV, TEST, UAT etc). But now the vendor says you can only have one license for their software. Now how do you run DEV, TEST, UAT in parallel with different data sets? Or you may have more than one license, but what if that service has maintenance schedules that clash with your build? Your build server is then completely exposed to something you don’t control.

I’ve expanded on these scenarios in the tool documentation here, but essentially these are the two main scenarios that MockingBird targets.

The Origins of MockingBird

MockingBird started life as MockWeb an internal tool that my former colleagues (Senthil Sai and Will Struthers) and I developed. All credit must go to them as Senthil first came up with the concept and Will then contributed a lot to the code-base.

It started when we got rather fed up with having to set up multiple instances of a third party service and build data-sets just to help with testing our BizTalk orchestrations reliably. It grew rapidly and organically. While we felt that this concept and tool would be useful to the .NET dev community at large (not just BizTalk teams), the structure of the codebase (at the time) would not lend itself to easy extension and needed to be refactored before making it publicly available. Its taken a while , but now I’ve completely rewritten MockWeb and as I have mentioned in the roadmap, I intend this to go beyond HTTP Web Services and hence the new name.

Whats with the name – ‘MockingBird’ ? Sounds daft !!

Ok, so its not a tool that sits there and laughs at you when you try to test !! . Thats not what the name is intended to convey. Wikipedia says that “They (mockingbirds) are best known for the habit of some species mimicking the songs of insect and amphibian sounds as well as other bird songs,[1] often loudly and in rapid succession” .

So the intent is to mimic web-services and I quite like the name ‘MockingBird anyway!!

The current implementation

There are two main elements to the current implementation

• An ASP.NET HttpHandler which is configured to return pre-set responses
• A WinForms GUI to setup the handler and associated configuration for a given WSDL.

The Roadmap

• The first thing I intend to do is revise the implementation to be fully WCF based as i want to be able to intercept calls irrespective of transport protocol. This is 0ne of the reasons i rewrote the code-base and i hope there wont be too much churn as i extend it
• The second thing is to make it a sort of platform to test webservices, generate BizUnit test steps, maybe C# test fixtures, possibly to code-gen Biztalk artifacts from given WSDLs and schemas.
• Thirdly to take it into the realm of mock Biztalk adapters. WCF is key to that. I envisage this as being a pair of custom WCF send and receive adapters that can dynamically be configured.
• Beyond that, lets see .

So, check it out. There’s an alpha release currently available and your feedback will be appreciated. If you have any thoughts on how the mock Biztalk adapters could be done, I will be glad to hear and of course, if you want to join the project, you will be welcome.