To DAL or not to DAL

Do BizTalk consultants need to care about Data Access Layers? Does a BizTalk solution really need a DAL?  These are the questions that I’ve been mulling over in the past few weeks. Let me explain.

There are a couple of places where a BizTalk solution encounters a DAL. The first is where the DAL acts as an integration enabler. Here the endpoint of the LOB application we are integrating with happens to be a database. The second is where the DAL acts as a process enabler. Here the DAL provides the underpinning of the business process (that is, as part of the business process, it is frequently necessary to update a database with the state of the business document being operated on).

In my current gig, we are using both BizTalk and SSIS. SSIS is great for the ETL and various data related actions. BizTalk then takes over and passes the data to an LOB application doing various business processes as part of that communication. The nature of the processes is such that there is a significant DAL. Early on in the project we went through the usual debate on whether a custom DAL was necessary or if we should just use the requisite database adapters. Isnt the database adapter an obvious choice?  Maybe, or maybe not. In an earlier post , i talked about just such a situation a few years ago where we had choose whether to link directly to the DB or wrap the system in a web-service first and as i explained, things didn’t turn out the way they were expected to.

So, what are the considerations?

1. Firstly, (as I explained in the post and the follow up posts) one of the key issues is the level of abstraction you are given. Especially when dealing with the scenario of integration enablers, a database endpoint is very rarely coarse grained enough to support a service oriented approach. Its more likely that you will be provided with CRUD level interfaces. Even if you decide to direct all communication via an orchestration that wraps all this, how does the orchestration actually call the backend system? Go via the adapter or use a DAL?
2. For the scenario of process enablers, abstraction comes into play again. You don’t want to be cluttering up your orchestrations with bits and pieces of database schema related stuff. You could choose to wrap the database calls in a coarser stored proc but this leads to the next key point which is
3. Performance. If you have a number of send ports (for all these stored procs) in the middle of your orchestrations, there is a cost associated with all those persistence points. If your transaction handling requirements permit, you could think about wrapping some of those calls in atomic scopes, but you have to be  very careful with this. If you do encounter an issue and everything gets rolled back, are your processes really designed to start at the right place all over again without compromising data integrity?
4. If your DAL is designed well, your orchestrations will benefit from having to call methods on business level entities and, just from a persistence point consideration, will, in my opinion, be better off.
5. Transaction Bridging : There were a few situations where we had to bridge the transaction across the database and the segment of the business process. Fortunately, the DAL being of extremely high quality (courtesy of an expert colleague) made this very easy to do.

But, having said all this, a DAL doesn’t come free. You have to write code. Sometimes lots of it. The more code you write, the higher the probable bug density. If the functionality can be satisfied with a code-generator then that will reduce the code you have to write, but it DOES NOT reduce the amount of code you have to MAINTAIN. I think many developers forget about this last point. I’m all in favour of code-gen, but don’t forget the maintenance cost.  (Further, if the functionality in the middle of your processes can be satisfied with boiler plate code, perhaps it’s an opportunity to question what it’s doing there in the first place. Can it be pushed to a later stage and componentized? )

I must confess, at one point, when wading through a sea of DAL code early on in the project, I was quite tempted to throw it all away and go for the adapters, but the considerations above outweighed the pain at that point. Now much later, with everything having stabilized, we know just where to go to make any changes and the productivity is quite high.

But I’ve seen cases where BizTalk developers didn’t care about the SQL they wrote and they ended up in a mess with locking and poor performance. And it takes a really good developer to write a first class DAL and having interviewed and worked with a number of devs I can say that its hard to find good skills in this area. Pop quiz: Do you know how to use System.Transactions yet ? 

There is always the option of using something like NHibernate. If you use some coarse grained stored procs and some business entities, you could kill all the “goo” in the middle by letting NH take care of the persistence. That, i wager would reduce the bug count in that area. But watch out for the maintenance times and the bug fixing. When there’s a component in the middle that you don’t know the internals of, it can make life very hard when trying to track down bugs.

That leads me on to the point of making choices based on knowledge and not ignorance. If you want to adopt “persistence ignorance”, don’t do it because you cant write proper DAL code yourself. Do it for the right reasons.

So I hope the points above have given some food for thought. Custom code is not always bad as long as it is approached and implemented correctly. Whether you choose to use a DAL or not, do it with careful thought on issues like the ones above. As always, your feedback is welcome.

Technorati Tags: BizTalk,DAL,NHibernate

WSCF.blue v1 RTW

I’m a little late in announcing this. WSCF.blue is now formally released to web as v1. Although its only been out 3-4 days now we’ve crossed over a 100 downloads already. Whew. Looks like a lot of folk were waiting for this release

Christian Weyer has blogged about the release. I would also like to call out Alex for taking so much initiative and coding till 3am on some days (and emailing the group at that time too !!). We definitely owe this release to him.

Now on to v2. My next post will be about the roadmap and our shopping list of features for the next few releases. Check out the final v1 release. Hope you like it. As always, your feedback and suggestions would be most welcome and appreciated.

WSCF.blue – Paste-As-Schema

WSCF.blue now has a new feature called “Paste-As-Schema” (which takes inspiration from the REST starter kit’s Paste-As-Type). Alex explains the new feature in detail in his post.

The MSI has now been updated to version 1.0.4. It is still formally in Beta-2. It also includes a few bug fixes in the original Beta-2 (1.0.3) release. Please check the CodePlex workspace for details. Hope you find the new feature useful. If you run into any issues do let us know.

WSCF.blue Beta-2

Beta-2 of WSCF.blue is now available.

The big feature of this new beta is the standalone Data Contract Generation and we have also put in some other fixes and addressed some outstanding issues (as well as rolling up fixes since the last beta).

The feature list is as follows. The “More Information” links point to posts on Alex’s blog with screenshots and lots of details.

• The Generate Data Contract Code feature is now available and supports the selection of multiple XSD/WSDL source files. (More Information)
• When the Enable WSDL Endpoint option is chosen a reference to the Thinktecture.ServiceModel.Extensions.Metadata.dll assembly is automatically added.
• Support for VB.NET has been added to the code generation.
• You can choose if operation methods on the service class will throw a NotImplementedException, call an implementation method in a partial class, or will be defined as abstract methods. (More Information)
• Force the SOAP actions (Action and ReplyAction) applied to each operation contract follow the standard WCF format: <namespace>/<service>/<operation>[Response]

Hope you find this release useful. As always , keep the feedback and suggestions rolling in. Enjoy 

My first year at MCS

Last Tuesday (the 18th Aug) marked my first year at Microsoft Consulting Services (UK). I can’t believe how fast time has gone by. Years ago, I used to wonder how it would be to work in MCS and dismissed the idea as too high to aspire to and yet here I am after a full year at the very same place. So what’s life been like working for Microsoft? Here’s what I’ve seen. [Remember, this is just the view through my lenses , nothing indicates company policies etc ]

My first 3 months were rather quiet, as they are considered a “ramp-up” period and usually one is expected to network and round-off your skills. However, I was warned that the 3 months would fly and then it would be liked drinking from a fire-hose, and the warning was absolutely spot on. It’s been a rollercoaster from last November.

The first thing that struck me (not surprisingly though) was the calibre quotient of my colleagues. While I have, in the past, worked with some top notch developers and architects, it just seems that the “average” knowledge is a mile higher than at other places. For instance, I could think I know all about SQL and then find myself sitting next to a chap who did a stint on the SQL product group and knows more about the core engine than anyone else in the world (now, this part is anecdotal, i didn’t actually encounter that myself, but heard of someone who did, but you get the picture  . Darren Jefford is the architect on my current gig. A couple of years ago, when just reading through Darren’s blog I couldn’t have imagined working on the same team as Darren. C’est la vie!  (I couldn’t resist that bit of name dropping).

The second thing that took me by surprise was the how committed Microsoft is to partners both at the platform / product level as well as in Services. With our products, we pride ourselves on building the best platform and tool support possible and empowering partners and vendors to build on top of that. Sure, MS has a record of being somewhat ‘predatorial’ in the past, but it looks and feels different in that respect now.  In the Services side, it’s common to find a number of partner companies working alongside us to deliver the projects and there too, it’s common to find that many of the developers and architects from partners  are leading bloggers. MVPs and generally, well known in their respective dev communities. [Ok, one more name-drop.. did you know i’m actually working with the legendary BizTalk “Arch Hacker” ? It’s true he wears the mask of Zorro to work !! …(ha ha just kidding….)]

Then there’s the veritable flood of information and access to stuff ages before it becomes public!  Of course, while a lot of that is non-disclosable to external audiences, within the company, information does flow quite freely (which I heard, from someone who joined us from a competitor, is not always the case in large software houses). Unfortunately, there’s seldom the time to actually use all that and in this respect, you’re on your own. There is a lot of structured learning on offer, but balancing that with project commitments is a fine art.

Ok, so that’s all about the company. So, what’s the personal impact been?

Firstly, its the amount I’ve learned technically (which is, of course, to be expected in a place like this). My knowledge of BizTalk, especially, has been deepened considerably both by the requirements of the current gig (which is in its 6th month now) as well as just absorbing tons of stuff from my colleagues.

Secondly, it’s been great to see, first-hand, very large scale projects delivered successfully and on time. I have been involved in some quite big (and successful) projects through the years, but the sheer scale of the projects here is far beyond what I’ve worked with in the past). [On the technical side of this, its been eye opening to see how useful TFS is for project tracking and reporting].

Thirdly, it’s also made me raise the bar for my own community contributions. I take longer to make something public than I would have done in the past.  Although my community contributions are personal and not Microsoft IP,there is the implicit association with MS and i find myself thinking “is this good enough to be associated with an MCS guy!” . Take MockingBird, for instance. This ‘implicit’ association ensured it was better at first release than it would have been, say, 2 years ago. There’s both the knowledge gained over the years as well as the “this has to be good enough to be published by a Microsoft chap” that helped it. Not that I’ve reached some sort of ‘coding guru’ status (or ever will), and  there’s always more to be learnt and improvements to be made in all areas but I’m definitely more satisfied with the quality of my output now.

So, it’s been a good year. I’m looking forward to the rest of year-2. One year at a time!

WSCF.blue – Updates

So there’s been a few things happening on the WSCF.blue front. We’ve had some excellent feedback and also some feature suggestions, so all that has been rolled up into the install package. It’s still Beta-1, but the MSI has been updated to v1.0.2.

The main changes are

(1) the option to format the soapActions when setting the Action and ReplyAction so that it conforms to W3C standards. We had some issues with certain WSDLs where we were getting the replyAction set to “*” (and even svcutil was doing that) which affected the communication.

(2) Options to control the way the service class is implemented. Hitherto we have had the default behavior of the method throwing a “NotImplementedException” (which is a normal thing for a code-generator). But this can be limiting sometimes. So we now have options to either define the class as abstract or partial. Alex explains all this in his post.

Hope you find the new features useful.

MockingBird@SBUG

The UK SOA BPM User Group (SBUG) is holding a series of mini-meetings over the next few months. These are short live meetings / webcasts. I’m delighted to be presenting my web-service mocking tool , MockingBird at the next mini-meeting to be held on the 29th July.

I will be discussing how MockingBird assists and simplifies integration testing with web services, how it is used in Biztalk and non Biztalk scenarios, how the system works, how it can be extended and the roadmap.

Please check the event link on the SBUG site for details.  All are welcome to join (no membership needed). Hope to see a good turnout

BizUnit 3.1

BizUnit 3.1 is now released. (Actually its been out for over a week). There are a few bug fixes in this release and most notable is the addition of the following steps that Bram Veldhoen contributed.

• BizTalkSteps/ExecuteMapStep
• BizTalkSteps/ExecuteReceivePipelineStep
• BizTalkSteps/ExecuteSendPipelineStep

The Pipeline steps use Tomas Restrepo’s Pipeline Testing Library.

I had originally planned on incorporating Bram’s steps into Extensions as I wrote previously.  However after discussing with Kevin, those plans changed so its now in the core BizUnit.

There is a similar MapExecutingStep in Extensions that has been around for a while now and when I get some time I’ll take a closer look at Bram’s step and see if there are any differences that are worth keeping or retire the Extensions step if not required anymore. I have had a couple of bugs reported on Extensions beta-1 which I shall fix soon and get the RTM released which will be compiled against BizUnit 3.1.

WSCF blue: Beta-1

WSCF-blue, the WCF successor to the popular WSCF (Web Services Contract First) tool is now available in its first beta release at CodePlex. I’m delighted to be a part of the team working alongside guys like Christian Weyer, Buddhike De Silva, Edward Bakker and Alex Meyer-Gleaves.

If you aren’t familar with the tool, basically WSCF provides tooling to approach web service development in a contract first manner, specifically WSDL/XSD first rather than code-first. WSDL is often the favored ‘neutral’ ground for specifications to be agreed on in large projects where interoperability is key between heterogenous technologies. It is also common to find that the WSDL for web services is the first thing to be discussed between an application development team and a service provider (in any development / integration project). So , having a tool that allows you to work with WSDL and XSD in a consistent, reliable manner without getting bogged down in the intricacies of WSDL and associated specifications is a must. WSCF aims to address the lack of tooling in this space (especially in the .NET world).

WSCF is a Visual Studio 2008 add-in that provides the following features

• A WSDL Wizard that allows the developer to step through the creation of a WSDL from one or more XSDs
• A DataContract Generator (similar to XSD.exe, XSDObjectGen.exe and Svcutil.exe) that generates the . NET equivalent of the XSD types
• A Service/endpoint stub (SVC) generator and
• A client proxy generator.

In the current beta release, the stand-alone Data Contract generation option is NOT available, but that is something we will be addressing as a priority and should be available in the RTM release. However,  data contracts are generated as part of the service side stubs and client side stubs.

You can either install the MSI which makes the add-in available for an open solution (as the context menu for an XSD or WSDL) or if you are using the source version, you can debug the add-in directly using F5 which launches VS in the experimental hive as is normal for all add-ins.

I have been an regular user of WSCF since its ASMX days and found the quality of the code generated to be very high. We do hope you find that this tool goes some way toward addressing your needs with WCF development and are all keen to get feedback, bug reports and feature suggestions. With the number of project templates that are available for WCF (Web Site, Web Site -Remote IIS, WCF Class Library, WCF Service Application) and the rules that some of them have around structure, it has been quite a bit of work to get the features such as SVC generation sorted and I’m pretty sure that there will be other scenarios you are involved with where the tool needs improvement or enhancements.

Here are some other posts on the beta release by the other members of the team

• Next generation of a success story: WSCF.blue for WCF goes Beta 1
• WSCF.blue Beta 1 is out!
• WSCF blue – Are you doing contract first?

Keep an eye on the CodePlex space for information relating to the roadmap etc. Do check it out and let us know what you think of it.

Blueprints – Down but not out

Some of you may have noticed that the Microsoft Blueprints project has gone quiet and the site taken down from CodePlex and MSDN. I had written a couple of posts about Blueprints in the past and how it was eventually going to take over from GAT/GAX.

Anyway, if you are wondering what’s happening, the answer is that we are making some changes around Blueprints. The Blueprints project was an important incubation whose purpose was to explore process guidance and automation. They helped us (that is, the product teams involved in developing this) gain a better understanding of the requirements for this kind of technology through customer and partner feedback. The reason we took down the external projects was to focus the incubation internally and we are looking at taking this forward so we can support process guidance and automation in a manner that is aligned with VS Team System product direction.

As users of VSTS know, the suite is not based around any particular methodology and can support a range of project methodologies ranging from very lightweight to very formal and process driven. The tooling supports this range of methodologies. Software Factories are an important development discipline which we support and as Jezz Santos  and Edward Bakker  and other thought leaders have written, it is possible to approach factory development itself in agile or very formal ways and its quite often found that a rapid iterative approach works very well with developing factories That said, VS should and will support factory development going forward in a manner thats agnostic of methodology. Exactly how this manifests itself in the product suite remains to be seen but having looked at the features that are available already in Dev10 Beta-1 in the VS Team Architect edition, we can be sure that it will be of high quality.

GAT/GAX will be available in VS10 and the DSL Toolkit has been improved quite a bit and aligns well with the new “Extensions” model so if you havent checked out the latest developnments in the DSL Toolkit space, I would encourage you to take a look. Stuart Kent has a nice video on the new deployment method for DSL Toolkit based packages.

I will post more on this topic when there is more information that I can share and especially when there are bits available to play with. Watch this space .