To DAL or not to DAL

Do BizTalk consultants need to care about Data Access Layers? Does a BizTalk solution really need a DAL?  These are the questions that I’ve been mulling over in the past few weeks. Let me explain.

There are a couple of places where a BizTalk solution encounters a DAL. The first is where the DAL acts as an integration enabler. Here the endpoint of the LOB application we are integrating with happens to be a database. The second is where the DAL acts as a process enabler. Here the DAL provides the underpinning of the business process (that is, as part of the business process, it is frequently necessary to update a database with the state of the business document being operated on).

In my current gig, we are using both BizTalk and SSIS. SSIS is great for the ETL and various data related actions. BizTalk then takes over and passes the data to an LOB application doing various business processes as part of that communication. The nature of the processes is such that there is a significant DAL. Early on in the project we went through the usual debate on whether a custom DAL was necessary or if we should just use the requisite database adapters. Isnt the database adapter an obvious choice?  Maybe, or maybe not. In an earlier post , i talked about just such a situation a few years ago where we had choose whether to link directly to the DB or wrap the system in a web-service first and as i explained, things didn’t turn out the way they were expected to.

So, what are the considerations?

1. Firstly, (as I explained in the post and the follow up posts) one of the key issues is the level of abstraction you are given. Especially when dealing with the scenario of integration enablers, a database endpoint is very rarely coarse grained enough to support a service oriented approach. Its more likely that you will be provided with CRUD level interfaces. Even if you decide to direct all communication via an orchestration that wraps all this, how does the orchestration actually call the backend system? Go via the adapter or use a DAL?
2. For the scenario of process enablers, abstraction comes into play again. You don’t want to be cluttering up your orchestrations with bits and pieces of database schema related stuff. You could choose to wrap the database calls in a coarser stored proc but this leads to the next key point which is
3. Performance. If you have a number of send ports (for all these stored procs) in the middle of your orchestrations, there is a cost associated with all those persistence points. If your transaction handling requirements permit, you could think about wrapping some of those calls in atomic scopes, but you have to be  very careful with this. If you do encounter an issue and everything gets rolled back, are your processes really designed to start at the right place all over again without compromising data integrity?
4. If your DAL is designed well, your orchestrations will benefit from having to call methods on business level entities and, just from a persistence point consideration, will, in my opinion, be better off.
5. Transaction Bridging : There were a few situations where we had to bridge the transaction across the database and the segment of the business process. Fortunately, the DAL being of extremely high quality (courtesy of an expert colleague) made this very easy to do.

But, having said all this, a DAL doesn’t come free. You have to write code. Sometimes lots of it. The more code you write, the higher the probable bug density. If the functionality can be satisfied with a code-generator then that will reduce the code you have to write, but it DOES NOT reduce the amount of code you have to MAINTAIN. I think many developers forget about this last point. I’m all in favour of code-gen, but don’t forget the maintenance cost.  (Further, if the functionality in the middle of your processes can be satisfied with boiler plate code, perhaps it’s an opportunity to question what it’s doing there in the first place. Can it be pushed to a later stage and componentized? )

I must confess, at one point, when wading through a sea of DAL code early on in the project, I was quite tempted to throw it all away and go for the adapters, but the considerations above outweighed the pain at that point. Now much later, with everything having stabilized, we know just where to go to make any changes and the productivity is quite high.

But I’ve seen cases where BizTalk developers didn’t care about the SQL they wrote and they ended up in a mess with locking and poor performance. And it takes a really good developer to write a first class DAL and having interviewed and worked with a number of devs I can say that its hard to find good skills in this area. Pop quiz: Do you know how to use System.Transactions yet ? 

There is always the option of using something like NHibernate. If you use some coarse grained stored procs and some business entities, you could kill all the “goo” in the middle by letting NH take care of the persistence. That, i wager would reduce the bug count in that area. But watch out for the maintenance times and the bug fixing. When there’s a component in the middle that you don’t know the internals of, it can make life very hard when trying to track down bugs.

That leads me on to the point of making choices based on knowledge and not ignorance. If you want to adopt “persistence ignorance”, don’t do it because you cant write proper DAL code yourself. Do it for the right reasons.

So I hope the points above have given some food for thought. Custom code is not always bad as long as it is approached and implemented correctly. Whether you choose to use a DAL or not, do it with careful thought on issues like the ones above. As always, your feedback is welcome.

Technorati Tags: BizTalk,DAL,NHibernate

Blueprints – Down but not out

Some of you may have noticed that the Microsoft Blueprints project has gone quiet and the site taken down from CodePlex and MSDN. I had written a couple of posts about Blueprints in the past and how it was eventually going to take over from GAT/GAX.

Anyway, if you are wondering what’s happening, the answer is that we are making some changes around Blueprints. The Blueprints project was an important incubation whose purpose was to explore process guidance and automation. They helped us (that is, the product teams involved in developing this) gain a better understanding of the requirements for this kind of technology through customer and partner feedback. The reason we took down the external projects was to focus the incubation internally and we are looking at taking this forward so we can support process guidance and automation in a manner that is aligned with VS Team System product direction.

As users of VSTS know, the suite is not based around any particular methodology and can support a range of project methodologies ranging from very lightweight to very formal and process driven. The tooling supports this range of methodologies. Software Factories are an important development discipline which we support and as Jezz Santos  and Edward Bakker  and other thought leaders have written, it is possible to approach factory development itself in agile or very formal ways and its quite often found that a rapid iterative approach works very well with developing factories That said, VS should and will support factory development going forward in a manner thats agnostic of methodology. Exactly how this manifests itself in the product suite remains to be seen but having looked at the features that are available already in Dev10 Beta-1 in the VS Team Architect edition, we can be sure that it will be of high quality.

GAT/GAX will be available in VS10 and the DSL Toolkit has been improved quite a bit and aligns well with the new “Extensions” model so if you havent checked out the latest developnments in the DSL Toolkit space, I would encourage you to take a look. Stuart Kent has a nice video on the new deployment method for DSL Toolkit based packages.

I will post more on this topic when there is more information that I can share and especially when there are bits available to play with. Watch this space .

What’s new in WCF 4?

Christian Weyer has been writing a very interesting series of posts on some of the new features of WCF 4.  There is still a long way to go for the release, so as with all products / technology stacks, it is wise to anticipate change. However, there is some really good stuff here and if all of this makes it into RTM in this form (or a better one), it should help quite a bit in doing advanced stuff with WCF.

Here is a list of the posts. Also a note on the disclaimer from Christian’s posts : all the information was gathered based on a close-to-Beta 1 build of .NET Framework 4.0 and Visual Studio 2010. Details may vary and change

•  Simplified configuration – or: “Look ma: my config shrinks!”
•  .svc-less Activation – or: “Look ma: my [REST] URLs look good!”
•  Dynamic service and endpoint discovery – or: “Look ma: I just need the contract to talk to my service!”
•  Standard endpoints – or: “Look ma: streamlined infrastructure and system endpoints!”
• Discovery announcements – or: “Look ma: I can see when my service goes online or offline!”
•  Routing Service – or: “Look ma: Just one service to talk to!”
• Protocol bridging & fault tolerance with the Routing Service – or: “Look ma: Really just one service to talk to!”

Happy reading !!

Look Ma ! An evil middleware product!

I couldn’t resist commenting on this issue. I was just doing some final prep for my VBUG talk tomorrow and came across Richard Hallgren’s oddly titled post – Does BizTalk have man-boobs?. Richard writes about a QCon webcast of a session done by Martin Fowler and Jim Webber and writes

“Their main point is that we use much to bloated middleware (BizTalk is mentioned as an example here) and that we should have a more agile approach to implement SOA and ESBs. They’ve used all the agile ideas (testing, isolated testable functionality, deliver small and often, continuous builds etc, etc) and applied them to integration..”.

Richard goes on to make some good points which I totally agree with. Check out the post for details. I commented on the post and decided I would make those points again here and add a couple more, which is also quite good timing (for me) considering I’m speaking on BizTalk tomorrow.

So, I agree that BizTalk is totally unsuitable for small situations and if it is naively (read – most often) used without any performance tuning whatsoever, then its latency can be bad. But if its tuned, it can totally rock (and I know this from MCS colleagues who’ve experienced it in very large BizTalk projects).

BizTalk is big. Yes, totally. But it has to be because it addresses a vast set of use cases. Same goes for its competitors like TIBCO, webMethods etc. If you dont want all the features, dont use them. They are in the box anyway and they wont slow you down if you dont use them. You want content based routing – its in the box; you want long running business processes – check, you want ‘aspect oriented’ interception of messages for tracking , you’ve got BAM. You want monitoring tools – check. You don’t want tracking – turn it off. You dont want business rules – ignore the BRE. The list goes on.

There’s also a case to be made for using something like the ESB toolkit to give you a decent jumpstart on your routing infrastructure.

If you only have a couple of systems to integrate (and you are very sure there wont be more) then go ahead and custom code it. Its not worth buying a product for that.

The problem I have is with some “agilists” who, it would seem, want to custom code every darn thing under the sun. As I commented on Richard’s post, the irony is that the very same folk will then stress the importance of having good, robust pre-built frameworks and good tools to help with the “agile” approach. But wait, those tools have to be on the “acceptable” short list. Take NHibernate for example. An excellent tool, no doubt. And for many folk, since its open source, so it must be sent from above, totally divine, but not BizTalk. That’s way too big. And besides, its from Microsoft. Gasp! horror! It must be evil !!

Some take the view that its about having confidence in your code. I can understand that. Having been exposed to a fair amount of TDD etc, I can attest to the feeling of security when your edge cases have all been tested and you see all those “green signals”. But in an integration scenario where use cases are similar , how many times will you write a test, write the code, refactor, refactor, blah.. till the code comes out of your ears. Pretty soon customers are going to wonder how many times they have to pay for something thats already been written.

If Ican have confidence in code I write, I can have the same amount and more in a product thats been tested out in hundreds of more scenarios than I could imagine. Do these products (such as Biztalk and its competitors – I’m not selling anything here ) have bugs? Well of course they do. But in many cases, you can be just as confident in commercial closed source as in open source. Besides, if pre-built tools werent a good idea, there wouldnt be any open or closed source tools would there?

I’m not partial. Go in for the Neuron ESB if you a want a pure WCF ESB, or go in for something thats gaining a good reputation like NServiceBus. Just don’t give me that story about having to write a 100 tests first and inherit a dozen interfaces before I can deliver anything of value. Bah! Humbug!

Ok, so enough of the rant.. Got to go and write some custom components now 

GAT is now Blueprints

The old GAT is dead! Long live the new GAT aka “Blueprints“. This news caught me by surprise. I thought I was fully abreast of the happenings in this area, but was caught napping. [Ok, so I'm being a bit dramatic here. GAT isnt actually dead but has evolved into a new offering thats IMO much better. As far as the old GAT (runtime, support, tools etc) is concerned , I've put in a couple of points at the bottom of the post, but make sure you check the official support information before making any long term decisions on all this].

So, what’s all this Blueprint stuff about?. Well, essentially GAT/GAX and the first set of factories (Service Factory, WCSF etc) all validated the concept of factories and established how tooling could work and where it fell short. VS Extensibility has also improved a lot since these factories first came out. So its now time to take the concept of Factories up one level into Software Product Lines etc which is closer to the original vision of factories. Along with the P&P factories, there was a separate project called Glidepath for ISV’s which provided lightweight tooling and was instrumental in delivering the S+S Blueprints. It made sense to align the two areas together and thus the new project.

An excellent video which introduces the Blueprints concept including demos of currently available blueprints can be found on Channel 9 titled “Beyond VS Packages – Adding Value with Blueprints” and is presented by Jack Greenfield and Michael Lehman. Some of the things that I find really cool and worth calling out here are

• Dynamic Versioning
• Workflow Based Process
• Workflow Based Commands
• Design & Runtime Composition
• Designer Integration
• TFS Integration

I’m not going to talk about them in depth as Jack and Michael explain them  very well in the C9 video.

The big problems I found with GAT were (a) the authoring experience and (b) the appalling lack of good documentation. In terms of the documentation, the CHM files provided with the install gave a quick overview of recipes and actions and then dived straight into class library level. Having info on class libraries in the tool is ok when you have grasped the basics and are productive to some extent and then need to look deeper. There were a few blog posts by folk who had played with it, but no official stuff. You had to learn by poking into existing factories like Service Factory etc.

In terms of the authoring experience, GAT is too low level. Hand editing XML files doesn’t make for a great authoring experience. Clarius Consulting played a big role in helping matters with their Software Factories Toolkit (SFT) and in fact that toolkit was even used in the Service Factory and also in the BizTalk Solution Factory.  The SFT introduced the concept of a visual designer for recipes and actions. The P&P team did a nice thing in releasing the Guidance Extensions Library (GEL) where you could cherry pick the recipes and actions they used for various factories and put them into your own. But IMO, the thing that was missing was an overall design tool, which, for instance, could allow you to put all the items from the GEL and your own into a sort of toolbox and then drop them on to a design surface to build your package.

Now I was pleasantly surprised to discover that the Blueprints does feature this sort of design tooling, specifically using WF to achieve this (essentially GAX Actions become WF Activities). WF is a perfect fit for this sort of thing so I’m glad we’re getting this kind of authoring support. WF is not only for the package building but the whole guidance process is represented as a workflow. With the WF based process, the guidance can even take on a “parallel” nature. This will  be useful for some packages where after a certain number of steps which lay the foundation, you can do any other actions that dont have to follow a sequence or which arent dependent on each other.

Now, one of the limitations with GAT was that if a menu item  (corresponding to an action) was missing or disabled, you were left wondering if there was a problem with the package. With Blueprints however, we can see what actions are available or not depending on pre-conditions and post-conditions so there is no guesswork involved in knowing what comes next. Blueprint authors can use pre and post conditions to examine the solution or check an external system (say a live web service) and do various other things as part of the flow. This makes ‘composition’ of actions much easier because an action can come with a little ‘pack’ of conditions to help the author decide where it slots in the oiverall flow and what guidance or extra steps may be required between one action/step and another.

The story now gets even better with Design & Runtime composition which is explained in the video.

Going back to GAT/GAX, two key things are

(a) there will be a ‘rev’ of GAT/GAX for VS10, but no new functionality and no changes to the current set of factories and

(b) There will be a migration blueprint which will allow authors of current factories to move their code-base into the new system and the GAX runtime will still be available so actions will continue to work. I imagine this migration blueprint will be intensely dog-fooded as the P&P factories would need to move to this platform and they have made a large investment in GAT.

Another thing which is worth pointing out is that we’ve still got a long way to go to Dev10 etc and although we are in the Blueprints 2.0 era already, in terms of the GAT ‘integration’ I think there’s a long way to go. GAT may have authoring issues but its pretty rich functionally and once you really get into it you can be quite productive. I’ve started to poke into Blueprints now and trying to ascertain whether it really is ready to take over from GAT. At this point, I’m not totally sure it is, but I could be wrong. Let’s see. GAT isn’t really dead yet, but if you are just starting to look at factories, its best to understand where things are headed.

Looks like the world of VS10 and associated products/offerings is going to be very exciting and productive indeed.

Service Oriented Modelling with DSL Tools

I was sent a link to a mind blowing Channel 9 video on Service Oriented Modelling presented by Blair Shaw and Hatay Tuna. IMO, thats the most inspiring piece of kit i’ve seen so far. The work Hatay has put into it is fab. If you havent seen it , then do yourself a favor and take a look.

It demonstrates what can be done with the DSL tools and more importantly shows the kind of tooling that can be produced to support a proper SOA where services actually represent business capabilities rather than random little components dressed up with WSDL and pretty diagrams.

Key thing to remember though is that this is not a free tool. Its used by Microsoft Consulting Services as part of customer engagements. But even watching the video is highly educative on the steps to go from requirements and capabilities to service and technology models through to implementation and also highlights how it is possible to code-gen even WF activities to match the business process steps.

This is the kind of tool I dreamt of building in the past. (Yeah, i dream a lot  . But seriously, while VS2005 had some of the basic foundations, we  didnt have the depth of tooling to make something like this possible and now we do. Of course MCS had the EFx software factory and eventually the DSL Tools matured to the extent where the Service Factory modeling edition could be delivered on it but this kit goes way beyond all that. The platfom its implemented on is VS2008/.NET 3.5, so I can well imagine what it could become on the VS2010/NET 4 platform.

Its only 30 mins or so and the quality of the video is very high (although the download is a bit heavy due to the quality).  Check it out. And while you’re there on Channel 9, take a look at the videos for VS 2010. There’s some really cool stuff there too.. Enjoy…

Is Oslo just like Access?

eWeek publlshed a couple of articles on Oslo very recently. They are

• The Origins of Microsoft’s Oslo Software Modeling Platform
• Microsoft’s Distributed Destination: Oslo

I found the background material on how it all started to be quite interesting and also to hear that its been in the works for more than a decade. At least, thats how the writer puts it. I think they mean that attempts at incorporating modeling into the implementation of enterprise applications have been in incubation for that long.  However the point that really disturbed me was on the description of the tool. To quote the article , (emphasis mine)

 

“Lovering said the Oslo tool is novel with respect to development tools in that it will feel familiar to the masses. “If you’re [a Microsoft] Access user, it will be more familiar to you, let me put it that way,” he said. Indeed, said Lovering, the tool is basically an interactive database development tool. “So, if you kind of think of Access, [Microsoft] Excel, …” that is an approximation of the tool.Lovering said.”

 

 

I have heard a lot about the platform,most of which I am not allowed to talk about obviously, but even with the amount of public information available i thought it was pretty obvious that

• (a) it isnt a tool but a code name for a platform and
• (b) its most certainly not a ‘database’ development tool. There is a database component but its a repository.

I think that it must be a misquote. The article does go on to quote Lovering saying

“However, “you have to be a little bit careful with that comparison because it could be misleading. I’m trying to give you sort of a general feeling of the center; it is not [Access and Excel], but those are the best approximations I have if you haven’t experienced the tool.”

Hmmmm…Was it some sort of UI for the repository thats being referred to it? I dont know …Maybe he’s trying to say that like Access lowered the bar for database development, this will lower the bar to creating enterprise applications which sounds better but calling it a database tool is odd. I certainly hope it isnt the revenge of Microsoft Access

 

 

 

 

 

Mock HttpContext and Session State

One of the most useful things I came across when doing the asp.net project that i wrote about in my last 2 posts was this article and sample solution by Jason Bock on mocking the HttpContext session state in .net 1.1 and 2.0. Actually this is not a ‘mock’ interface in the sense of mock objects but a real HttpContext which is artificially built by calling into some API’s and which can be used in non web scenarios. I’ll explain why I needed this and how I used it below.

I had to make use of caching in the solution and started of using the HttpRuntime Cache which can also be used from non web applications and is very useful when driving your app through NUnit. Of course, with my new found love of interfaces and mock objects i wrote up an ICacheHandler interface and implemented a WebCacheHandler that wraps the System.Web.Cache. I did consider using EntLib but i dont really need any offline caching etc. System.Web is more than enough. This cache is a dependency for the Service Agent so i stuck my DI hat on and injected it into the service agent through the constructor.

Now heres a gotcha. If you simply use the System.Web.Cache object and try to call the Add() method, then when there is no web context  (say when you are unit testing), the Cache will be null and you will get an “object reference not set to instance of an object” errors. So you need to use HttpRuntime.Cache.Add() .

We then decided that for certain situations the application level caching could actually prove counterproductive and what we really needed was Session level caching. There isnt much data to store anyway so theres no chance of knocking down the server with all the session level caching. Since I had my nice lCacheHandler, i wrote up another SessionCacheHandler to use session state. This is where the wheels fell off for a couple of hours. Session state is not like the HttpRuntime Cache. It cannot exist without a valid HttpContext (which is rather obvious and sensible and all that but not helpful in that situation) so the system which was working with the WebCacheHandler when called through NUnit now simply barfed at all calls to add data to the cache.

I knew i had to fool the system into thinking there was a valid HttpContext so i could test the code properly offline without needing to launch the web application so after trawling through the web i found the article by Jason Bock (linked above). The way i used this tool was to set a dependency on HttpContext in my SessionCacheHandler and allow a HttpContext to be injected via property injection. A CacheHandlerFactory class then does the job of setting up the session handler and injecting either this MockHttpContext or the real HttpContext.Current and returning the sessionCacheHandler for use by any caller. This works fine now and i can peacefully run the tests without needing a full web context. 

Thinking about EntLib again, i guess a more EntLib oriented approach would be to make this custom component into a sort of ‘SessionCache’ provider for the Caching Application Block and let entlib read a config setting. That way it can also be used through PIAB. Gotta start looking into that option now at least for self education.

It should be interesting to see whats coming up in ASP.NET 3.5 . Apparently there are lots of interfaces for these static things like HttpContext etc which will allow them to be mocked or substituted at runtime/testing so maybe this kind of thing will get easier as time goes on. Meanwhile, i’ve got a nice addition to my toolset. I’m beginning to really like dependency injection. Now for the next step – DI Containers!!

[UPDATE- 18 Feb 2009]: Just to point out that there are limitations in Jason’s utility. It is good for session usage but I ran into problems when I wanted the HttpContext to have stuff like Physical Application Path and things like Headers in the context which cant be done through Jasons code. So the tool that i then turned to was Phil Haack’s HttpSimulator. This was incredibly useful when developing MockingBird.

SOA-2(b): More on the integration layer

I had discussed the design of our integration layer in an earlier post and Stuart commented asking if we implemented the messaging layer using messaging only or if we needed to implement orchestrations. As i started writing the response, it got rather long so i decided to make it a full fledged post. So here goes.

The way we designed the orchestrations was to hold composite large messages instead of multiple smaller messages and each orchestration would communicate with one or more webservices in the backend. So if we decided to, say, create a new ‘Business Document Application’, we would pass it customer information, department info etc and the orchestration would check if the customer existed using a lowlevel backend webmethod, create/update the customer, then invoke another interface to setup the document and so on. Some other orchestrations encompassed a set of ‘actions’ and data specific to them, such as ‘renew’, ‘invalidate’ etc in a single message and then called different backend services depending on the action.

Similarly search handlers acted as aggregators. Initially the searches were pass through orchestrations (and all the data was available from single webmethods in the backend) but as time has gone by some have been augmented with data from two or more sources. 

If you dont have any composition/aggregation, then schema webservices would work just as well and perhaps we could have implemented the searches in this way – but i had some ‘preview’ of future requirements and decided to put in orchs early. With Schema webservices, I have been concerned about business level validation and returning SOAP faults etc from Schema ASMX’s and havent done a lot of probing in that area so i have been rather reticent to go with those alone. 

It also depends a lot on the backend interface. For us we could use a solicit response port with a custom pipeline to set the soap operations/actions. In other implementations, for one of our backend systems we have had to call an external assembly (provided by the vendor to wrap their webservice operations) and this needs an orchestration (unless, maybe, we go down a custom adapter route and then a port would do just as well).

Schema webservices are also a good way to bring in a slim mediation layer initially and then beef it up with some orchestrations wherever necessary (if you can get the error handling sorted).

With all these of course, you must keep performance in mind and tune the receive and send portions of your solution accordingly. Some of the perf related material on MSDN and in the new deployment guide talk about settings which can help you tune the solution. We’re going to be taking  a fresh look at the system and putting into practice some of the stuff we’ve been learning in this area so i’ll post more info as we progress.

SOA-2: Debating the nature of the Integration Layer

In a previous post on project goings on I talked about our choice of adapters and the lessons we learned. One thing we are never short on is stress, excitement and debate (mostly the last one). .

Before you read any further, let me warn you this is going to be a long one. I’ve been chewing over it for a couple of months now. Ok, back to business.

Once that project got over we had to start on the next assignment. The requirement was like this. There is an existing backend system, call it system A (which we integrated with in the previous project) which was used by a particular department and the business wanted to build another system based on some processes which had hitherto been paper based. Let’s call that System B.

The way they wanted it work was that System A and B would actually be subsystems of one big application and that they would share some core database entities such as person & address. Now we have a portal, which is a sort of mash-up presenting a view of the customer across multiple systems including A and B. (You could argue that this not really a mash-up because it processes transactions across systems rather than just presenting an aggregated view). System B did not need its own UI because it was to be presented via the portal so all we needed was some web services.

We take our lessons seriously and once we had decided that we would avoid the database adapter in the service layer, a more profound argument arose as to the value and maybe more importantly the location of the service layer.

In a general integration situation where you have existing systems and you are placing a hub into the mix, the design is fairly straightforward. The hub obviously goes in the middle. But when you have a total green field what do you do?

The key thing that got people tied in knots for a while is that if the portal is to call web services and the backend system exposes web services then why put a web service layer (backed by orchestrations) in the middle?  If the backend system had a different granularity from what we needed, then the hub would provide the mapping and a reason for its existence, but we had full freedom to design the backend service with coarse enough granularity from the beginning. Yes, we had to send some notifications about data changes in system B to other applications as well, but why do that from a central mediation layer? Why not just emit notifications from system B to the hub and have it then publish them to other systems?

The options are shown in the block diagram below.

 

From the portal point of view, its just connecting to a web service and it doesn’t particularly care if Biztalk is behind the web service or some vanilla .NET components but when you are designing those services, what’s behind the service is very important.

So, what are the considerations in this situation? Here are some of my thoughts on the items that influenced our decision.

Granularity: The way I normally approach things is to is to categorise the backend service as an Application Service which can legitimately expose CRUD operations while the main Business Service that would span system A and B and others would be designed with more coarse grained functionality (for instance, ProcessCustomerNotification, which could be mapped on to one or more Application Service calls such as SearchCustomer & Insert/UpdateCustomer. Of course, in this particular project, as I mentioned above, the granularity was up for grabs so System B might as well expose a ProcessCustomerNotification instead of CRUD.

Consistency: From the point of view of consistency option 2 is awful. Just because you have web services doesn’t automatically mean you have SOA. It becomes a mess of point to point web service connections very easily which is why we go for hubs in the first place. “So what about “distributed SOA then?” you may legitimately ask. I have heard that term bandied about a lot in various blogs by folk who don’t seem to like hubs for various reasons. Now option 2 when extended to the nth degree may or may not resemble a distributed SOA but I don’t know because I haven’t actually seen a distributed SOA or a solid reference architecture for the same which covers all the bases.

Isolation of systems: One of the classic arguments for the hub is “what about the time when the backend system changes? You need to avoid breakage in consumers”. That still holds good.

One point offered by a colleague was that you could always introduce some mapping into the WS to cater to future changes in System B and it didn’t have to necessarily be in a Biztalk layer. I agree with that but the implementation of the backend service needs to be really well partitioned to allow inserting a translation layer without causing issues.

Cost: This is an interesting consideration. Option 1 may cost more when you consider that System B could well expose whatever the portal needs so why pay to build something in between that would only (at least initially) be a pass through layer? But then again, in the short term point to point is always attractive from the cost point of view and mediation is more expensive. It begins to bite soon enough when the inevitable changes begin. If it didn’t, there wouldn’t be an integration business at all. If you were to design System B to be well factored so you could cater to potential change, then the cost of doing that could well be equal to the cost of putting in the hub components in the first place (assuming that the hub already exists – if it didn’t, then the cost of acquisition could prove to be too much). Secondly, when it comes to the notifications to other systems, you are actually adding extra costs to write the notification emitting component (which is not always a simple database polling option).

Opportunity: (or is it opportunism ). This issue is somewhat related to cost. When things start to break and the business starts to worry, there is hardly any time to bring in a proper mediation layer. Further, if the business didn’t want to pay for something like Biztalk early on (because they were convinced that all they needed was the web services) they are hardly likely to change their minds and pay for it at the end. In our case it helps that the hub is existing infrastructure so we are plugging everything into it early and routing through it for all messages so there’s no last minute purchasing considerations to worry about.

Composition (vs.) Mediation: Depending on your needs, providing a joined up processing could take the form of Process Composition or just an ESB based mediation /routing. If all you need to do is inform other systems of an occurrence in one system, publishing a notification to a bus and transmitting from there will work fine. If the clients (like our portal) actually want to compose the data and view & act on it as a single unit, then elements like orchestrations (with various different patterns) come into play. This is where option 1 is more suitable.

Management & Operations: it’s a heck of a lot more work to try and track messages in option 2 than it is in option 1. Even if you call out to the BAM Interceptor (in the Biztalk world) from custom components, it will be a significant effort to track all the way. Option 1 would be much easier because the hub is in control right from the start.

The nature of the consumer: I used the term mash-up rather loosely when I described my portal above. IMO mash-up refers to more of a read-only approach and so you can use various techniques to pick up the data and present it but when the consumer is a transactional system (to an extent) and when the data elements on the portal (in various portlets) are linked in some way, then things are not so easy. Dealing with a set of business services in one place is much better than pointing all over the place to get/set data.

Performance: This is a big one especially when you consider that SOAP over HTTP doesnt make for rocket speed. Calling one web service can be bad enough but passing through one and onto another can be painful. I’ll post about some considerations here and where we have got to in my next post.

Our conclusions and the present: Congratulations if you have come this far! It may seem obvious as to what our conclusion was – we went for option 1. It has proved to be correct (as I knew all along ) because the core business service that links System A and B is now extending to pull in assorted data sets from related systems and display them in the portal as well as managing changes to those data sets. For the portal, there is no change at all (except new element blocks being added to the service schemas that relate to distinct segments on the UI) and managing them is much easier. The service layer which was initially a pass through is, as anticipated, much more solid. System B still needs only pass through maps because we designed the API to be very granular but we can handle any changes easily. System A, being an existing one, has an API that’s more fine grained so it’s easy to see where the responsibilities of all the components lie.

So, those are my opinions on the value and location of the services layer. I hope it was a good (and maybe even vaguely useful) read. I’d love to hear your thoughts on this issue and how you make similar decisions and if you have successfully done a distributed SOA and would like to share your thoughts and pointers to where I can learn more about it,  do let me know.