Oslo – Its about time

Wow!! finally something huge to look forward to. Just got alerted to the Oslo announcement at the SOA BPM conference and theres good details on blogs from Tim Rayburn and Charles Young. Looks like factories will make their way into the Biztalk space and not only that, theres even a full blown repository.

Gosh, I wish i could have been in Seattle. It must have been really exciting to hear it first hand. I applied for a British passport over a month ago and the Home Office has my current Indian one as well so im stuck. I really wanted to go to Seattle this time and even opted for that conference instead of TechEd. Hope the passport arrives in time for me to go to India for my hols at the end of November.

Well, anyway, the poor cousin of the Seattle SOA/BPM conference is being held at Reading, UK for a day on Nov 14th. Its only one day, which is rather sad. Maybe next year they will have something longer. I will definitely make it for that. I guess they will be talking about Oslo at that event as well.

Anyway, as i have posted on many previous occasions, factories and biztalk development are made for each other and it was rather suprising that we didnt have anything on that front all this time. But according to Tim’s post 2009 is the earliest date when we can see anything. In the meantime i guess we can all continue plugging away at tools, utilities, maybe even little factories to help with biztalk development at least for a small audience. This announcement definitely validates the approach..

I recently read about the SOA World Readers Choice Awards for BPM Servers and was suprised to find not even a single mention of Microsoft in that. There was a complete Java bias. Hey, even Oracle got a mention and having used some of their stuff a while ago, i couldnt resist a snigger. Perhaps the acquisition of Collaxa and re-branding it as their BPEL server did the trick. I was impressed with some of the functionality of the BPEL Server after having read the BPEL CookBook which was more or less an Oracle advertisement, but IMO Biztalk compared quite favorably in the areas covered in that book. By the way, the book is still a decent read. Many Biztalk developers are rather comfortably cocooned in a world of Biztalk and all things MS alone and its really cool to learn how other products handle the same kinds of tasks. After all BPM servers are pretty much standardised now in terms of the platform services, features etc, so its mostly the tooling that differentiates them.

I also read some other research articles (quite a while ago) into how flimsy some of those products are, requiring “sticky tape” code in various flavors (xml, javascript, java) all over the place to get a solution developed and one would think that the more consistent story with Biztalk would have helped gather at least a mention, but it didnt.  You could interpret this in two ways. Either its just a Java bias which isnt surprising in this space (and no fault of the magazine either. I subscribe to that so i know it has fair coverage of all things SOA) or that Biztalk has a long way to go to become really user friendly and easy to develop with.

Integration will never be trivial, but rich, deep tooling can certainly help simplify some of the challenges in this field. It is rather ironic, that MS is usually recognised for GUIs but in this case, there isnt much to boast about. Its also worth noting that some of those awards were for a “Web Services development platform” which we dont have. Biztalk isnt such a platform. The closest match could be the Service Factory Modelling edition, but that isnt a full fledged product either.

I realize that for Biztalk, the internal integration with WCF, WF etc is, in the long term, much more valuable than just a flashy GUI, but hey, theres a lot of people out there who arent going so deep (and may not ever require the power of a WCF based communications stack ) and they want a tool they can rapidly get up to speed with. The learning curve for ol Biztalk is still pretty steep.

Anyway, If MS can deliver on Oslo, it would beat the pants off any of their competitors. The Readers Choice awards may still ignore Biztalk, even in 2010  (wonder if it will still be called Biztalk then!!), but, at that point, who cares? Go CSD!!!

WSE 2&3, Biztalk and VS 2005 Web App Projects

I previously posted about message serialization issues while testing out a particular WSDL. One thing i found out from their documentation was that they use Web Service Security. It initially set  some alarm bells ringing because i immediately thought of using WSE3 and remembered that there is no out of the box adapter for Biztalk 2006 to use that. Of course, if we could justify the cost to the client, then we could purchase the third party adapters but lets just say that making such a business case isnt very easy. Further, its only a UserName token required and we dont need any of the other WSE infrastructure so the purchase option looked even more distant.

Anyway, one good thing i found is that the Biztalk WSE2 Adapter SP1 works for both v2004 and v2006. The last time i worked with WSE (v2) and that adapter was over a year ago so i decided to refresh my memory and try out a sample project. Since i already had the WSDL and had worked out some simple consumers as i indicated in my previous post, i thought it would be a good idea to WSE enable the service and then try to consume it from Biztalk.

The first thing i installed was WSE 2 SP3. Apparently it works fine with VS2005 but there is no design time support in the IDE. That meant i would have to go down the WSE3 route and that was a bit of a blow since in the back of my mind there was an association that WSE2 Adapter only talks to WSE 2 enabled services, so enabling a service with WSE3 wouldnt really help me.

Anyway, I then installed WSE3, started up my project, launched the Settings tool inside the IDE and lo and behold, the check box labelled ”Enable Microsoft Web Services Enhancements Soap Protocol Factory” was greyed out. Further, when i checked the option “Enable this project for WSE”, it proceeded to create an app.config file instead of placing stuff in my web.config. Curiouser and curiouser. So after much digging around i stumbled on an MSDN forum post here where the moderator, Mark Fussell said

“The Visual Studio 2005 Web Application Projects (RC1) project type does not work with the WSE 3.0 integrated Visual Studio configuration tool. To use WSE 3.0 with the Visual Studio 2005 Web Application Projects (RC1) type you have to use the configuration tool standalone, launching this by; Start-> All Programs-> Microsoft WSE 3.0-> Configuration Tool and then loading with the app.config file or the web.config file depending on the type of project.”

Well, how do you like that? The Web Application Projects has now been an official part of VS2005 (from SP1) but no change to WSE support, but thats probably because of the focus on WCF.

So, i have now started to use the external config editor. Basically all i am attempting is a POC to connect Biztalk to a service expecting the WS-Security UserName mechanism. I am assuming that when i get back to office and try to connect to the java WS it should still work. Perhaps thats a leap of faith ?

I also think, that the whole WSE2 and WSE3 adventure today could just be a red herring. Maybe what matters is the version of WS-Security that they have implemented and it is that which decides whether Biztalk will choke on consuming the contract. (I somehow doubt that because there isnt anything in the WSDL that makes a difference and pointing Biztalk at my local WSDL didnt throw up any problems . AFAIK WS* is all external to the WSDL and the policy files control these settings.

I’ll find out for sure in a couple of days. but if anyone has some pointers or knows of any gotchas in linking BTS with third party web services requiring WS-Security (and if the adapter makes any difference) please let me know. It would save a lot of time.

[UPDATE: 25th October 2008]: its been a while since I posted this entry but as i have had questions on this post and what we did with WSE finally, i thought it would be good to update it with the following info

(1) The work-around for the lack of IDE integration is to use the external configuration editor. This works for both WSE2 SP2/SP3 and WSE3.

(2) While calling the webservice, a colleague (who did the final implementation) decided to use a custom pipeline component (send side) to handle the WS-Security related elements.

(3) We finally went with WSE2 rather than WSE 3 (Its been so long, i dont remember why ]

Failed to serialize the message part

I finally got my hands dirty with some “proper” work after spending ages in the world of Powerpoint, Visio and Word. We are starting up a new project in a few days and i wanted to do some advance R&D on the vendors schemas to see if we would come to any grief with WSDL formats as has happened in the past (e.g WSDLs with Import directives for the XSD’s and also indicating relative paths for those XSD’s). 

I picked up a sample WSDL from the set they provided and since i was working from home and couldnt connect to the server at the client site, i decided to create a mock web service and some consumers.

I created the mock web service using thinktecture’s WebServicesContractFirst (WSCF) which we have been using for a long time. All i did inside the service was to create a new response object fill it with some dummy data and send it back. One of my colleagues actually a wrote a more generic WS mocking tool and i’ll post about that sometime soon.  I then wrote a regular .NET consumer and a simple orchestration to pickup the WS request message from  a file, send it to the mock WS, get & dump the response. Nothing fancy and no mapping needed.

Now comes the interesting part. When i created a sample message for the .NET consumer to send, i left out many elements which, incidentally, were specified as mandatory. I did not tick the option in WSCF to validate the messages (as i have had problems in the .NET 1.1 version with the SoapExtension they generated). The call worked fine and i was able to dump out my request and response messages.  Then i decided to submit the same message to Biztalk.  Immediately the system threw an error

“  Failed to serialize the message part <RequestMessage> into the type <RequestMessageType> using namespace <Namespace>. Please ensure that the message part stream is created properly. “

I found this article on Saravana Kumars blog where he got the same error while calling a webservice in a messaging only scenario. He advises checking that serialization works for all target types. Although the scenario here is different i decided to check if the same error appears if all elements are filled out, mandatory or otherwise. Now once i filled all the elements, the processing worked!!.

A little more digging around and i found this post from Bryan Corazza where he had exactly the same error and this was because an xsd:date element was missing in his request message. I then removed the date element from my working request message and this error popped up again. I put the date back in, took out some other mandatory elements and the error disappeared.

So, it really was the xsd:date after all that couldnt be left blank.

In terms of the serialization, maybe the option to validate the message would have thrown an error. Alternatively, my usual approach is to create a sample file, de-serialize that into the request message object and then call the web service. I decided to skip that and just instantiated the request message and put in some values. Perhaps the de-serialization would have found the error. I need to verify this. I certainly hope it does. But even if it doesnt, the safest thing to do is validate your request message against the XSD first before sending it across. A bit of a no-brainer really but it can save a lot of frustration.  Watch out for xsd:date elements!!!

Orcas Team Architect and Biztalk

I have always been fascinated by the VSTS Team Architect and the potential the design surfaces have.  I kinda looked on from the sidelines at VS2005 from TechEd 2004 onwards cos I was still stuck in VS2003 land and actually that remained the case till the middle of this year. So although I installed the stuff on VPC’s and tried playing around with things I never got very far, but I did read enough TechNotes and followed along with the newsgroups to gather that while the Application Designer promised a lot, it probably didn’t go far enough or at least, the extension and customization wasn’t for the faint hearted.

I was also disappointed with the lack of support for Biztalk. Sure, there is a prototype for an external Biztalk webservice, but thats ok if you are an ordinary app trying to connect to a Biztalk service. Actually, that also raises an important question. Why have a prototype for a “Biztalk”  web service as though Biztalk web services are any different from ordinary services? What does it matter whats behind the webservice? If you are a Biztalk developer, then a prototype which helps you implement that web service makes sense because you may choose to publish  a schema as a web service or a orchestration, but from the consumer standpoint, its all the same but the prototype doesnt do anything for the BTS developer (If anyone knows the rationale behind that prototype, drop me a line. I’d like to know what they were thinking of when they designed it). Since there wasn’t much Biztalk there, I left it without going any further. However automating Biztalk development and deployment is my idea of development nirvana so I have been looking at various options and i have written some posts on my earlier DNJ blog about these topics.

But, back to the present, I’ve been reading up on the future of VSTS and the features in Orcas, Rosario, Hawaii etc (all code names for the next releases of VS and VSTS). I came across the VS Team Architect blog and I was particularly taken with the features such as Top Down System Design and the Conform to WSDL. The enhancements to Importing & Exporting Custom Prototypes also looks like a great addition. I dashed off a note to the Team Architect team (and also one to Marty Waz) to quiz them on the level of support for Biztalk and what plans they have in that area. I also downloaded Beta 2 and will get cracking on investigating these features as soon as possible.

I think that Biztalk could do with more GUI’s and wizards. This will definitely make it more easy to use and lower the steep learning curve the product currently has. When you are running a team with vastly different skill levels, you cant have enough of automation. No amount of documenting your “standard patterns” helps when you are trying to get everyone at a common level and meet tight deadlines. Automation is definitely no silver bullet, but its a huge bonus for the harried architect.

I have grumbled many times in the past that the Solution Designer promised for the “vNext” never materialized and there’s absolutely no info on whether thats completely dead or if we can expect something in the next version (perhaps in 2010)? I dislike the lack of support in BTS for contract first development or to be more specific WSDL first development. A lot of the implementations I do now require a WSDL to be defined first with the third parties and then for development to proceed on that basis. But there is no out of the box feature for Biztalk to do this. Aaron Skonnard wrote an article in MSDN magazine a long time ago about contract first development, but the options to get Biztalk to conform to a WSDL are messy to say the least. There’s a BPEL import wizard but no WSDL import wizard. Why? Surely more developers care about WSDL than about BPEL!! I know they position the product as fully supporting BPEL4WS + some extensions and so the BPEL import will score some points there, but why not a WSDL import?

Its this area where I think that some integration between Biztalk and the Orcas Team Architect would be very powerful. I think it would be liked a dream come true to be able to feed a WSDL into a wizard and choose if we want a schema based web service or orchestration based web service and have it generate the orchestration. Of course it cant generate the “business logic” of the orchestration but it could generate the Facade which links to the ASMX or SVC interface and contains the correct handling of incoming messages, validations, returning of SOAP faults etc. The “inner” orchestrations can be hand coded or a stub could be created where the facade either calls the inner orchestration directly or through the message box or any other mechanism.

The TopDown system design would also be a great feature and the prototype import/export could easily link in with Jon Flanders Pattern Wizards tool (with a prototype encapsulating a project template or one of the pattern items).

We could go even further. How about an adapter factory that can generate WCF based adapters or a regular one? At this point currently we have an adapter framework & some wizards, but then there are also some adapter base classes to use (and there’s some pretty detailed guidance in a whitepaper on writing transactional adapters) and we have to plow through numerous options and figure out the correct interfaces to implement. After that we have to consider how to link in with SSO and all that. This area is ripe for a guidance package.

Also consider the amount of guidance in the end to end scenarios. Getting them all up and running and wading through the docs and code is not a trivial effort, but making all that guidance executable would be such a bonus.

I think a Biztalk solution software factory makes a lot of sense and when you look at the kinds of stuff coming out from P&P (the service factory, the new modeling edition of the service factory), a Biztalk offering would be really cool and would be able to leverage the advances in GAT, DSLs etc. Maybe a community effort to build something like this would be worth exploring. Biztalk, having such a wide range of applications, would probably need more than one factory (SOA, BPM, CBR etc + some horizontal facilities- adapters, pipelines, deployment etc) and a factory to build solutions on top of the ESB toolkit would also be great. These kind of things would take Biztalk usability far beyond its competitors.

What do you think? would this be worth starting up as a community effort? let me know. I came across this post from someone who is working on something similar and have pinged him to enquire if thats a private effort or something intended to be bigger. Lets see what comes up.