Using INTERSECT with LINQ to XML

In terms of hands-on coding (not general awareness) I’m a bit of a newbie to the world of LINQ actually, having only dabbled with a little LINQ to XML in MockingBird and even there I wasnt too impressed with it in the area of XPath queries. But I came across something yesterday that is a testimony to the power of LINQ.

My scenario was that I wanted to compare two XML documents that followed the same schema, but I wanted to do this in  a fairly generic way without writing code to explicitly pick up every element in the hierarchy. My requirement was to find all common elements between the two documents and also the elements in one and not the other.

Take the following example:

Document-1
<Authors>
  <Author ID=”1″ Name=”AuthorA” JoinDate=”3/1/2009″/>
  <Author ID=”2″ Name=”AuthorC” JoinDate=”3/1/2009″/>
</Authors>
Document-2
<Authors>
  <Author ID=”1″ Name=”AuthorA” JoinDate=”3/1/2009″/>
  <Author ID=”2″ Name=”AuthorB” JoinDate=”3/1/2009″/>
</Authors>

I quickly found that LINQ has this powerful INTERSECT function which would allow me to find the common elements and the EXCEPT function which will find the distinct elements.

My first attempt (at finding the common elements) was like this:

var commonFromA = aDoc.Descendants(“Authors”).Intersect(bDoc.Descendants(“Authors”));

But this did not work. After much more attempts and discussions with colleagues, it was beginning to look like i could only use INTERSECT with native types and I would either have to write a custom IEqualityComparer<T> or write more complex code involving anonymous types (which are , by the way, a brilliant feature of the framework).

But LINQ is supposed to be elegant, right? So I posted the question on the MSDN Forums and got an immediate reply from Martin Honnen   a MVP in this area, and yes, the solution was elegant and just in one line.

var commonFromA = aDoc.Descendants(“Author”).Cast<XNode>().Intersect(bDoc.Descendants(“Author”).Cast<XNode>(), new XNodeEqualityComparer());

As Martin explained, the set operators like INTERSECT and EXCEPT work on object identity not value comparisons and as I had distinct XElement objects in different documents my initial attempt would not work. However, the XNodeEqualityComparer comes to the rescue and casting the XElement to XNode was all that was required.

What’s even more interesting is that in .NET 4.0, we have something called “contravariance” which will allow the INTERSECT code above to work without the explicit cast. Martin explains this very well in this post on “Exploiting Contravariance with LINQ to XML”. I always wanted to understand what Covariance and Contravariance were all about and this is a great explanation. On the same subject, also check out Eric Lipperts blog article.  I had come across that post earlier but didnt have any immediate need for that functionality so I didnt pay attention, but this time, I did.

So, there you have it. A one line solution for comparing XML documents. (The “EXCEPT” code was also one line). Of course if you want to find out specific attribute values and changes, then the code becomes more involved, but you’ve gotta admit that this is elegant. Can you imagine how much code this would need in the Xml DOM world!!

I’m starting to get hooked on LINQ! 

BizTalk, WCF and the Bad Gateway

We ran into an interesting problem a couple of days ago and I decided it would be worth posting the solution in case anyone else runs into it.

The scenario involves a Windows Service communicating with a BizTalk WCF endpoint which is hosted using WCF-CustomIsolated and some fairly complex custom bindings, notably using X.509 Mutual Certificates. This is the second half of the communication, the first being where BizTalk sends a request to a vanilla WCF endpoint and gets an ACK, the service does some processing and the WinService then does a “call back” with the “real response” / outcome of the processing.  Both communication routes are secured via the same bindings. We didnt have any trouble with the transmission from BizTalk to the WCF service but the WinService initiated communication always failed with a HTTP 502 : Bad Gateway message. This had us stumped for a while as some articles we read seemed to suggest that there was a problem on the server. Now if you have looked into custom bindings and certs in BizTalk, you will know that there is some really heavy duty and scary stuff there so we wondered whether the way we had setup the certs and credentials info was playing havoc with the communication.

So I used the trusty WSCF.blue and mocked up another WCF service to mimic the BizTalk endpoint replete with the bindings (no, WSCF Blue doesnt support custom bindings yet, but will sometime soon) and swapped the URI in the WinService and it worked just fine. Still puzzled, I then created a console app to mimic the winservice and it was able to communicate with both my mock BizTalk service as well as the real endpoint. Curiouser and curiouser. At that time the only difference was in the App Pool between the BizTalk service and mock service, but i used the same app pool and still the problem only manifested between the Winservice and Biztalk.

Anyway, with some guidance from an expert colleague (who, unfortunately wasnt around when I was writing those mock clients and services) It turned out that the problem was with a setting “UseDefaultWebProxy”  in the Win Service. The difference between the WinService and the console client was that the console was running under my credentials and in the IE settings I had turned off the proxy for the specified URL so although I didnt specify the setting as “false” in the client it automatically picked up my settings. But the WinService was running under another user account, and so the system didnt care about my proxy settings and we had to explicitly set the value to “false”. 

That was it. A little setting buried deep in the bindings that caused us the grief. After sorting this out I stumbled across this post from Kenny Wolf  which points to the offending setting. Wish I had found that earlier, but then I wouldnt have realized that aspect of the console client where it picked those settings. So, hope this helps someone in a similar situation.

MockingBird v1 RTM

MockingBird v1 is now formally released. With the squeeze in available time for personal projects, it’s taken me nearly 6 months since the beta and only recently did the site see any check-ins. There’s quite a few updates, and enhancements to the suite. Actually the beta was pretty stable with hardly any bugs that needed to be fixed but there were a few feature requests and to be honest, I started getting rather impatient to move to WCF but needed to formalize the v1 release first. Here’s a laundry list of all the features of the new release. The documentation, included with the package has also been substantially updated.

This release will be the last to support the ASP.NET HttpHandler. V2 will be completely revised to support WCF. The change in the engine should be transparent to the end user and of course with WCF we will get a lot more benefits including transport independent request interception. It will also be the last to support an independent “Studio” module as I have some plans for a brand new UI

Simulator functionality changes

· Now supports Enterprise Library logging as well as log4net

· Logging all request and response operations.

· Support for "mapped soap Actions" (when soapActions do not correspond one to one with operation names or when the default parsing of  soap actions cannot be done)

· Supports mapping of operations to request element root names (when soap actions are all the same, such as in Apache Axis web services)

Simulator Packaging changes

· Config for unity and services now externalized

· MockingBird system config now externalized

Simulator Internal changes

· System Configuration Manager abstraction layer for config data storage

· StrongTyped application configuration.

· New OperationNameInferenceEngine

· Support for tokens and token expansion in configuration settings.

Studio Fixes

· Disabling of context menu on all nodes except for message level nodes.

· WSDL Browser "Save As " dialog now supports proper file masks (also takes in the service name and suggests the name for the file)

· No unhandled exceptions in the studio.

· Auto resize of all controls on form size change.

XmlInstanceGenerator Functionality

· Additional entry points to the XIG.

A big thank you to all who sent comments and gave me feedback. I hope you find the updates in this release to be useful.

V2 is going to be really big, so watch this space.

For v2, There are a number of work items published on the CodePlex space  so please weigh in on them and vote for the ones that will be most useful to you and also please feel free to send me suggestions for other features as well as bug reports.

To DAL or not to DAL

Do BizTalk consultants need to care about Data Access Layers? Does a BizTalk solution really need a DAL?  These are the questions that I’ve been mulling over in the past few weeks. Let me explain.

There are a couple of places where a BizTalk solution encounters a DAL. The first is where the DAL acts as an integration enabler. Here the endpoint of the LOB application we are integrating with happens to be a database. The second is where the DAL acts as a process enabler. Here the DAL provides the underpinning of the business process (that is, as part of the business process, it is frequently necessary to update a database with the state of the business document being operated on).

In my current gig, we are using both BizTalk and SSIS. SSIS is great for the ETL and various data related actions. BizTalk then takes over and passes the data to an LOB application doing various business processes as part of that communication. The nature of the processes is such that there is a significant DAL. Early on in the project we went through the usual debate on whether a custom DAL was necessary or if we should just use the requisite database adapters. Isnt the database adapter an obvious choice?  Maybe, or maybe not. In an earlier post , i talked about just such a situation a few years ago where we had choose whether to link directly to the DB or wrap the system in a web-service first and as i explained, things didn’t turn out the way they were expected to.

So, what are the considerations?

1. Firstly, (as I explained in the post and the follow up posts) one of the key issues is the level of abstraction you are given. Especially when dealing with the scenario of integration enablers, a database endpoint is very rarely coarse grained enough to support a service oriented approach. Its more likely that you will be provided with CRUD level interfaces. Even if you decide to direct all communication via an orchestration that wraps all this, how does the orchestration actually call the backend system? Go via the adapter or use a DAL?
2. For the scenario of process enablers, abstraction comes into play again. You don’t want to be cluttering up your orchestrations with bits and pieces of database schema related stuff. You could choose to wrap the database calls in a coarser stored proc but this leads to the next key point which is
3. Performance. If you have a number of send ports (for all these stored procs) in the middle of your orchestrations, there is a cost associated with all those persistence points. If your transaction handling requirements permit, you could think about wrapping some of those calls in atomic scopes, but you have to be  very careful with this. If you do encounter an issue and everything gets rolled back, are your processes really designed to start at the right place all over again without compromising data integrity?
4. If your DAL is designed well, your orchestrations will benefit from having to call methods on business level entities and, just from a persistence point consideration, will, in my opinion, be better off.
5. Transaction Bridging : There were a few situations where we had to bridge the transaction across the database and the segment of the business process. Fortunately, the DAL being of extremely high quality (courtesy of an expert colleague) made this very easy to do.

But, having said all this, a DAL doesn’t come free. You have to write code. Sometimes lots of it. The more code you write, the higher the probable bug density. If the functionality can be satisfied with a code-generator then that will reduce the code you have to write, but it DOES NOT reduce the amount of code you have to MAINTAIN. I think many developers forget about this last point. I’m all in favour of code-gen, but don’t forget the maintenance cost.  (Further, if the functionality in the middle of your processes can be satisfied with boiler plate code, perhaps it’s an opportunity to question what it’s doing there in the first place. Can it be pushed to a later stage and componentized? )

I must confess, at one point, when wading through a sea of DAL code early on in the project, I was quite tempted to throw it all away and go for the adapters, but the considerations above outweighed the pain at that point. Now much later, with everything having stabilized, we know just where to go to make any changes and the productivity is quite high.

But I’ve seen cases where BizTalk developers didn’t care about the SQL they wrote and they ended up in a mess with locking and poor performance. And it takes a really good developer to write a first class DAL and having interviewed and worked with a number of devs I can say that its hard to find good skills in this area. Pop quiz: Do you know how to use System.Transactions yet ? 

There is always the option of using something like NHibernate. If you use some coarse grained stored procs and some business entities, you could kill all the “goo” in the middle by letting NH take care of the persistence. That, i wager would reduce the bug count in that area. But watch out for the maintenance times and the bug fixing. When there’s a component in the middle that you don’t know the internals of, it can make life very hard when trying to track down bugs.

That leads me on to the point of making choices based on knowledge and not ignorance. If you want to adopt “persistence ignorance”, don’t do it because you cant write proper DAL code yourself. Do it for the right reasons.

So I hope the points above have given some food for thought. Custom code is not always bad as long as it is approached and implemented correctly. Whether you choose to use a DAL or not, do it with careful thought on issues like the ones above. As always, your feedback is welcome.

Technorati Tags: BizTalk,DAL,NHibernate

WSCF.blue v1 RTW

I’m a little late in announcing this. WSCF.blue is now formally released to web as v1. Although its only been out 3-4 days now we’ve crossed over a 100 downloads already. Whew. Looks like a lot of folk were waiting for this release

Christian Weyer has blogged about the release. I would also like to call out Alex for taking so much initiative and coding till 3am on some days (and emailing the group at that time too !!). We definitely owe this release to him.

Now on to v2. My next post will be about the roadmap and our shopping list of features for the next few releases. Check out the final v1 release. Hope you like it. As always, your feedback and suggestions would be most welcome and appreciated.

WSCF.blue – Paste-As-Schema

WSCF.blue now has a new feature called “Paste-As-Schema” (which takes inspiration from the REST starter kit’s Paste-As-Type). Alex explains the new feature in detail in his post.

The MSI has now been updated to version 1.0.4. It is still formally in Beta-2. It also includes a few bug fixes in the original Beta-2 (1.0.3) release. Please check the CodePlex workspace for details. Hope you find the new feature useful. If you run into any issues do let us know.

WSCF.blue Beta-2

Beta-2 of WSCF.blue is now available.

The big feature of this new beta is the standalone Data Contract Generation and we have also put in some other fixes and addressed some outstanding issues (as well as rolling up fixes since the last beta).

The feature list is as follows. The “More Information” links point to posts on Alex’s blog with screenshots and lots of details.

• The Generate Data Contract Code feature is now available and supports the selection of multiple XSD/WSDL source files. (More Information)
• When the Enable WSDL Endpoint option is chosen a reference to the Thinktecture.ServiceModel.Extensions.Metadata.dll assembly is automatically added.
• Support for VB.NET has been added to the code generation.
• You can choose if operation methods on the service class will throw a NotImplementedException, call an implementation method in a partial class, or will be defined as abstract methods. (More Information)
• Force the SOAP actions (Action and ReplyAction) applied to each operation contract follow the standard WCF format: <namespace>/<service>/<operation>[Response]

Hope you find this release useful. As always , keep the feedback and suggestions rolling in. Enjoy 

My first year at MCS

Last Tuesday (the 18th Aug) marked my first year at Microsoft Consulting Services (UK). I can’t believe how fast time has gone by. Years ago, I used to wonder how it would be to work in MCS and dismissed the idea as too high to aspire to and yet here I am after a full year at the very same place. So what’s life been like working for Microsoft? Here’s what I’ve seen. [Remember, this is just the view through my lenses , nothing indicates company policies etc ]

My first 3 months were rather quiet, as they are considered a “ramp-up” period and usually one is expected to network and round-off your skills. However, I was warned that the 3 months would fly and then it would be liked drinking from a fire-hose, and the warning was absolutely spot on. It’s been a rollercoaster from last November.

The first thing that struck me (not surprisingly though) was the calibre quotient of my colleagues. While I have, in the past, worked with some top notch developers and architects, it just seems that the “average” knowledge is a mile higher than at other places. For instance, I could think I know all about SQL and then find myself sitting next to a chap who did a stint on the SQL product group and knows more about the core engine than anyone else in the world (now, this part is anecdotal, i didn’t actually encounter that myself, but heard of someone who did, but you get the picture  . Darren Jefford is the architect on my current gig. A couple of years ago, when just reading through Darren’s blog I couldn’t have imagined working on the same team as Darren. C’est la vie!  (I couldn’t resist that bit of name dropping).

The second thing that took me by surprise was the how committed Microsoft is to partners both at the platform / product level as well as in Services. With our products, we pride ourselves on building the best platform and tool support possible and empowering partners and vendors to build on top of that. Sure, MS has a record of being somewhat ‘predatorial’ in the past, but it looks and feels different in that respect now.  In the Services side, it’s common to find a number of partner companies working alongside us to deliver the projects and there too, it’s common to find that many of the developers and architects from partners  are leading bloggers. MVPs and generally, well known in their respective dev communities. [Ok, one more name-drop.. did you know i’m actually working with the legendary BizTalk “Arch Hacker” ? It’s true he wears the mask of Zorro to work !! …(ha ha just kidding….)]

Then there’s the veritable flood of information and access to stuff ages before it becomes public!  Of course, while a lot of that is non-disclosable to external audiences, within the company, information does flow quite freely (which I heard, from someone who joined us from a competitor, is not always the case in large software houses). Unfortunately, there’s seldom the time to actually use all that and in this respect, you’re on your own. There is a lot of structured learning on offer, but balancing that with project commitments is a fine art.

Ok, so that’s all about the company. So, what’s the personal impact been?

Firstly, its the amount I’ve learned technically (which is, of course, to be expected in a place like this). My knowledge of BizTalk, especially, has been deepened considerably both by the requirements of the current gig (which is in its 6th month now) as well as just absorbing tons of stuff from my colleagues.

Secondly, it’s been great to see, first-hand, very large scale projects delivered successfully and on time. I have been involved in some quite big (and successful) projects through the years, but the sheer scale of the projects here is far beyond what I’ve worked with in the past). [On the technical side of this, its been eye opening to see how useful TFS is for project tracking and reporting].

Thirdly, it’s also made me raise the bar for my own community contributions. I take longer to make something public than I would have done in the past.  Although my community contributions are personal and not Microsoft IP,there is the implicit association with MS and i find myself thinking “is this good enough to be associated with an MCS guy!” . Take MockingBird, for instance. This ‘implicit’ association ensured it was better at first release than it would have been, say, 2 years ago. There’s both the knowledge gained over the years as well as the “this has to be good enough to be published by a Microsoft chap” that helped it. Not that I’ve reached some sort of ‘coding guru’ status (or ever will), and  there’s always more to be learnt and improvements to be made in all areas but I’m definitely more satisfied with the quality of my output now.

So, it’s been a good year. I’m looking forward to the rest of year-2. One year at a time!

WSCF.blue – Updates

So there’s been a few things happening on the WSCF.blue front. We’ve had some excellent feedback and also some feature suggestions, so all that has been rolled up into the install package. It’s still Beta-1, but the MSI has been updated to v1.0.2.

The main changes are

(1) the option to format the soapActions when setting the Action and ReplyAction so that it conforms to W3C standards. We had some issues with certain WSDLs where we were getting the replyAction set to “*” (and even svcutil was doing that) which affected the communication.

(2) Options to control the way the service class is implemented. Hitherto we have had the default behavior of the method throwing a “NotImplementedException” (which is a normal thing for a code-generator). But this can be limiting sometimes. So we now have options to either define the class as abstract or partial. Alex explains all this in his post.

Hope you find the new features useful.

MockingBird@SBUG

The UK SOA BPM User Group (SBUG) is holding a series of mini-meetings over the next few months. These are short live meetings / webcasts. I’m delighted to be presenting my web-service mocking tool , MockingBird at the next mini-meeting to be held on the 29th July.

I will be discussing how MockingBird assists and simplifies integration testing with web services, how it is used in Biztalk and non Biztalk scenarios, how the system works, how it can be extended and the roadmap.

Please check the event link on the SBUG site for details.  All are welcome to join (no membership needed). Hope to see a good turnout