Using INTERSECT with LINQ to XML

In terms of hands-on coding (not general awareness) I’m a bit of a newbie to the world of LINQ actually, having only dabbled with a little LINQ to XML in MockingBird and even there I wasnt too impressed with it in the area of XPath queries. But I came across something yesterday that is a testimony to the power of LINQ.

My scenario was that I wanted to compare two XML documents that followed the same schema, but I wanted to do this in  a fairly generic way without writing code to explicitly pick up every element in the hierarchy. My requirement was to find all common elements between the two documents and also the elements in one and not the other.

Take the following example:

Document-1
<Authors>
  <Author ID=”1″ Name=”AuthorA” JoinDate=”3/1/2009″/>
  <Author ID=”2″ Name=”AuthorC” JoinDate=”3/1/2009″/>
</Authors>
Document-2
<Authors>
  <Author ID=”1″ Name=”AuthorA” JoinDate=”3/1/2009″/>
  <Author ID=”2″ Name=”AuthorB” JoinDate=”3/1/2009″/>
</Authors>

I quickly found that LINQ has this powerful INTERSECT function which would allow me to find the common elements and the EXCEPT function which will find the distinct elements.

My first attempt (at finding the common elements) was like this:

var commonFromA = aDoc.Descendants(“Authors”).Intersect(bDoc.Descendants(“Authors”));

But this did not work. After much more attempts and discussions with colleagues, it was beginning to look like i could only use INTERSECT with native types and I would either have to write a custom IEqualityComparer<T> or write more complex code involving anonymous types (which are , by the way, a brilliant feature of the framework).

But LINQ is supposed to be elegant, right? So I posted the question on the MSDN Forums and got an immediate reply from Martin Honnen   a MVP in this area, and yes, the solution was elegant and just in one line.

var commonFromA = aDoc.Descendants(“Author”).Cast<XNode>().Intersect(bDoc.Descendants(“Author”).Cast<XNode>(), new XNodeEqualityComparer());

As Martin explained, the set operators like INTERSECT and EXCEPT work on object identity not value comparisons and as I had distinct XElement objects in different documents my initial attempt would not work. However, the XNodeEqualityComparer comes to the rescue and casting the XElement to XNode was all that was required.

What’s even more interesting is that in .NET 4.0, we have something called “contravariance” which will allow the INTERSECT code above to work without the explicit cast. Martin explains this very well in this post on “Exploiting Contravariance with LINQ to XML”. I always wanted to understand what Covariance and Contravariance were all about and this is a great explanation.

Essentially, with Contravariance, you can pass in the base type XElement even though the comparison (with XNodeComparer) is expecting an XNode , (the derived type) and you dont need to mess with casting etc. With Contravariance you are also not mutating the object itself (actually, you cannot change the object) so this works.

On the same subject, also check out Eric Lipperts blog article.  I had come across that post earlier but didnt have any immediate need for that functionality so I didnt pay attention, but this time, I did.

So, there you have it. A one line solution for comparing XML documents. (The “EXCEPT” code was also one line). Of course if you want to find out specific attribute values and changes, then the code becomes more involved, but you’ve gotta admit that this is elegant. Can you imagine how much code this would need in the Xml DOM world!!

I’m starting to get hooked on LINQ! 

My first year at MCS

Last Tuesday (the 18th Aug) marked my first year at Microsoft Consulting Services (UK). I can’t believe how fast time has gone by. Years ago, I used to wonder how it would be to work in MCS and dismissed the idea as too high to aspire to and yet here I am after a full year at the very same place. So what’s life been like working for Microsoft? Here’s what I’ve seen. [Remember, this is just the view through my lenses , nothing indicates company policies etc ]

My first 3 months were rather quiet, as they are considered a “ramp-up” period and usually one is expected to network and round-off your skills. However, I was warned that the 3 months would fly and then it would be liked drinking from a fire-hose, and the warning was absolutely spot on. It’s been a rollercoaster from last November.

The first thing that struck me (not surprisingly though) was the calibre quotient of my colleagues. While I have, in the past, worked with some top notch developers and architects, it just seems that the “average” knowledge is a mile higher than at other places. For instance, I could think I know all about SQL and then find myself sitting next to a chap who did a stint on the SQL product group and knows more about the core engine than anyone else in the world (now, this part is anecdotal, i didn’t actually encounter that myself, but heard of someone who did, but you get the picture  . Darren Jefford is the architect on my current gig. A couple of years ago, when just reading through Darren’s blog I couldn’t have imagined working on the same team as Darren. C’est la vie!  (I couldn’t resist that bit of name dropping).

The second thing that took me by surprise was the how committed Microsoft is to partners both at the platform / product level as well as in Services. With our products, we pride ourselves on building the best platform and tool support possible and empowering partners and vendors to build on top of that. Sure, MS has a record of being somewhat ‘predatorial’ in the past, but it looks and feels different in that respect now.  In the Services side, it’s common to find a number of partner companies working alongside us to deliver the projects and there too, it’s common to find that many of the developers and architects from partners  are leading bloggers. MVPs and generally, well known in their respective dev communities. [Ok, one more name-drop.. did you know i’m actually working with the legendary BizTalk “Arch Hacker” ? It’s true he wears the mask of Zorro to work !! …(ha ha just kidding….)]

Then there’s the veritable flood of information and access to stuff ages before it becomes public!  Of course, while a lot of that is non-disclosable to external audiences, within the company, information does flow quite freely (which I heard, from someone who joined us from a competitor, is not always the case in large software houses). Unfortunately, there’s seldom the time to actually use all that and in this respect, you’re on your own. There is a lot of structured learning on offer, but balancing that with project commitments is a fine art.

Ok, so that’s all about the company. So, what’s the personal impact been?

Firstly, its the amount I’ve learned technically (which is, of course, to be expected in a place like this). My knowledge of BizTalk, especially, has been deepened considerably both by the requirements of the current gig (which is in its 6th month now) as well as just absorbing tons of stuff from my colleagues.

Secondly, it’s been great to see, first-hand, very large scale projects delivered successfully and on time. I have been involved in some quite big (and successful) projects through the years, but the sheer scale of the projects here is far beyond what I’ve worked with in the past). [On the technical side of this, its been eye opening to see how useful TFS is for project tracking and reporting].

Thirdly, it’s also made me raise the bar for my own community contributions. I take longer to make something public than I would have done in the past.  Although my community contributions are personal and not Microsoft IP,there is the implicit association with MS and i find myself thinking “is this good enough to be associated with an MCS guy!” . Take MockingBird, for instance. This ‘implicit’ association ensured it was better at first release than it would have been, say, 2 years ago. There’s both the knowledge gained over the years as well as the “this has to be good enough to be published by a Microsoft chap” that helped it. Not that I’ve reached some sort of ‘coding guru’ status (or ever will), and  there’s always more to be learnt and improvements to be made in all areas but I’m definitely more satisfied with the quality of my output now.

So, it’s been a good year. I’m looking forward to the rest of year-2. One year at a time!

VS Color Schemes : Rejuvenating Development

Ok, so I’ve been really late to this particular party, but I gotta say, I’m absolutely thrilled with the effect that changing the color schemes of VS has on improving my coding morale!! I’ve been using several schemes from Tomas Restrepo’s collection and its done wonders for me  (specifically Ragnorak Blue, Grey and Moria Alternate).  Since I’m using VS 2005 and 2008 side by side, I have quite different color schemes for them and it makes things more interesting than the mundane white background. Maybe it’s also age and the fact that my eyes get tired more easily but hey, Consolas at 15pt looks awesome.

Having said this, I also started work on Dev10 and I must say, the OOB color scheme is nice. The new WPF editor renders the fonts much crisper and neater so I’ve been content to leave it without changing to a dark background. I guess we’ll have to wait a while for some new schemes to emerge. Quite sure the new editor has various new options for color schemes.

Another thing that its done, aside from make my IDE look nicer, is that it’s given me a coding boost. In fact, my releasing BizUnitExtensions 3.0 is more down to the new color scheme than anything else  .

So, if you havent taken this particular plunge yet, why not try it out?

Waltzing with WSDL

One of my immediate goals for MockingBird is to have the “Configurator” UI done where you can just pick a WSDL on file or point to a URL and have the tool shred it and generate sample request and response messages into the correct folders (and generate the Handler Configuration) after which the user can then do any extra XPath based configurations to make the system more dynamic. Sounds easy enough, doesnt it? Well, thats what i thought too, until i delved into the intricacies of WSDL Parsing.

The class most people use is ServiceDescription and to be fair, it gives some information about the WSDL but navigating it is not easy at all and its not all that intuitive after you get to the point of listing all Operations inside the PortTypeCollection. There is also another big issue with trying to use ServiceDescription alone and that is the fact that WCF by default gives you multipart WSDLs. So getting hold of the WSDL from a WCF endpoint is not straightforward but one class that comes to the rescue is DiscoveryClientProtocol as Mike Hadlow points out in his article. This behavior can be changed as pointed out by Tomas Restrepo in hs post on “Inline XSD in WSDL with WCF” and Christian Weyer takes it up a notch in his article on Flat WSDL in WCF.

Anyway after pondering this a while, I decided it would be in my best interests to factor the code into a WSDLRetriever and a WSDLDescriptor. The Descriptor takes a dependency on WSDLRetriever (an interface of course, so its easy to mock) and expects  a single WSDL XDocument which it can work with. Now with this approach I dont care if the endpoint (if the WSDL is not on file) is a WCF endpoint or not, the Retriever does all that work. So the Descriptor can do its work and with that sorted i wrote up the first few tests for the Descriptor using a mock Retriever. So far so good.

Now, along with the requirement stated above, I also want to build a nice UI, where, you can not only do a single click to generate messages, but you can exercise finer control over the message content before storing them in the appropriate folders. So imagine if you will, a tree view which shows all the operations and the messages in them and immediately under the message name, the name of the complex type and additionally the body of the type (not in the tree node, but in an associated panel).

Of course, I’m not trying to write a completely generic WSDL parser, but something thats immediately usable by MockingBird and for other tools which can use a WSDL as input.

I had a look around at some blog posts and samples but none seemed to go as far as I wanted. I also took a look through the code for WSCF Blue  but couldnt make immediate use of that API. (I’ve since chatted with Buddhike about that so there may be something i could do with WSCF libraries in future). I looked at WebServiceStudio as the license allows reuse of the code there, but the code structure is simply awful. It works as is but everything is so deeply tied into the UI that it will take a feat of coding to extend it or pull out any reusable logic from it. I tried re-factoring a local copy but gave up pretty soon. So I had to work through the requirement myself.

Anyway, parsing WSDL is not easy at all. Once you get to the operation and message, you then have to wak through all schemas and their associated object tables to pick out the correct complex type corresponding to the message. This took me the best part of a few hours. Given that the user will not supply any namespaces, it gets very hard indeed. Thankfully the Schema Object Model is very decent. I used LINQ to XML for little bits such as extracting endpoint information (which also required me to resort to XPath extension methods because LINQ insisted i give it qualified names but I didnt know what the namespaces were !!) but for the schemas in WSDL, i think the SOM is better than LINQ. But I’m happy to be proved wrong!!

So I’ve got it to the point where it brings back the complex types. Now to generate sample instances from that for which I’ve got some resources. Once thats done and a simple Retriever implemented it wont be hard to put a UI on that because none of this is tied to the UI.

Thats it for now, a sort of ‘progress’ post for those waiting for the GUI for MockingBird and also to point out some of the dragons that await those who venture into the WSDL arena. Talk about ‘design by committee.. its a minefield!!

Couple of really good articles are Understanding WSDL by Aaron Skonnard (an old one, but an excellent read) and Walking the SOM by Stan Kitsis

Till next time

Merry Christmas

The Christmas season is well and truly underway and its just a couple more days to Christmas day itself. Let me take the opportunity to wish you a Merry Christmas and a Happy New Year 2009 and I hope the season is good for you wherever you are.

I know i’ve been rather quiet on the Biztalk front lately. I got my first official MCS assignment and it had nothing to do with BizTalk, but it is an interesting one nonetheless and learned some useful stuff that I should be able to share shortly. Am enjoying my first vacation @ home in ages, got to watch 2 movies in one day  ”Pirates Of the Caribbean -1 ” (again, just in preparation for seeing part 2) and “Ice Age 2″ (London feels very much like the latter).

…And of course, being a good BizTalker  , I couldnt resist getting my hands dirty building out a BizTalk 2009 Beta VM (which ended up consuming most of the weekend, but its all configured and ready to get my grubby hands on it soon).  My blog to-do list has grown pretty big and am looking forward to writing out lotsa things which I hope you will also find useful.

Have a great season.

VBUG Annual Conference 2008

VBUG, a user group that I have been involved quite a bit with over the past few years is holding its 11th annual conference on Nov 4th and 5th. Check out their website for a full color brochure of the agenda and Craig Murphy has got a short summary of the topics on his blog as well.

Lots of good topics will be presented at the conference. By the way, (shameless plug coming next ], guess who’s on the list of speakers? None other than yours truly. That’s right. I’m going to be speaking on “The Integration Landscape – BizTalk, WF and WCF“. Seeing as it comes right after the PDC, I’m hoping to present some info that will be hot off the PDC presses, let’s see how that works out.

The abstract for my session is below

Biztalk Server is Microsoft’s integration and BPM platform, WCF is the distributed communications platform for the future and WF has gained a lot of traction for implementing workflow based solutions. This raises interesting questions on whether the technologies are rivals or complementary. Do we have to choose? Can we use both together? If we do have to choose then what factors do we have to keep in mind when designing a solution? How can we leverage the WCF adapters in Biztalk 2006 R2 and where is Biztalk headed in future? In this session we will explore these issues and look at how we can leverage these technologies to build robust integration solutions. The session will include hands on demos of various integration points between Biztalk, WF and WCF. 

I’m currently at the Microsoft Services University at Redmond with a 109 MCS colleagues from all over the world and its been a great experience so far. I wont have much time to blog for the next couple of weeks and I will be going directly from this programme into the VBUG conference so its going to be really hectic but I’m looking forward to it.

If you are interested, i think spaces are still available so sign up at the site.

Powershell and Arrays as named parameters

While theres a ton of stuff on the usage of Powershell as an interactive scripting system there is not as much from a general programming perspective of a developer , at least not much that i could find today.

For example, while there is a flood of posts showing how you can manipulate the “args” system variable and how you can pass in an array on the command line by simply putting in several arguments with spaces, i was looking for a way to pass parameters into a function where 1 of the several parameters happened to be an array and then i needed to parse that array. After much trial and error, i found a solution. Here it is for any powershell newbies to benefit.

My main script file has a function like this  (very trivialised example)

function showfriends([string] $mainPerson, [string[]] $friends)
{
 Write-Host “Main person is “,$mainPerso
 Write-Host ‘friends count is : ‘ ,$friends.Count
 foreach($friend in $friends)
 {
  Write-Host $mainPerson “has a friend named:” ,$friend
  }

}

Now to call it i use the following line

showfriends -mainPerson:Smith  -friends:@(‘Jon’, ‘Joshua’)

which will then print

Smith has a friend namd Jon

Smith has a friend named Joshua

The reason i got stuck intiially was that i was trying to use foreach-object and PS insisted on prompting me to enter the objects. Another thing that got me stuck was that on a site with tutorials , the illustration of some syntax had showed the usage of [array]  which did not work for me.

Classic Menus for Office 2007

I’m one of those folk who cannot stand the new Office Ribbon. I frequently collate material from various sources into single documents for easy reference and Just remembering how to set up the page orientation and do a print preview itself is very cumbersome. So i found a tool which resets all the menus to the Office 2003 or ‘Classic Style. The article which pointed me to the tool itself is here and the shareware tool can be downloaded here.

I’m all in favor of the revamped object model and the new ways to programmatically create and access documents but IMO the UI is a travesty. But some people like it.

Anyway, I got something to get rid of that UI and i feel so much better already.

Rosario is now VSTS 2010

I first saw this article in SDTimes on VSTS 2010, and then found some official word on the subject in Brian Harry’s post titled Shining the Light on Rosario  which is a fairly detailed note (and an earlier note titled “Charting a course for Rosario” which provides even more infomation )

To quote a section from the article in SDTimes (emphasis mine)

“ The company has not committed to a release date for VSTS 2010. Dave Mendlen, director of developer tools at Microsoft, noted that it would be part of the next wave of the .NET Framework, .NET 4.0. Microsoft will also ship a standard edition of Visual Studio 2010 concurrently.

VSTS 2010, formerly known by the code name “Rosario,” expands the company’s ALM vision with more roles and fewer walls between them, effectively “democratizing” ALM, said Mendlen. Modeling plays a heavy role in how that is accomplished.”

While perhaps a bit disappointing if you were waiting for it to be out any time now (considering it was initially positioned as a rev up from the Team System component of VSTS08), it actually makes a lot of sense at least to me. There are some good resources on the net with explanations of the new features especially with regard to “baking in” of the factory approach and of course, theres an April 08 CTP of Rosario available if you want to play with it. Looking at the sheer breadth of features, i didnt think it would be possible to incorporate all that as a kind of feature pack to 2008 as it would make a substantial change in the VS IDE and I was also hoping that they would link it into the next version of the VS IDE (code named Hawaii) so we didnt have to upgrade twice in quick succession.

Check out that article. Now we just gotta wait for the PDC with bated breath for more to be revealed.

Pet Peeves 2

Ok, so one more complaint on language usage and i’ll go back to proper tech stuff, I promise.

2 things that i see very often which are more than just typos are the following:

• Using ‘extravert’ instead of extrovert: This comes up very often in posts and articles about personalities. The opposite of an ‘introverted’ personality is ‘extroverted’ not ‘extrAverted’.
• Using ‘ance’ instead of ‘ence’: This appears in lots of code and posts about the same. For example, it should be written as Persistence  (e.g Persistence Ignorance) not ‘Persistance’. In the same way its Dependence not Dependance (eg) Dependency Inversion.

Now you may not care much about these things but when i see code with this kind of writing, it goes down a couple of notches in my esteem. So, maybe i’m being nitpicky, but hey, this is my opinion.

Update: This had me in splits. I found an incoming link to this post from the Animal Shelters Directory and I was rather flabbergasted (nice old English word eh?). I wondered about the connection between this post and that site and then figured out that they probably spidered everything to do with “Pet” so the title Pet Peeves triggered it . With ‘pets’ and ‘tenants’ this must seem like a temporary shelter for animals that pay rent , or would it be a shelter for ‘principled’ (tenet ) animals?