VBUG: VSX & Software Factories - slides

Thanks to all who attended my talk on VSX & Software Factories at VBUG last week. It was a decent turnout considering the last minute change of venue. The slides should be available from Chris Mitchell at VBUG and I’m also posting them here on this blog for anyone else who is interested in the topic.

As I said in the meeting, I will shortly be starting a series focussing on GAT. I’ll also keep a page here with all the best links to VSX & GAT resources and this will be continually updated as i find more sites and posts with information.

Couple of really cool resources that I didn’t include on the slides are

• LearnVSXNow, a great site on CodePlex written by a VSX MVP Istvan Novak. This chap should write a book. I collated the first 10 parts of his currently 23 part series on VSX, into one document and it already stands at a whopping 120+ pages of deep dive content. He uses VS2008 but most of the discussions will apply to VS2005 as well (except for areas dealing with the VSCT file of course - where those of us in the VS2005 world still have to live with the CTC files).
• MSDN VSX Videos and webcasts  presented by Hilton Giesenow, Dylan Miles and others.
• The VSX Team Blog - Great content with regular community newsletters and so on.

With sites such as these (and many more others to be put up here shortly), i think the VSX community really rocks. Now, if only we can get more solid information on GAT in this way, the world would be a much happier place (!), ok, so thats being melodramatic

Anyway, the slides are here : Feedback appreciated.

 

 

UK SOA & BPM User Group meeting in July

A quick heads up for those of you UK/London folk who arent aware that the UK SOA & BPM user group meeting for July has been fixed for Thursday the 17th July at the Microsoft offices in Victoria, London. Theres an accomodation limit of only 40 people so if you are keen, then please sign up fast. It will be really cool to meet up and put faces to familar names from the blogosphere.

BuildCop

Code criminals beware… there are a number of ‘cops’ on patrol . :-). As if FxCop, StyleCop and MapCop weren’t enough, we now have BuildCop.

But seriously, this looks like another nifty tool for a teams arsenal. Its written by GAX Guru Jelle Druyts and his post about it is here. From the blurb on the site

BuildCop is a tool that analyzes MSBuild project files (interactively or during e.g. a daily build) according to a customizable set of rules and generates reports - e.g. is strong naming enabled, are certain project properties set correctly, is XML documentation being generated, are assembly references correct, are naming conventions respected, …

BizUnit & Perf Testing on Technet

I just came across this excellent material on Technet - the Biztalk Server Performance Optimization Guide. Whats even cooler about this paper is that it goes into some pretty good depth on how to use BizUnit in conjunction with LoadGen to do performance testing. This is really good stuff. I was just thinking how good it would be to see a step by step guide to setting up performance testing, including how to get your solution fully isolated and ready etc and along comes this beauty. Its got a complete configuration file which we can customize and use for our projects.

It gets even better wherein there is an entire section on how BizUnit can be used with the end to end BPM scenario. This is an absolute gem. I remember talking with Gar when we first setup BizUnitExtensions on how good it would be to get hold of the end to end scenarios and setup BizUnit for them and this article provides a lot of that. This, IMO, is probably one of the most valuable articles around not only for BizUnit but also because it describes an approach to testing a complete solution.

Check out the articles and happy Biztalking and testing Enjoy.

Biztalk Solution Factory

Jean-Paul Smit has released a Biztalk Solution Factory on Codeplex. His post about it is here. I joined up on another project , the Biztalk Application Factory a while ago but have been so snowed under with project work that i couldnt really get any contributions there up and running. Jean-Paul also mentions the work done on BASF and Dick Dijkstra who coordinated BASF is also part of this.

According to the blurb on Jean-Pauls blog, the BSF does the following

• Create BizTalk multi project solution structures using a wizard
• Create BizTalk single project solution, grouping artifacts into folders, using a wizard
• Guidance for adding Schemas to a project
• Guidance for adding Maps to a project
• Guidance for adding Orchestrations to a project
• Guidance for adding pipelines to a project

He’s also included source code and documentation. The doc has some screenshots of how to setup and whats available once its setup. It looks really good. I was planning on migrating my Biztalk VS Template into a GAX package and this looks like it already does the trick so i may just use this. It should also be interesting to see how it could be customized in terms of setting up project namespaces etc which my ApplySettings AddIn does. One thing i was going to do was extend my template with a wizard to allow for solutions with multiple orchestration projects and/or multiple schema projects and this tool currently does not do that but then again, that kind of partitioning may only come later in development. Still it would be a useful checkpoint when designing a solution to think about whether we need such flexibility.

Anyway, all in all a promising start. What are you waiting for? Go on and check it out !! …

CodeStyleEnforcer

You gotta love this. Its like buses. You wait for one forever and then two come along at the same time. No sooner had i finished looking through StyleCop, I happened to go over to the Visual Studio Gallery (this is different from the MSDN Code Gallery) and came across the CodeStyleEnforcer. Downloads can be done from the homepage of the tool here.

According to the blurb

“Code Style Enforcer is a DXCore plug-in for Visual Studio 2005 / 2008 that checks the code against a configurable code standard and best practices. It is developed for C#, but some of the rules will also work for VB .NET, though not tested. The code standard is currently configurable.”

To paraphrase the rest - it supports name rules, visibility rules & implementation rules. It is based on the IDesign C# coding standard which is also a freely available and a popular standard.

On taking a look at the rather grainy screenshot provided, it appears that this is a ‘pro-active’ tool and highlights issues in code as they are written. Re-sharper provides some of this kind of checking as well but of course, thats commercial whereas this is free. I’ve also been impressed with the way the DXCore system almost blankets the entire codeModel allowing you to write plugins easily. More on this later.

It also seems , from various sources on the net, that loads of people are moving away from Resharper to Refactor Pro (written by the DXCore guys, DevExpress) especially with the instability of R# 3.x. I still use R# 2.5.2 and will wait for reviews of 4.x before i stump up for an upgrade, but in general, i love the tool. Still its good to have different options. Anyway, this post is not about R#. Check out CodeStyleEnforcer and see how you like it.

StyleCop - first impressions

A friend alerted me to the fact that StyleCop (formally named Microsoft Source Analysis for C#) is now available in the public domain and can be downloaded from the MSDN Code Gallery . Its been used as an internal tool in Microsoft for a long time.

It differs from FxCop which is targeted at compiled assemblies while this looks at source code and is only aimed at C#. Information on the tool can be found here and the comments are quite interesting. It looks like a controversial tool (for example, the preference for spaces over tabs) so now we are going to have more layout wars. 

This is delivered as a VS Package so when you install it, it automatically gets loaded into the VS 2005 Tools Menu under the caption - Run Source Analysis. There is a SourceAnalysisSettingsEditor application in the Program Files directory. No shortcuts are available on the desktop.

There is a gotcha when you try to launch the editor application from that folder directly. It will immediately throw up an error message saying “The command line arguments are incorrect. Include a path to the Source Analysis Settings file.

To overcome this simply drag the “Settings.SourceAnalysis” file onto the application and it will open up correctly.  Alternatively, you can open up a command prompt at the folder and enter “SourceSettingsAnalysisEditor.exe Settings.SourceAnalysis” to launch the application. The third method, which i guess is the way it was intended to be used (as it is a VSPackage) is to simply right click on the Project and choose the Source Analysis Settings option.

Ultimately, however you choose to launch it, the main screen looks like this

I ran it on a small application and it immediately threw up at least a hundred errors - a bit like FxCop - very daunting. The interesting thing about the settings file is that there is hardly anything there. Its an XML file (arent they all?) and i was expecting to see all the options that had been selected in the application showing up there. Actually the application launches with all the default settings and once you unselect or select one of them , the settings file immediately gets updated. However we are not expected to play around with this that much. According to the blurb on the MSDN post

The ultimate goal of Source Analysis is to allow you to produce elegant, consistent code that your team members and others who view your code will find highly readable. In order to accomplish this, Source Analysis does not allow its rules to be very configurable. Source Analysis takes a one-size-fits-all approach to code style, layout, and readability rules. 

Lone Ranger developers with strong opinions on code layout (esp if they differ from these) will most probably be up in arms at some of this, but the tool is aimed at team use and when you need consistency in a big code base, tools such as these are a definite asset (of course, in some teams you gotta be prepared to start arguing about the number of spaces /tabs etc .. sigh!! you cant win ‘em all )

It is also not an extensible tool and so you cannot add your own rules to it. FxCop does allow that, but then you need a stout heart to even attempt writing custom rules for FxCop so i dont think the absence of a facility for defining custom rules is that much of a downside (in a free tool , that is.. commercial tools are a different story).

There’s also a couple of good posts on the MSDN blog documenting the following items in more detail.

• Changing the settings for a project
• Sharing settings files across multiple projects

There is also an MSBuild task available for this in the Microsoft.Sdc.Tasks library (Microsoft.Sdc.Tasks.Tools namespace) and the CHM file for the library indicates that the task has the following parameters

<Tools.StyleCop FullAnalysis=”fullAnalysis”
                            PathToAnalyze=”pathToAnalyze”
                            ViolationsFile=”violationsFile”
                            UseVSBuildFiles=”useVSBuildFiles”
                            FileToAnalyze=”fileToAnalyze”
                            OptionsFile=”optionsFile”
                            UseCoreXTFiles=”useCoreXTFiles”>
                <Output TaskParameter=”TotalViolations” ItemName=”itemName” />
            </Tools.StyleCop>

So, its nice to have this and i’m sure that now its in the open, there will be some enhancements made in due course. Perhaps someone will work out a way to plugin new rules (maybe using extension methods!!) into the existing code base or maybe we’ll get a new version if it starts getting popular. Check it out and see if it helps your team. 

 

 

 

SOA-2(b): More on the integration layer

I had discussed the design of our integration layer in an earlier post and Stuart commented asking if we implemented the messaging layer using messaging only or if we needed to implement orchestrations. As i started writing the response, it got rather long so i decided to make it a full fledged post. So here goes.

The way we designed the orchestrations was to hold composite large messages instead of multiple smaller messages and each orchestration would communicate with one or more webservices in the backend. So if we decided to, say, create a new ‘Business Document Application’, we would pass it customer information, department info etc and the orchestration would check if the customer existed using a lowlevel backend webmethod, create/update the customer, then invoke another interface to setup the document and so on. Some other orchestrations encompassed a set of ‘actions’ and data specific to them, such as ‘renew’, ‘invalidate’ etc in a single message and then called different backend services depending on the action.

Similarly search handlers acted as aggregators. Initially the searches were pass through orchestrations (and all the data was available from single webmethods in the backend) but as time has gone by some have been augmented with data from two or more sources. 

If you dont have any composition/aggregation, then schema webservices would work just as well and perhaps we could have implemented the searches in this way - but i had some ‘preview’ of future requirements and decided to put in orchs early. With Schema webservices, I have been concerned about business level validation and returning SOAP faults etc from Schema ASMX’s and havent done a lot of probing in that area so i have been rather reticent to go with those alone. 

It also depends a lot on the backend interface. For us we could use a solicit response port with a custom pipeline to set the soap operations/actions. In other implementations, for one of our backend systems we have had to call an external assembly (provided by the vendor to wrap their webservice operations) and this needs an orchestration (unless, maybe, we go down a custom adapter route and then a port would do just as well).

Schema webservices are also a good way to bring in a slim mediation layer initially and then beef it up with some orchestrations wherever necessary (if you can get the error handling sorted).

With all these of course, you must keep performance in mind and tune the receive and send portions of your solution accordingly. Some of the perf related material on MSDN and in the new deployment guide talk about settings which can help you tune the solution. We’re going to be taking  a fresh look at the system and putting into practice some of the stuff we’ve been learning in this area so i’ll post more info as we progress.

The Adapter Diaries - Oracle Adapter - 4

“Pragma” killed the Post-Poll

Bembeng Arifin commented on my second post in this series calling out something he had found when working with polling and i decided to follow up on this and in my opinion, the technique is so simple, its absolutely brilliant!!

Basically, to avoid some of the problems with duplicate data pickup when using NativeSQL with Poll and PostPoll, Bembeng had used a function in the select statement which automatically did the postpoll statement inside an autonomous transaction (PRAGMA AUTONOMOUS_TRANSACTION - hence the title of this post ).

Now I had taken an instant dislike to the NativeSQL when i used it earlier , particularly because i didnt fancy hardcoding a map with index functoids. (I think it may be possible to write a nifty little generic parser for this and maybe will try my hand at it sometime, but dont have enough time for that now). I was happy with the schema provided by the TableChangeEvent so i needed a compromise. I decided to use a view that brought back all the data from the main table where the timestamp was greater than the value in the poll_control table. My aim was that as soon as i got the data successfully i would call an update statement on the poll_control table.

My data comes in through a port map and then gets debatched by an orch (calling a pipeline ) and then the normalized ’event’ documents get processed by another set of orchs. With this disconnected model it was rather difficult to spot where to update the poll control table because we dont want too much time to lapse such that we lose records (that arrived just after the poll). I tried to create a send port subscribing to the MessageType of the event message, but calling the UPDATE statement didnt work. I got some “object reference not set” type of errors. I guess i havent really understood how to use the UPDATE statement yet.

So, the dilemma was , should i go back to nativeSQL and use Bembengs approach or should i stick it out with the UPDATE? And then I had an epiphany : Why not both together?   ( terribly dramatic aren’t I?). The solution then was to use the function inside the view.  The view code is something like

“SELECT * FROM TABLE XYZ WHERE CREATED_DATE  > (SELECT GetLastPollTime FROM DUAL)”.

“GetLastPollTime” being the function of course. This works very well. Everytime the tablechangedevent is executed, the view is called which in turn executes the function. Bembeng provides a sample of the function code in the article.

I’m really chuffed with this discovery (and i learned a fancy new thing in Oracle too - PRAGMA directives - woohoo!!). Thank you Bembeng!

A Rant about embedded ‘connection strings’

Whilst the Poll and Post-Poll are a nice feature of the Oracle adapter over the SQL adapter (except of course for the NativeSQL and the frustrations with the oddly named TableChangeEvent that i moaned about earlier), one thing i absolutely detest about the adapter is the way it embeds the connection data in the targetNamespace. (OracleDB://SERVICE_OR_DSN_NAME/SCHEMA_OWNER/TABLES/TABLE_NAME).

Bah! Humbug!. What if you have a different DSN /SERVICE (as most people do) for development and production (and other environments in between such as TEST, UAT and so on). Do you have different schemas? Perish the thought! Maintaining them would be a nightmare.

The approach which I have currently taken is to create a fairly generic service name like ‘CRMSERVICE’ in my Oracle TNSNAMES. That internally maps to the development instance. I also created a System DSN with the same name ‘CRMSERVICE’. Fortunately, the schema owner is the same in Dev and Live, otherwise i would have had to ask the DBA to create the same schema owner name in all environments. Now when i generate the schemas it uses CRMSERVICE /SCHEMAOWNER so when i move to a new environment i can just change the TNSNAMES.ORA file. (Thats the theory anyway - i shall know in a couple of days if that works out or not - currently my local connections and the DEV connections point to the same oracle instance, but not so when we move to TEST).

I really appreciated the SQL adapters approach where it let you create any namespace so you could be creative and just change the connection parameters in the binding file.

Anyway, I guess using SSO (instead of embedded credentials) in conjunction with this generic service name approach would give equal flexibility.

So, tell me, how do you handle this issue with the adapter? does the approach i am using make sense? any suggestions to make it better?

The Adapter Diaries - Oracle Adapter - 3

Strange Errors and workarounds

So, after a while, i’m back to the series. I got called out to help with another project for a few weeks so there wasnt much to write about in this space. Anyway, i’m back to the Oracle adapter project and immediately came face to face with some strange error messages and some solutions that i obtained after  much trawling through posts and search results. Here are the key issues i faced. I hope this helps you if you get the same kind of errors.

(1) PersistenceException

Excerpt from the event log

Exception thrown from: segment 1, progress 18
Inner exception: Exception occurred when persisting state to the database.
       
Exception type: PersistenceException
Source: Microsoft.XLANGs.BizTalk.Engine
Target Site: Void Commit()
The following is a stack trace that identifies the location where the exception occured

   at Microsoft.BizTalk.XLANGs.BTXEngine.BTXXlangStore.Commit()  (etc etc)

Additional error information:

        A batch item failed persistence Item-ID 091955b1-b0a4-4c2c-ad51-1739d516ce82 OperationType MAIO_CommitBatch Status -1061151998 ErrorInfo The published message could not be routed because no subscribers were found. .
       
Exception type: PersistenceItemException
Additional error information:

        Failed to publish (send) a message in the batch. This is usually because there is no one expecting to receive this message.  The error was The published message could not be routed because no subscribers were found.  with status -1061151998.
       
Exception type: PublishMessageException

Reason :  I was publishing a message from one orchestration intending it to be consumed by another. I preferred the message box route instead of partner orchestrations for a looser coupling.  The problem was a small typo in the BTS.MessageType filter of the consuming orchestration. Fixed that and all was well.

Other Links

Todd Uhl got a similar error and posted about it here.

(2) Unable to load DLL (BAUtil.dll)

Excerpt from the event log

Event Type: Error
Event Source: Microsoft BizTalk Adapters for Enterprise Applications
Event Category: None
Event ID: 0
Description:
Runtime Error: Unable to load DLL (BAUtil.dll). 

(and also other messages like : “RuntimeAgent: Error trapped in constructor: No connection could be made because the target machine actively refused it)

Reason: This is a known issue. There is a Support article here that talks about it. This issue occurs if the domain user account that is configured as the BizTalk host instance cannot read the following registry key:
HKEY_LOCAL_MACHINE\Software\Microsoft\BizTalkAdapters\Config.

When i first did my POCs with the adapter on my local machine, I had the biztalk account as a local admin. (Yeah, i know.. we shouldnt do stuff like that… but thats a legacy from the 2004 days when i tore my hair out trying to set the minimum permissions for the various accounts. IMO, developing with Least User Privelege is nice but only in theory :-). Anyway, on the ‘real’ dev system, the biztalk account is a network user and that needed to have the permissions to the specified key.

(3) Calling a webservice: The request was aborted: The request was canceled

Excerpt from the event log

“WebException: The request was aborted: The request was canceled”

Reason and Solution:

It appears that this is to do with connection Timeout mismatch on client and proxy.In my solution, i need to call a Java webservice via a custom proxy (invoked in an expression shape). Under even small loads (e.g 15 simutaneous requests) i got this error showing up and those messages got suspended. I ended up losing on average about 25% of the messages in every batch. Looking through the post from Richard Seroter and one on the MSDN forums led me to track down the issue to the HTTP connections and KeepAlive Issues. As i had not done any changes to my machine.config (so the timeout stayed at the default 90 seconds) , i decided to poke into the Tomcat webserver on the other end of the solution and the MSDN forum post led me to the Tomcat server.xml config file. (The last reply on the MSDN forums shows an excerpt from the Tomcat log)

 <!– Define a non-SSL HTTP/1.1 Connector on port 8080 –>

<!– The port and connectionTimeout was changed from the default. The connectionTimeout is set to 110 seconds to be greater than .Net 2.0 framework of 100 seconds for HTTP Keep Alives –>

<Connector port=”8000″ maxHttpHeaderSize=”8192″

maxThreads=”150″ minSpareThreads=”25″ maxSpareThreads=”75″

enableLookups=”false” redirectPort=”8443″ acceptCount=”100″

connectionTimeout=”110000″ disableUploadTimeout=”true” />

In our server the connectionTimeout had been set to 2000. Once i changed it to the value shown here, the system didnt lose any more messages. I guess i had better test it with much larger batches just to be safe.

(4) AmbiguousXPathExceptions

This was an interesting one. The system that was working seemingly smoothly suddenly threw this message indicating that there was an error in promoting a specified property in a message obtained from mapping a response from the webservice i talked about above. I knew it had to be something to do with the data since the earlier tests were all fine. It was a simple mistake in a map. In my source schema, a particular element (call it EventID) is part of a repeating group. In my target i was only interested in one EventID but had linked the nodes directly. So when the map encountered a source with 2 events, it created two nodes of my target schema which contained the promoted property and hence the error. I fixed this by using an Index Functoid to pick only the first element.

(5) bts_UpdateMsgbox_BizTalkServerApplication Deadlocks

Excerpt from the event log

The following stored procedure call failed: “
{ call [dbo].[bts_UpdateMsgbox_BizTalkServerApplication]( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}”.
SQL Server returned error string: “Transaction (Process ID 59) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.”.

Solutions

This one i couldnt solve. There’s a post here that talks about the issue. I suspect its due to my development VM running out of memory occasionally. I will keep an eye on it when i deploy to DEV and beyond and hopefully its only a transient situation.

So, thats it for the weird errors and solutions for all but one.