BizTalk + ESB Toolkit–flat file debatching

 

 

Problem (identfied with: BizTalk 2010 / ESB Toolkit 2.1 )

The ESB Toolkit is a set of components and artefacts which provide Enterprise service bus functionality on top of BizTalk. One of the capabilities of BizTalk is debatching (Xml, Flat files).

During debatching a single message is debatched into several messages (e.g. an OrderBatch containing multiple orders). The default behavior is that each message will be placed onto the messagebox for additional processing. In a business context it is desired that the entire batch is processed before the flat file is removed, thus Recoverable Interchange Processing should be set to FALSE.

 

In an ESB Toolkit context it is possible to follow the same approach, but not only debatch the message, but also start of a new process using an itinerary. This means that each individual messages leads to a new process.

 

In this case, the disassembler and the ESB Dispatcher are required to interoperate with eachother. This behavior is however, not supported and will result in the following error:

 

Reason: Cannot access a disposed object”. Object name: ‘DataReader’.

clip_image002[7]

 

http://www.richardhallgren.com/streaming-pipeline-and-using-resourcetracker-to-avoid-disposed-objects/

Solution

There are multiple approaches to circumvent this problem:

·         Customize the default Flat file adapter / ESB Toolkit components

o   This is an unsupported approach, complex and

·         Use InterchangeProcessing = TRUE

o   This would mean that messages are handled on an individual basis, which prevents that there is a dependency on the entire stream.

o   The drawback of this is that messages that fail can not be traced back to the entire batch

·         Use a 2 step appoach

o   First the flat file is debatched

o   In the second step the Itinerary is assigned

How-it-works.

·         Use a 2 step appoach (instead of 1 step)

clip_image004[4]

·         In step 1a, the flat file is debatched in a pipeline that contains a flat file disassembler (default pipeline component)

·         In step 1b, the debatched file files are send to an intermediate location, awaiting processing

·         In step 2, the Itinerary is assigned using a pipeline with an ESB Itinerary selector and ESB Dispatcher pipeline component (default pipeline)

 

HTH,

Sander Nefs

Comments

Post a Comment

Popular posts from this blog

Azure implementation guidelines

In My post ‘ Service Bus Management ’ I pointed out a way of implementing a DTAP strategy for managing the Service Bus environment. There’s now a great post available from Microsoft which goes into more detail about other aspects of managing the Azure subscriptions and other artefacts which is quite useful when setting up your Azure environments. Follow this link to the ‘Azure Implementation Guidelines’ HTH, Sander
Read more

UK Connected Systems User Group – BizTalk Services questions

This Wednesday I’m attending the CSUG – The Hybrid Organisation (sold out!). During my evaluation of several Integration products (BizTalk / BizTalk Services / Neuron ESB / MuleESB) I had some concerns/questions on BizTalk Services. My goal is to get answers on the following scenario a) I receive a HTTP Post with an Xml Body payload b) This contains a signature which i need to validate c) Transform the Xml body payload to another Xml message d) Perform a HTTP Post with the new Xml message to an external HTTP url and sign the message e) Route the response back, similar to the aforementioned steps, where an error needs to be mapped to a dedicated error response Challenges I'm wondering how to cope with some of the following challenges in this scenario, and would like to know resource information available which addresses some of these challenges; Validating / Signing of messages A requirement is to validate the incoming message by inspecting an signature in the message
Read more

Setting up a build server with the BizTalk Deployment Framework

Image
This chapter describes the steps to a TFS Build Server to set up BizTalk deployment with the following functionality: • Build solution • Perform unit testing Using BizUnit • Create Deployment Packages Using BDTF • Deploy to Server Using R packages created in previous step • Create documentation BizTalkDocumenter For this the following steps are required: • Installation Components (30 min) • Configuring a build account (30 min) • Setting up the build service (30 min) • Scrum template installation (30 min) • Share for the build output directory (10 min) • Adjustments Build Template for use BTDF (30 min) • Design of build definitions (60 min) • Setting BTDF (60 min) • Use BTDF • Build execution / scheduling UPDATE (21-06-2013) - When using sharedassemblies, the following URL is a big help http://mohamedradwan.wordpress.com/2013/01/31/set-custom-assemblies-path-for-host-build-service-on-the-cloud/ - For viewing the basic instructions a video is available on the following blog http:
Read more