BizTalk Message Archiving - Custom Archiving vs DTA

What is the best method to archive message data in BizTalk?

I wrote custom archiving pipeline component, and after many hours of development effort, I realized that I couldn't match the functional capabilities of the built-in DTA database.

When assessing the native the load on the system, the DTA database approach uses the End Point Manager to put a copy of the raw data into the Message Box tables necessary for TDDS to move it from there to the DTA database. This process is highly optimized, the impact is not zero, but it is pretty darn low.

Back in the dawn of time (2004) some developers turned this on in production, it bit them… and negative blogs were written.

These days, it is better understood. If you enable the applicable SQL Agent jobs and do the appropriate testing, you can get an idea of the performance cost and plan for it.

In any case, you would be hard pressed to justify the expense and overhead of custom archiving components when you get it for free.

Understanding DTA Tracking Performance Behavior

Regardless, make a pass at understanding the two options you are looking at:

Functional Area	Custom Archiving	DTA
Microsoft support	None, you are on you own	Fully supported
What does it cost if the solution fails?	High	Zero – i.e. MS Support
Training	Knowledge of File system/OS/DB or where ever they are storing it.	Documented and there are existing professional training solutions
Upgrading BizTalk	Unknown	Supported
Upgrading OS	Unknown	Supported
Upgrading SQL	Unknown	Supported
Long term archive plan	Must be developed by the dev team	Supported
Searching for data	Date time stamp and opening files to find the data or what ever custom fields you have.	Query using a graphical tool
Deployment cost (versioning, bindings, configuration)	High	Zero
Performance impact	Unknown, historically very high, I haven’t see one that is better than DTA, you must plan and test for it	Very low, you must plan and test for it
Meets customer requirements	Depends on needs	Very flexible so most of the time
Ability to view the raw message	Yes, once you find it	Yes, and you typically grab a lot of context data, so it should give you a lot of ways to search for it
Long term cost to customer	Unknown, but not zero, and typically very high	Zero
Time phased archive	Manual, must be created	Yes

Howard Edidin
4/4/2015 10:17:42 AM | Reply

Very Interesting.

The only time when using a pipeline component to archive the inbound message is for non-XML message types. This could also include JSON

Nils Ole Bäckström
4/5/2015 12:47:16 AM | Reply

I use a custom archiving component as the very first stage, to have a copy of the message, in case the pipeline fails. In some cases when a pipeline fails, BizTalk does not retain a copy in the suspended service, and the data is lost.
- Tom Canter
  6/3/2015 1:39:31 PM | Reply
  
  Nils, if BizTalk is not keeping a copy of the message, then you need to open a support ticket with the product team. This is obviously a bug and you shouldn't be spending your time working around it. Instead you should be putting it back on the product development teams lap to resolve, and think of all the other people you will be helping.

Simon Perrin
6/8/2015 7:00:45 AM | Reply

If you have a mapping in the incoming port, the DTA saved message is not the original one but the mapped. The only solution is to add a custom backup pipeline component.

Phidiax Tech Blog

Adventures in custom software and technology implementation.