Friday, March 05, 2010

Integrations: In-line versus Queue/Batch Based

I love web services. They make it so that one system can send and receive data from some completely different system from anywhere over the web. Data gets updated, end users can see the latest and greatest data essentially instantaneously. With CRM plugins you can have in-line integrations that just push data from CRM to whatever system you want on a save, reassignment, deletion, etc. Or, you could pull data from another system whenever a page loaded, all through plugins. That's awesome.

Queue, or batch, integrations take note of everything that changes and then runs a process at a given time interval and pushes all of the data that has been queued up into the other system. Web services can be used to push/pull data in this data from the queue in batch fashion and keep systems in sync. But typically when these batches are run they put a load on the server and could be rather resource intensive. And data isn't updated instaneously, it gets updated shortly after the predetermined time interval. If this is nightly that might be too stale for some people. Depending on the data though, that may be more than sufficient.

However, when those web services don't work (when either server goes down, gets overloaded, etc) it's REALLY bad. Things either get out of sync, or else your integration just got a whole new level of complexity; waiting, or holding the data, until the other system comes back online. In the debate of in-line versus queue based integrations, queue based gets a point here. If the server is down, the queue based system gracefully errors, noting that the remote server is down, and the next time it runs it picks up where it left off and tries again. In-line dies and doesn't get the data transimitted across. The best it can do is error and put its data into a record to get queued up and processed at a later time...via a queue based integration.

I have recently worked with a system that went down at least twice a day. Originally it was thought that an in-line integration would work just fine. But since the receiving server has been down I now have to include or move to a queue based integration in case the in-line goes down.

Each has their use, and I want the in-line to work because in my mind it's the best. But in reality, the queue based, though "slower", is the most reliable.

If any one has any thoughts or input, we'd love to hear it.

David Fronk
Dynamic Methods Inc.

1 comment:

Josh Painter said...

We use custom workflow activities for our "queue." When a record is updated in CRM, a custom workflow activity runs that pushes that record data to a few external systems. If those systems are down or anything else bad happens, an exception is thrown from the workflow activity and the workflow is paused. Once the problem is fixed, just go back and resume all of your "Waiting" workflows and everything gets caught back up. You could even automate this Resume step, but we haven't found that necessary yet. Good post!

Post a Comment