Translating integration requirements: notification pattern

In this blog post I want to highlight a very simple yet powerful integration pattern: the notification pattern. The reason why it's not often used is because it does not map requirements one-to-one (and let’s be honest, it's also more complex than a simple point-to-point connection).

A fictional – but very representative – discussion:

Architect:

Let’s discuss the integration with our new customer master data system.

Analyst:

Yeah, it looks real simple to me. A simple web service should do the trick, don’t you think?

Architect:

Well, maybe. But I thought that we’re talking about quite a lot of traffic. Do you have some estimates?

Analyst:

I’ve checked this yesterday and since customer information is needed in a lot of use cases, we’re talking about 100 k messages/hour.

Architect:

Hmmm, seems like a performance risk, also given the fact that the new customer master data system is not yet scaled for a heavy load. What about reversing the dependency?

Analyst:

How do you mean? We’re the client of the master data system so we need to get data from that system, I don’t see a lot of possibilities.

Architect:

Well, on a functional level you’re right, but on a technical level there are different patterns that fill this need. I’m thinking about a local cache of customer information. Updates to customer information are not possible from our system, right?

Analyst:

That question came up during our workshops, but we agreed that editing should happen through the master data system GUI. But … wait a minute! A local cache will not have the latest updates! And you know our business, they always need the latest information!

Architect:

Challenging that need is a topic for another discussion . But for now, let’s go ahead with this. It does complicate things a bit, so let me think …

Architect:

… The best solution would be to combine the local cache with a notification pattern in the customer master data system. Not only will this serve the real time publication of changes, but it may also be a reusable solution for future integrations. Let me explain.

Notification pattern explained

As usual, a schema is more clear than a 1000 words:

The data provider system provides change functions and some interfaces to retrieve the data (what is commonly referred to as “pull” behavior because the clients are responsible for pulling the information out of the master system). So far so good. But when implementing the notification pattern, an extra “push” flow is added to the picture. When there are changes, the provider system pushes data out to any interested party.

However – and here comes the catch! – as stated in the architect-analyst discussion, this kind reverses the dependency. And here lies the danger! We have to avoid the situation where the provider needs to have knowledge of all possible clients and needs to be updated in relation to every new client or changing requirement in any existing client system. This can/should be done by minimizing the data sent by the notification flow to: - An identifier of the updated data - An indication of the change (add – update – delete – etc.)

Simple but effective! It is then again up to each client system to decide how to act upon the notification, putting the responsibility again on the client systems:

  • Immediately retrieve the actual data through the APIs that get (and store it locally).
  • Piggyback some requests and get the data in bulk.
  • Only get the updated data when required by their clients.
  • Ignore certain updates because they are irrelevant.
  • And so on.

Takeaway

I hope by now you are convinced of the added value of the notification pattern, and also are aware of the biggest pitfall when implementing it. Do take it into account when real time master data requirements are combined with high message volumes!

Vincent Verstraete