The Aggregation data integration pattern is the act of taking or receiving data from multiple systems and consolidating them into one. For example, customer transaction data could reside in three different systems, and a data analyst would like to generate a report which uses data from all of them. One solution could be to create a daily migration process from each system to a data repository and then perform an aggregation query against that database. But, then we would need to maintain this aggregate structure and maintain its synchronizations.
An additional downside is that the data would be a snapshot that is a day old. Thus, for today’s real-time numbers, the analyst would have to manually initiate the migrations or wait another day. Or, they could set up three broadcast applications, where the aggregate database is always synchronized with the most recent changes in each of the systems. But, once again we would need to maintain this database which only stores replicated data so that it can be queried every so often.
This is the scenario that depicts the advantages of the aggregation pattern. If you build an application, or use one of our templates that is built on it, you will notice that you can on demand query multiple systems, merge the data set, and do as you please with it.
For example, you can build an integration app which queries the various systems, merges the data and then produces a report. This way you avoid having a separate database and you can have the report arrive in a format like .csv or the format of your choice. You can place the report in the location where reports are stored directly.
Value of Aggregation
The aggregation pattern derives its value from allowing you to extract and process data from multiple systems in one united application. This means that the data is up to date at the time that you need it, does not get replicated, and can be processed or merged to produce the dataset you want.
Usefulness of Aggregation
The aggregation pattern is valuable if you are creating orchestration APIs to “modernize” legacy systems, especially when you are creating an API which gets data from multiple systems, and then processes it into one response. Another use case is creating reports or dashboards that pull data from multiple systems and create an experience with that data.
Finally, you may have systems that you use for compliance or auditing purposes which need to have related data from multiple systems. The aggregation pattern is helpful in ensuring that your compliance data lives in one system but can be the amalgamation of relevant data from multiple systems. You can therefore reduce the amount of learning that needs to take place across the various systems to ensure you have visibility into what is going on.