Aggregation is the act of taking or receiving data from multiple systems and inserting into one. For example, customer data integration could reside in three different systems, and a data analyst might want to generate a report which uses data from all of them. One could create a daily migration from each of those systems to a data repository and then query that against that database. But then there would be another database to keep track of and keep synchronized.
In addition, as things change in the three other systems, the data repository would have to be constantly kept up to date. Another downside is that the data would be a day old, so for real-time reports, the analyst would have to either initiate the migrations manually or wait another day. One could set up three broadcast applications, achieving a situation where the reporting database is always up to date with the most recent changes in each of the systems. But there would still be a need to maintain this database which only stores replicated data so that it can be queried every so often. In addition, there will be a number of wasted API calls to ensure that the database is always up to X minutes from reality.
This is where the aggregation pattern comes into play. If you build an application, or use one of our templates that is built on it, you will notice that you can on demand query multiple systems, merge the data set, and do as you please with it.
For example, you can build an integration app which queries the various systems, merges the data and then produces a report. This way you avoid having a separate database and you can have the report arrive in a format like .csv or the format of your choice. You can place the report in the location where reports are stored directly.
Why is aggregation valuable?
The aggregation pattern derives its value from allowing you to extract and process data from multiple systems in one united application. This means that the data is up to date at the time that you need it, does not get replicated, and can be processed or merged to produce the dataset you want.
When is aggregation useful?
The aggregation pattern is valuable if you are creating orchestration APIs to “modernize” legacy systems, especially when you are creating an API which gets data from multiple systems, and then processes it into one response. Another use case is creating reports or dashboards that pull data from multiple systems and create an experience with that data.
Finally, you may have systems that you use for compliance or auditing purposes which need to have related data from multiple systems. The aggregation pattern is helpful in ensuring that your compliance data lives in one system but can be the amalgamation of relevant data from multiple systems. You can therefore reduce the amount of learning that needs to take place across the various systems to ensure you have visibility into what is going on.