DMQueue: an efficient dynamic heuristic : a strategy for task scheduling in application integration platforms to handle large volume of data in message-based application integration

Abstract

Integration platforms are tools deployed locally or in the cloud that software engineers use for designing, implementing, executing and monitoring integration processes that process data retrieved from remote third parties. Typically, at the heart of an integration platform is a runtime system that processes data on demand, that is, as it arrives. Small arriving rates of the order of 1 message/second can be processed without worries about efficiency or resource exhaustion. However, the continuous expansion and technological transformation of companies’ ecosystem has resulted in the generation of large volumes of data of the order of 100 messages/ second that runtime systems are not capable of processing efficiently unless they are assisted by adaptation mechanisms. Examples of sources that produce large volume of data are IoT agents and web services that collect events, for example, business events. Computational efficiency measures the relationship between the degree of performance and the amount of computing resources consumed. In this dissertation we argue that runtime systems need to be assisted by adaptation mechanisms that enable them to process large volumes of data efficiently and without increasing the amount of computational resources consumed. To support our argument, we have designed, implemented and experimented with DMQueue heuristic. We have implemented DMQueue in Java to assist the runtime systems of integration platforms that follow the task-based model. Its salient feature is that it dynamically calculates the optimal number of threads needed to process the incoming data under the consideration of the amount of computational processing resources available. We also designed and implemented a new architecture for the runtime system of the integration platform, where tasks from queues are run in parallel by threads from local thread pools. To validate our solution, we conducted experiments with data inputs that exhibit six different load swing patterns. The statistical results that we collected demonstrate that DMQueue heuristic provides greater efficiency to the runtime systems of integration platforms. The reason for that is that DMQueue has a frequent monitoring period that adjusts the number of threads in reaction to the arriving workload. The efficiency that DMQueue achieves results from the combination of several properties: we designed it to operate with multicore processors, elastic thread pool configuration and dynamic thread pool creation. We have implemented DMQueue to manage thread pool by mapping, to conduct task complexity analysis and to work with local pools. The statistical results confirm our research hypothesis that: DMQueue is able to provide the Guaraná integration platform runtime system with efficiency, performance and dynamic adaptation to the increasing volume of data input. It is also worth noting that the heuristic DMQueue can be deployed on task-based model integration platforms both on-premises and in the cloud.