Abstract
We introduce Agilis—a lightweight collaborative event processing platform that can be deployed in a Semantic Room to facilitate sharing and correlating event data generated in real time by multiple widely distributed sources. Agilis aims to balance simplicity of use and robustness on the one hand, and scalable performance in large-scale settings on the other. To this end, Agilis is built upon the open source Hadoop’s MapReduce infrastructure augmented with a RAM-based data store and several locality-oriented optimizations to improve responsiveness and reduce overhead. The processing logic is specified in a flexible high-level language, called Jaql, which supports data flows and SQL-like query constructs. We demonstrate the versatility of the Agilis framework as well as its utility for collaborative attack detection by showing how it can be leveraged in the following two attack scenarios: stealthy inter-domain port scanning, and a botnet-driven HTTP session hijacking attack. We evaluate the performance of Agilis in both these scenarios and, in the case of inter-domain port scanning, compare it to Semantic Room, which deploys the centralized high-end event processing system called Esper. Our results show that while Agilis is slower than Esper in a local area network, its relative performance improves substantially as we move toward larger scale distributed deployments.