JOURNAL OF COMPUTING, VOLUME 5, ISSUE 4, APRIL 2013, ISSN (Online) 2151-9617 https://sites.google.com/site/journalofcomputing WWW.JOURNALOFCOMPUTING.

ORG

Complex Event Processing - A Survey

V. Govindasamy and Dr. P. Thambidurai
Abstract Complex Event Processing (CEP) deals with generating event notifications by deriving complex events from primitive events. The primitive events are from multiple distributed sources. In this paper, we present a survey of the recent publications with respect to Complex Event Processing in Distributed Event Processing Systems. A comparative study of all the major approaches are listed and discussed. Major contribution of this paper is i)Identification eight trends in CEP systems in recent years ii) Detailed description of existing approach and iii) Comparative study of existing approaches. Index TermsComplex Event Processing, Recent Work

1 INTRODUCTION
Database Management System (DBMS) is used for storing persistent data and indexing data before the process of the information. DBMS returns the result by processing the query when explicitly asked by users. Data Stream Management system (DSMS) is capable of performing detection, analysis, manipulation and storage of complex events. DSMS is developed as special DBMS. DBMS works with persistent data, where updates are infrequent; DSMS deals with transient data where updates are continuous. DBMS runs queries just once, to return 1.1 a complete answer, DSMS executes standing queries, which runs continuously and provides updated answers when new data arrives. DSMS system can handle both relational data and stream data. Windowing is a technique used to process stream data. Publisher/subscriber systems allow users to express stateless subscriptions. They are evaluated over each event that arrives at the system. With the advent of event based systems that are highly distributed, loosely coupled components, where the publisher/subscriber middleware system is responsible for interpreting event notification. Event Stream Processing (ESP) deals with continuous queries over data streams, often with a focus on high frequency events and scalability. In DSMS, the transient data are stored, while in ESP, queries are stored instead of data. The publisher events are data and subscriptions are queries on data. The subscription is matched with the published events either by topic, type or content.Complex Event Processing (CEP) [11] is about inferring patterns from continuously arriving data. The CEP engine examines continuously arriving events from various sources and detects patterns of interest and notify to the end users. Event patterns are detected by combining simple events to composite or complex events. A simple event is a set of attributes with timestamp. The complex event query or pattern is represented as a SQL like la nguage

which is transformed into CEP operators. CEP differs from DSMS in their dedication to detect event patterns rather than process streams in general. CEP systems focused on optimizing parameters, like bandwidth utilization and end-to-end latency, which were usually ignored in DSMSs. CEP systems are in relation with publisher subscriber system in terms of scalability and performance.

Features Of CEP Systems Input: The data streams are continuous, infinite, and volatile. Data Streams exhibits strong temporal qualities. Data Streams are from more than one source Queries: The queries are in real time and process large number of events. They are interested in newly arrived events and not on historical data. 1.2 Different Types of Applications The types of applications are i) Responsive Application, ii) Reactive Application, iii) Proactive Application. A responsive application is the one where a student is querying the web in order to get material needed for writing a school assignment. Many consumer and enterprise applications fall under this category where users get quote from insurance company, query a data warehouse for a summary of sales in a certain segment. A reactive application detects a traffic jam, either by a single camera observation (simple event), or by summing up the number of vehicles that enter a road segment within a certain time interval (a complex event), that reacts by changing a street light schedule. The third is an example of a proactive application: a technician is delayed at a customers house, and because there is also a traffic jam, the technician is expected to miss the next scheduled customer he should visit. While this has not happened yet, we wish to eliminate this anticipated event, by rescheduling the planning of all the technicians in a team.

V. Govindasamy is with Department of Computer Science and Engineering, Pondicherry Engineering College,Pondicherry-605011,India. Dr. P. Thambidurai is Principal of Perunthalaiver Kamarajar Institute of Technology, Karaikal-609603,Pondicherry,India

JOURNAL OF COMPUTING, VOLUME 5, ISSUE 4, APRIL 2013, ISSN (Online) 2151-9617 https://sites.google.com/site/journalofcomputing WWW.JOURNALOFCOMPUTING.ORG

1.3 Generic Architecture of CEP System

a general model for detection of events with network of sensors during disaster. Michael Olson et al [13] analyzed streams of data from large numbers of heterogeneous sensors to detect rare geospatial events.

2.3 Event Based System Cordies Cordies [16] is a distributed system for the detection of correlated events. CORDIES is designed for the operation in large-scale, heterogeneous networks. It adapts dynamically to changing network conditions. Cayuga Cayuga [19] is a newer event processing system in development at Cornell University. The system can be used to detect event patterns in event streams. The Cayuga system is designed to leverage traditional publication/subscription techniques to allow for high scalability. CoDA Context Delivery Architecture (CoDA) [20] is aimed for service oriented or event-driven applications, where the functioning of the software elements (services or event handlers) is determined by the input to the element, and the context. Fig.1
The Fig.1 highlights the generic architecture of CEP system where in EPA means Event Processing Agent. The EPA may be a Filter agent, Transformation agent or Pattern Detection Agent.

2 RECENT WORKS IN PROCESSING SYSTEMS

2.1 Rule Management

DATA

EVENT

Hannes Obweger et al[2] has proposed a rule management framework which provides distinction between infrastructure rules and respond rules in the business logic to provide flexibility and agility. The requirements for infrastructural rules are 1. 2. Expressiveness, Efficiency of use,

3. Full and system-wide access. The requirement for sense and respond rules are, 1. Decoupling, 2. 3. 4. Reusability, Ease of use, Hot deployment,

2.4 Languages TESLA TESLA [18] [1] is a CEP language. TESLA provides a high degree of expressiveness to users while keeping a simple syntax with a rigorously defined semantics. In particular, TESLA provides selection operators, parameterization, negations, aggregates, sequences, iterations, and fully customizable event selection and consumption policies. Further TESLA supports reactive and periodic rule evaluation within a common syntax. CUDA Compute Unified Device Architecture (CUDA) [8] is a general purpose parallel computing programming. It offers a new parallel programming model and instruction set for general purpose programming on Graphical Processing Units (GPUs). CUDA Content-based Matcher (CCM) algorithm has been composed of two phases: i) a constraint selection phase and, ii) a constraint evaluation and counting phase. When an event enters the engine, the first phase is used to select each attribute of the event with all the constraints having the same name. These constraints are evaluated in the second phase, using the value of the attribute. 2.5 Proactive Event Based System Proactivity refers to the ability to mitigate or eliminate undesired future events, or to identify the advantage of future opportunities by applying prediction and automated decision making technologies. We investigate an extension of the event processing conceptual model and architecture with respect to proactive event-driven applications, and propose the main building blocks of a novel architecture. We first describe several extensions to the existing event processing functionality that is required to support proactivity. In the next step, we extend the event

5. Security. Yulia Turchin et al[3], has presented a framework for automating domain expert rule specification using intelligent techniques for specification of rule parameters. A discrete Kalman filter algorithm is designed to determine and tune the parameter values.

2.2 Geospatial Distributed Event Based Systems Geospatial distributed event based systems [12] provides

JOURNAL OF COMPUTING, VOLUME 5, ISSUE 4, APRIL 2013, ISSN (Online) 2151-9617 https://sites.google.com/site/journalofcomputing WWW.JOURNALOFCOMPUTING.ORG

processing agent model to include two more type of agents: i) Predictive agents that may derive future uncertain events based on prediction models, and ii) Proactive agents that compute the best proactive action that should be taken. Vinod Muthusamy et al [4] in their paper has designed a publisher/subscriber system that will predict an event will match in future. Using these knowledge applications can take proactive steps to prevent the situation or initiate steps to react to the situation. A new algorithm based on Markov model has been designed in this regard. Yagil Engel et al [5] introduce a high level conceptual architecture for proactive event driven computing. K. R. Jayaram et al [17] proposes a event based distributed system that analyze and separates related and unrelated events.

contains a machine learning component. Segev et al [25] define the representation and semantics of event composition for probabilistic settings and has shown how to apply these extensions to the quantification of the occurrence probability of events. These results enable any active system to handle such uncertainty.

2.6 Uncertainty in Event Based System Waldemar et al [26] designed a model which captures specifics of different variants of distributed event-based systems. In particular, the model is derived from various previous publications in five sub-areas such as (1) eventdriven interaction paradigms (EDIP), (2) event stream processing (ESP), (3) complex event processing (CEP), (4) event-driven monitoring networks and wireless sensor networks (WSN), and (5) event-driven business process management (EDBPM). Alexander Artikis et al [27] describes that events may occur during an interval, or more generally inside a union of intervals. Second, the time in which the event occurred or even the fact that it occurred at all might be uncertain. Amirhossein Malekpour et al [23] proposed a publisher/Subcriber API that recovers the frequently lost messages. Diallo et al [24] the subscriber meets the requirements of the publisher. Dan OKeeffe et al *28+ proposes a mechanism to detect a missing events, that the publishers to attach sequence numbers to each event that they send. By examining the sequence numbers of events received, the mediator can determine when events sent by a publisher have been lost. Gabriella Tth et al [9] proposed for a synergy between CEP and Predictive Analysis. The Predictive Analysis part

2.7 Semantics and Context based Event Processing System Kia Teymourian et al [21] proposes huge amounts of domain background knowledge stored in external knowledge bases can be used in combination with event processing in order to achieve more knowledgeable complex event processing. The ALERT [29] system is an active collaboration platform, in which a virtual actor that interacts with the developers, processes and recognizes various kinds of interactions and suggests actions on the basis of these, thus enabling developers to work better together. 2.8. Quality Measures Scalability Ella Rabinovich et al [6], presents a novel approach for pattern rewriting that aims at efficiently processing patterns which comprise all levels of complexity A cost model based on balancing processing latency and event throughput according to user's preference is given. Pattern cost is then estimated using simulation-based techniques. SangJeong Lee et al [7], advocates for an event centric sharing approach to handle scalability issues for multiple queries and multiple event sources. They have proposed for a novel data structure Event-Centric Composable Queue (ECQ) to build a single shared network where multiple queries are evaluated. Zhaoran Wang et al [10], shows near-linear scalability on a commodity multi-core machine based on DFGS mechanism. Flexibility Naomi Seyfer et al [14] present the stream-relational languages, both visual and textual, modular abstraction benefits from type parameterization, the ability to capture types from the inputs, and the ability to manipulate unexpected fields opaquely.

3. SUMMARY OF VARIOUS TECHNIQUES

Paper Topic 1 Technique Data Set Metric Future Enhancements Throughput, Distributed event Completeness, low processing, scaprocessing time lability NA NA

Complex Event TESLA Events Processing with TREX Rule Management Separates NA framework Infrastructure rules and sense and respond rules Automated rule Discrete DARPASpecification Kalmann Intrusion

NA

Find good initial parameter values

JOURNAL OF COMPUTING, VOLUME 5, ISSUE 4, APRIL 2013, ISSN (Online) 2151-9617 https://sites.google.com/site/journalofcomputing WWW.JOURNALOFCOMPUTING.ORG

filter to determine and tune parameter values Prediction of Algorithm matching subbased on scription in future Markov model Prediction in High level event processing conceptual system architecture Pattern rewriting Policy based for agility and pattern recalability writing

detection system

Synthetic data set

Precision

To handle imprecisely defined subscription NA

NA

NA

Simulation Throughput, laten- Heuristic based cy, tradeoff approach for se,sensitivity analysis lection of rewriting alternative

10

11

Event centric Event centric sharing approach composable for scalability queue data structure Use of CUDA CUDA conarchitecture tent based matching algorithm Synergy between PA using CEP and PA machine learning Multi-Core Based DeadlineMessage Broker aware FineGrained Scheduling mechanism Complex Event XChange Processing

Synthetic data set

Processing and Nested queue storage cost, scala- monitoring bility Event processing time, Speed up Probalistic Inexat matching

NA

NA

NA

NA

Predicates Processing time, NA and event throughput, attributes resource allocation

NA

NA

12

13

14

Towards a Discip- With clusClustered Time, speed line of Geospatial tered sensors data set accuracy Distributed Event and Monte Based Systems Carlo Simulation Rapid Detection Sensor-side Simulation NA of Rare Geospatial Picking AlgoEvents: Earthrithms: quake Warning STA/LTA Applications (Short Term Average over Long Term Average) Capture Fields Stream Base NA Flexibility Modularity in a Trading SysStream-Relational tem FrameEvent Processing work Langauge

Formal, language expressiveness, query optimization and NA

NA

15

Controlled Eng-

event

vocabulary Flexibility

Schema Inheritance, Separate Compilation, Runtime Type Parameterization, Down casting in Extension Points NA

JOURNAL OF COMPUTING, VOLUME 5, ISSUE 4, APRIL 2013, ISSN (Online) 2151-9617 https://sites.google.com/site/journalofcomputing WWW.JOURNALOFCOMPUTING.ORG

16

17

18

19

20

21

22

23

24

25

26

27

28

lish Language for processing Production and network Event Processing Rules Cordies Correlation Synthetic Latency, efficiency NA :Expressive event detection data and resource concorrelation in dis- algorithm sumption tributed Program Analysis The EventJa- analysis Delay, throughput, NA for Event-based va runtime latency, spurious Distributed Sys- framework events. tems TESLA: A Formal- Event detec- attributes Expressive, flexible combine lanly Defined Event tion algoguage expresSpecification Lan- rithm based siveness with guage on automata system scalability Cayuga: A Gener- Query engine Synthetic Consume less NA al Purpose Event data memory Monitoring System Context Aware NA NA NA NA Computing and its utilization in event-based systems Fusion of BackNA Synthetic Throughput Plan based apground Knowdatasets proach, optimizaledge and Streams tion of Events Safe Autonomous Bayesian Fil- Occupancy Safety NA Transport Veter grid hicles Content-Based Message NA Reliability, NA Publish/Subscribe oriented throughput, delay Networks middleware, reduction,recovery. end-to-end method Information Over- IOA-CBR Datasets Better trade-off NA load-Aware Content-Based Routing A Model for Rea- Probabilistic Datasets Active Prediction of fusoning with Un- Reasoning ture events certain Rules in Event Composition Systems Deriving a Uni- Taxonomy Events, Fault tolerance Machine Learnfied Fault Taxclasses ing techniques onomy for Eventfor monitoring Based Systems Event Processing ProbLog EC Variables Weight, noise NA Under Uncertain- Program ty Reliable Complex NFP Detector events Correctness Event Detection for Pervasive Computing ALERT: Semantic BTS Sensors events NA NA

JOURNAL OF COMPUTING, VOLUME 5, ISSUE 4, APRIL 2013, ISSN (Online) 2151-9617 https://sites.google.com/site/journalofcomputing WWW.JOURNALOFCOMPUTING.ORG

29

event-driven collaborative platform for software development

tional conference on Distributed event-based system, Pages 125-136,ACM, 2011. Ella Rabinovich, OpherEtzion, Avigdor Gal, Pattern Rewriting framework for Event Processing Optimization, Proceedings of the 5th ACM international conference on Distributed event-based system, Pages 101-112,ACM, 2011. SangJeong Lee, Youngki Lee, Byoungjip Kim, K. SelukCandan, Yunseok Rhee, Junehwa Song, HighPerformance Composite Event Monitoring System Supporting Large Numbers of Queries and Sources, Proceedings of the 5th ACM international conference on Distributed eventbased system, Pages 137-148,ACM, 2011. Alessandro Margara, GianpaoloCugola, High Performance Content-Based Matching Using GPUs, Proceedings of the 5th ACM international conference on Distributed eventbased system,Pages 183-194,ACM, 2011. Gabriella Tth, LajosJenoFlp, L{szlVid{cs, Complex Event Processing Synergies with Predictive Analytics, Proceedings of the Fourth ACM International Conference on Distributed Event-Based Systems,Pages 95-96,ACM, 2010. Zhaoran Wang, Yu Zhang, Xiaotao Chang, Xiang Mi, Yu Wang, Kun Wang,Huazhong Yang, Pub/Sub on Stream: A Multi-Core Based Message Broker with QoS Support, Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems,Pages 127-138,ACM,2012. Michael Eckert and FrancoisBry, Complex Event Processing (CEP), Informatik-Spektrum, Volume 32, Issue 2, pages 163-167, Springer, 2009. Annie Liu, Michael Olson, Julian Bunn, Towards a Discipline of Geospatial Distributed Event Based Systems, Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems,Pages 95-106,ACM,2012. Michael Olson, Annie Liu, Matthew Faulkner, Rapid Detection of Rare Geospatial Events: Earthquake Warning Applications, Proceedings of the 5th ACM international conference on Distributed event-based system,Pages 89100,ACM, 2011. Naomi Seyfer, Richard Tibbetts, Nathaniel Mishkin, Capture Fields Modularity in a Stream-Relational Event Processing Langauge, Proceedings of the 5th ACM international conference on Distributed event-based system,Pages 15-22,ACM,2011. Mark H. Linehan, Sylvain Dehors, Ella Rabinovich, Fabiana Fournier, Controlled English Language for Production and Event Processing Rules roceedings of the 5th ACM international conference on Distributed event-based system,Pages 149-158,ACM, 2011. Gerald G. Koch, Boris Koldehofe, Kurt Rothermel, Cordies: Expressive event correlation in distributed systems, Proceedings of the Fourth ACM International Conference on Distributed Event-Based Systems,Pages 26-37,ACM, 2010. K. R. Jayaram, Patrick Eugster, Program Analysis for Event-based Distributed Systems, Proceedings of the 5th ACM international conference on Distributed event-based system,Pages 113-124, ACM, 2011. GianpaoloCugola, Alessandro Margara, TESLA: A Formally Defined Event Specification Language, Proceedings of

4. DISCUSSIONS AND CONCLUSION In this paper, we have examined the research area of CEP and other related works such as DSMS, ESP, from which the CEP has been derived from. In addition, we have studied various techniques, languages used by various researches. We have discussed in detail how they were implemented in CEP. In conclusion, it is evident that this area is going to play a vital role in future in the fields of software engineering, software maintenance and IT operations. Rule Management [2] provides a framework of tools and workflows for modeling processing logic and business logic based on a unified rule-evaluation model. Also the degree of control and complexity is adjusted to the needs and skills of the different user groups in an enterprise. Training [3] is beneficial and continuous tuning can compensate for changes in dynamic environments. Automatic tuning becomes beneficial in settings where manual tuning of a complex set of parameters becomes a cognitive challenge. The algorithms [12] for rapid detection of geospatial events which is used on Cloud computing architectures and many servers collaborate to detect events by analyzing data streams from large numbers of sensors. It [13] is a community-based event detection system, in which individual members of the community can install and operate sensors and responders. It can help society to deal with disasters collectively.The event based systems [16, 19, 20] has been developed to detect the correlated events based on input and output. Also languages [1, 8, and 18] provide better expressiveness and computing process of events.

[6]

[7]

[8]

[9]

[10]

[11]

[12]

[13]

REFERENCES
[1] GianpaoloCugola, Alessandro Margara, Complex Event Processing with T-REX, Journal of Systems and Software, Volume 85 Issue 8, Pages 1709-1728, ACM, 2012, HannesObweger, Josef Schiefer, SzabolcsRozsnyai, UserOriented Rule Management for Event-Based Applications,Proceedings of the 5th ACM international conference on Distributed event-based system , Pages 39-48, ACM, 2011. YuliaTurchin, Avigdor Gal, SegevWasserkrug, Tuning Complex Event Processing Rules using the Prediction Correction Paradigm, Proceedings of the Third ACM International Conference on Distributed Event-Based Systems, Article No. 10, ACM, 2009. VinodMuthusamy, Haifeng Liu and Hans-Arno Jacobsen, Predictive Publish/Subscribe Matching, Proceedings of the Fourth ACM International Conference on Distributed Event-Based Systems, Pages 14-25, ACM, 2010. Yagil Engel, OpherEtzion, Towards Proactive EventDriven Computing, Proceedings of the 5th ACM interna-

[14]

[2]

[15]

[3]

[16]

[4]

[17]

[5]

[18]

JOURNAL OF COMPUTING, VOLUME 5, ISSUE 4, APRIL 2013, ISSN (Online) 2151-9617 https://sites.google.com/site/journalofcomputing WWW.JOURNALOFCOMPUTING.ORG

[19]

[20]

[21]

[22]

[23]

[24]

[25]

[26]

[27]

[28]

[29]

the Fourth ACM International Conference on Distributed Event-Based Systems,Pages 50-61,ACM, 2010. Alan Demers, Johannes Gehrke, Biswanath Panda, MirekRiedewald, Varun Sharma, and Walker White, Cayuga: A General Purpose Event Monitoring System, ACM, 2007. OpherEtzion, YonitMagid, Ella Rabinovich, Inna Skarbovsky, NirZolotorevsky,, Context Aware Computing and its utilization in event-based systems, Proceedings of the Fourth ACM International Conference on Distributed Event-Based Systems,Pages 270-281,ACM, 2010. Kia Teymourian, Malte Rohde and Adrian Paschke, Fusion of Background Knowledge and Streams of Events, Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems,Pages 302-313,ACM, 2012. Christian Kuka, Sebastian Gerwinn, Daniela Nicklas, Demo: Context-Model Generation for Safe Autonomous Transport Vehicles, Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems,Pages 365-366,ACM,2012. AmirhosseinMalekpour, Antonio Carzaniga, Fernando Pedone, End-to-End Reliability for Best-Effort Content-Based Publish/Subscribe Networks, Proceedings of the 5th ACM international conference on Distributed event-based system,Pages 207-218, ACM, 2011. Mohamed Diallo, Serge Fdida, IOA CBR:Information Overload Aware Content-Based Routing, Proceedings of the Fourth ACM International Conference on Distributed Event-Based Systems,Pages 85-86,ACM, 2010. SegevWasserkrug, Avigdor Gal, OpherEtzion, A Model for Reasoning with Uncertain Rules in Event Composition Systems, proceeding of: UAI '05, Proceedings of the 21st Conference in Uncertainty in Artificial Intelligence,IBM. Waldemar Hummer, Christian Inzinger, Philipp Leitner, Benjamin SatzgerSchahramDustdar, Deriving a Unified Fault Taxonomy for Event-Based Systems, Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems,Pages 167-178,ACM, 2012. Alexander Artikis, OpherEtzion, Zohar Feldman, Fabiana Fournier, Tutorial: Event Processing Under Uncertainty, Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems,Pages 32-43,ACM, 2012. Dan OKeeffe, Reliable Complex Event Detection for Pervasive Computing, Proceedings of the Fourth ACM International Conference on Distributed Event-Based Systems,Pages 73-84,ACM, 2010. LjiljanaStojanovic, SinanSen, Jun Ma, DominikRiemer, ALERT: Semantic event driven collaborative platform for software development,Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems, Pages 385-386, ACM, 2012.

Alagappa University, Karaikudi in 1995. He is Fellow Institute of Engineers, Life Member in CSI and ISTE. He has published more than hundred research papers in National and International Journals and Conferences. His area of interest includes Networking, Image Processing, Natural Language Processing, and Information Security.

V. Govindasamy is a research scholar in Department of CSE, Pondicherry Engineering College, Puducherry, India. He is currently working as Assistant Professor in Department of IT, Pondicherry Engineering College, Puducherry, India. Dr. P. Thambidurai is at present working as Principal and Professor of Computer Science & Engineering, Perunthalaivar Kamarajar Institute of Engineering and Technology (PKIET), Karaikal, Union Territory of Puducherry, India. He has completed his M.E.(CSE) from Anna University, Chennai in 1984 and Ph.D.(CSE) from