Priority-Based Scheduling Techniques for a Multitenant Stream Processing Platform

Apache Storm is a distributed stream processing system which is heavily used by the key players in the industry including YAHOO and Twitter. An application running in Storm is called a topology that is characterized by a Directed Acyclic Graph. Isolation Scheduler, the default scheduler for a multitenant storm platform running multiple topologies assigns resources to topologies based on static resource configuration information and does not provide any means to prioritize topologies based on their business significances. One of the problems with this scheduler is that performance degradation, even complete starvation of topologies is possible on a resource constrained cluster. Two priority based resource scheduling strategies are proposed in this thesis to overcome these problems. A performance analysis based on prototyping and measurements is conducted to demonstrate the effectiveness of the proposed techniques. A comprehensive analysis of the results leading to key insights into system behavior and performance is presented.