TaskExecutionDemo

Class Overview

Demo for execution of task Dag using primitives provided by Helix. This demo sets up a Dag of tasks for providing analytics for impression and click events. Each node on the dag has an id and declares the desired parallelism and IDs of the nodes it depends on. When we submit this dag for execution using submitDag() TaskCluster, we create a Helix resource for each node with number of partitions based on desired parallelism. We use a "OnlineOffline" state model. The demo starts NUM_WORKERS workers. Each worker is given a TaskFactory and TaskResultStore. Each worker is assigned a bunch of task partitions by Helix and gets state transition messages for the task partitions it is assigned. When the worker gets a state transition message for a task partition, it first checks if all upstream dependencies are satisfied by making sure that corresponding partitions have transitioned to "Online" state. It then creates a Task object using TaskFactory based on resource name (since task IDs have been mapped to resource names) The demo has the following steps

1. Start zookeeper
2. Setup task cluster TaskCluster
3. Start Helix controller
4. Populate dummy impression and click data
5. Start workers
6. Submit dag

Summary

[Expand] Inherited Methods
From class java.lang.Object Object clone() boolean equals(Object arg0) void finalize() final Class<?> getClass() int hashCode() final void notify() final void notifyAll() String toString() final void wait() final void wait(long arg0, int arg1) final void wait(long arg0)