irfanuruchi/distributed-orchestrator-worker repository overview

⁠Distributed Orchestrator Worker

This is the worker image for my Capstone Project, Distributed Task Orchestration System.

The worker is the part of the system that actually runs the tasks. It starts a gRPC server, registers itself to the coordinator and waits for tasks. When the coordinator sends a task, the worker runs the matching Python script and returns the result.

The same worker image is used for all workers. The role of each worker is changed through environment variables, mainly WORKER_CAPABILITIES.

⁠Docker image

docker pull irfanuruchi/distributed-orchestrator-worker:v10

⁠Worker capabilities

In version 10, each worker can tell the coordinator what type of work it supports.

Example:

WORKER_CAPABILITIES=GENERAL,CPU

The setup I used for testing was:

This means CPU tasks can run on worker-50051 and worker-50053, while memory tasks can run on worker-50052 and worker-50053.

⁠Run workers locally

CPU worker:

docker run -d --name worker-50051 --network orchestrator-net -e WORKER_HOST=worker-50051 -e WORKER_PORT=50051 -e WORKER_CAPABILITIES=GENERAL,CPU -e COORDINATOR_URL=http://orchestrator-coordinator:8000 irfanuruchi/distributed-orchestrator-worker:v10

Memory worker:

docker run -d --name worker-50052 --network orchestrator-net -e WORKER_HOST=worker-50052 -e WORKER_PORT=50052 -e WORKER_CAPABILITIES=GENERAL,MEMORY -e COORDINATOR_URL=http://orchestrator-coordinator:8000 irfanuruchi/distributed-orchestrator-worker:v10

Mixed worker:

docker run -d --name worker-50053 --network orchestrator-net -e WORKER_HOST=worker-50053 -e WORKER_PORT=50053 -e WORKER_CAPABILITIES=GENERAL,CPU,MEMORY -e COORDINATOR_URL=http://orchestrator-coordinator:8000 irfanuruchi/distributed-orchestrator-worker:v10

⁠Running worker from another device

The worker does not have to run on the same machine as the coordinator. It can also run from another computer on the same LAN.

Example:

docker run -d --name worker-remote -e WORKER_HOST=192.168.1.60 -e WORKER_PORT=50051 -e WORKER_CAPABILITIES=GENERAL,CPU -e COORDINATOR_URL=http://192.168.1.50:8000 -p 50051:50051 irfanuruchi/distributed-orchestrator-worker:v10

It can also work through Tailscale or another private VPN by using the coordinator machine's VPN IP in COORDINATOR_URL.

Example:

COORDINATOR_URL=http://100.x.x.x:8000

⁠What this image does

The worker registers itself to the coordinator, reports CPU and memory usage through the health check method and executes task scripts through gRPC.

The supported task scripts are:

quick_check
cpu_benchmark
cpu_heavy
data_processing
stress_test

If the coordinator restarts, the worker keeps trying to register again in the background. Because of this, the worker can reconnect without manually restarting the container.

Current version: V10
Status: working Dockerized release