sapiosciences/sapio_worker repository overview

⁠Sapio Worker

This serves as a base image for the Sapio Worker. The worker is a Java application that connects to a Sapio Server to extend functionality through Sapio Sciences⁠ Java API.

The primary use case for the Sapio Worker is to allow customers who don't control the server to extend the functionality of the server in the same way that Sapio Sciences does. This is done by creating plugins that are loaded into the worker and are then bound to the server through Dirmi and Apache Fory for low overhead communication. The other way to extend the server is through the Sapio Sciences Webhook API.

⁠Docker Structure

Multiple JVMs are installed in the container to support multiple versions of Java. On startup the docker bootstrap will pick the closest version of Java as the Sapio Server to avoid potential serialization issues.

To add your plugins to the worker, you want to extend this image and add your plugins to the /opt/deployment/plugins directory. The worker will scan this directory for jars containing plugins and load them into the worker JVM.

Shared libraries can be added to the /opt/deployment/sharedlib directory. All jars in this directory will be added to the classpath of a parent ClassLoader that is shared between all plugins. This is useful for sharing common libraries between plugins. This functionality is identical to the applib directory in the Sapio Server. You can also have plugin manifest classpath reference libraries present in the /opt/deployment/lib directory.

⁠Required Configurations

• VELOXWORKER_WEBSERVICE_URL (String):
  • The Sapio Platform webservice that the worker will use during startup to configure and negotiate the dirmi connection to the same server
  • Value set should end with /webservice/api
  • Example Value: https://localhost:8443/webservice/api
• VELOXWORKER_AUTH (String)
  • An API key that is typically generated by the server to identify and authorize this worker to interact over the API
  • Example Value: V2VsY29tZSB0byB0aGUgSmF2YSBab25lLCBsZXQgeW91ciBwcm9ncmFtcyBzb2FyLApJbiB0aGUgSmF2YSBab25lLCB3ZSBjb2RlIGZvcmV2ZXJtb3Jl

⁠Optional Configurations

• VELOXWORKER_WEBSERVICE_TRUSTING (Boolean):

  • Default Value: false
  • If set to true the worker will trust all HTTPS certificates presented by the server. This is useful for testing but should not be used in production as it's susceptible to MITM.

• VELOXWORKER_DEFAULT_RMI_HOSTS (String):

  • Default Value: ``
  • A comma-separated list of hosts that the worker will attempt to connect to establish a dirmi connection. If the server RMI socket is behind a load balancer or proxy this should be set to the hostname of the load balancer or proxy. After attempting to connect to the hosts in this list the worker will attempt to connect to the hosts returned by the webservice.

• VELOXWORKER_DEFAULT_RMI_PORT (Integer):

  • Default Value: ``
  • The port that the worker will attempt to connect to establish a dirmi connection. If the server RMI socket is behind a load balancer or proxy this should be set to the port of the load balancer or proxy. After attempting to connect to the hosts in this list the worker will attempt to connect to the hosts returned by the webservice.

• VELOXWORKER_SERVICE_API_ENABLED (Boolean):

  • Default Value: true
  • If set to true the worker will expose a REST API for interacting with the worker service.

• VELOXWORKER_SERVICE_API_HOST (String):

  • Default Value: 0.0.0.0
  • The host that the worker service API will bind to
  • Example Value: 0.0.0.0. Will bind to all hosts

• VELOXWORKER_SERVICE_API_PORT (Integer):

  • Default Value: 8088
  • The port that the worker service API will bind to

⁠OpenTelemetry Auto-Instrumentation

The image ships with the OpenTelemetry Java agent at:

/opt/sapiosciences/worker/opentelemetry-javaagent.jar

The Sapio Velox instrumentation extension sidecar is also included at:

/opt/sapiosciences/worker/lib/velox_telemetry_agent-all.jar

It is intentionally not enabled by default. To auto-instrument the worker JVM at boot, set VELOXWORKER_JVM_ARGS so the boot process passes -javaagent: to the worker process it spawns. For example:

VELOXWORKER_JVM_ARGS=-javaagent:/opt/sapiosciences/worker/opentelemetry-javaagent.jar

Standard OpenTelemetry environment variables (e.g. OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_RESOURCE_ATTRIBUTES, OTEL_TRACES_EXPORTER, and OTEL_METRICS_EXPORTER=otlp) can then be used to configure the agent. In SAPIO 26.8+, with metrics export enabled and VELOXWORKER_REMOTE_WORKER_METRICS_OTEL_ENABLED=true, the worker publishes remote-worker lifecycle operation duration/count and the local active-executor gauge; the server publishes registered-worker capacity using the same bounded app identity.

⁠Autoscaling and graceful drain (SAPIO 26.8+)

The boot process and the remote worker run in separate JVMs. The status API controls the child JVM through a local stdin/stdout channel so that request admission and the in-flight executor count are authoritative.

• GET /status/healthcheck is a liveness check. It remains 200 while a manually drained worker finishes its existing work, so existing target-group health checks are not disrupted.
• GET /status/readiness returns 200 only while the worker is accepting new plugin requests. It returns 503 after drain begins.
• GET /status/drain returns {"state":"ACCEPTING|DRAINING|DRAINED|STOPPING","activeExecutors":N}.
• POST /status/drain prevents all new plugin executors and leaves the worker drained until it receives SIGTERM. It responds 202 when drain starts and 409 when it cannot control a running child.

The image boot JVM can be newer than the worker service downloaded from the connected server. Before it enables drain controls, boot confirms the child protocol with a one-time nonce. A legacy service, including an unmodified 24.12 service, continues normal worker operation but does not support safe autoscaling: GET /status/drain and an authenticated POST /status/drain return 409 with state=UNSUPPORTED, and SIGTERM uses direct termination rather than claiming a graceful drain. Deploy a server-provided worker service that supports this SAPIO 26.8+ control protocol before enabling scale-in.

The status API is intentionally private. Restrict its security group to the workload that is allowed to initiate a drain. POST /status/drain is disabled by default until VELOXWORKER_DRAIN_AUTH_TOKEN is configured; callers must send Authorization: Bearer <token>. To deliberately run an unauthenticated private endpoint, set VELOXWORKER_DRAIN_AUTH_UNSECURED_ENABLED=true; this emits no authentication check and must not replace the network restriction.

On SIGTERM, the parent asks the child to drain, waits up to VELOXWORKER_DRAIN_TIMEOUT_SECONDS (default 120), then terminates it. A timeout is logged and emitted through the worker lifecycle metrics before the bounded shutdown continues. The controlled path bypasses the legacy post-exit sleep.

For ECS Managed Instances, set VELOXWORKER_ECS_TASK_PROTECTION_ENABLED=true. The child then uses the injected ECS_AGENT_URI endpoint to protect a task immediately before it accepts its first executor, renews that lease while work remains active, and releases protection after its last executor closes. If protection cannot be set, the request is rejected and the worker drains rather than accepting unprotected work. Configure the lease and request timeout with VELOXWORKER_ECS_TASK_PROTECTION_LEASE_MINUTES (default 5) and VELOXWORKER_ECS_TASK_PROTECTION_TIMEOUT_MS (default 5000).

Fargate does not use the task-protection endpoint. Keep automatic scale-in disabled unless the ECS stop timeout is explicitly budgeted to be at least the worker drain timeout; the generic SIGTERM drain will then give in-flight work a bounded chance to finish.

⁠Remote-worker lifecycle metrics (SAPIO 26.8+)

Remote-worker lifecycle metrics have independent CloudWatch and OpenTelemetry exporters. They expose the same bounded metric names and dimensions: lifecycle operation duration/count and the local active-executor gauge. Neither exporter depends on the other.

• VELOXWORKER_REMOTE_WORKER_METRICS_OTEL_ENABLED (Boolean):
  • Default Value: true
  • Controls only this lifecycle metric family through the configured OpenTelemetry meter. It does not install, configure, or disable the OpenTelemetry Java agent itself.
• VELOXWORKER_REMOTE_WORKER_METRICS_CLOUDWATCH_ENABLED (Boolean):
  • Default Value: false
  • Sends the lifecycle metric family directly to CloudWatch without requiring an OpenTelemetry agent, OTLP endpoint, or ADOT Collector. The worker IAM role needs permission to publish CloudWatch metrics.

When direct CloudWatch lifecycle metrics are enabled, use CLOUDWATCH_NAMESPACE (default Velox/RemoteWorker), CLOUDWATCH_COLLECTION_INTERVAL (default 30 seconds), and optional CLOUDWATCH_ADDITIONAL_DIMENSIONS (Name=Value,Name2=Value2). The active-executor gauge is published at this interval, including zero while idle, so scaling policies can distinguish an idle worker from missing telemetry. These settings do not change the existing boot-process system/JMX CloudWatch metric configuration.