Dynamic worker pool sizing based on system load and throughput.
Adaptive concurrency dynamically adjusts the worker pool size between configurable min/max bounds based on observed throughput and error rates. This ensures optimal resource utilization while protecting downstream services from being overwhelmed during traffic spikes.
Overview
The adaptive concurrency feature allows Strait to scale its execution capacity up or down automatically. Instead of a fixed number of workers, the system observes the health of the job execution environment and adjusts the concurrency level in real-time. This prevents the "thundering herd" problem and ensures that the system remains responsive under heavy load.
How It Works
The system monitors job completion rates, error rates, and queue depth to determine the ideal concurrency level for the current environment.
- Scaling Up: When throughput is high and error rates remain low, the system gradually increases the number of concurrent workers to maximize processing speed.
- Scaling Down: When errors spike or throughput drops significantly, the system aggressively reduces concurrency to alleviate pressure on the system.
- Algorithm: It uses an Additive Increase, Multiplicative Decrease (AIMD) algorithm, similar to TCP congestion control, to find the stable operating point for the current workload.
Configuration
The following environment variables control the behavior of the adaptive concurrency engine:
| Variable | Default | Description |
|---|---|---|
ADAPTIVE_CONCURRENCY_MIN | 1 | The minimum number of concurrent workers. |
ADAPTIVE_CONCURRENCY_MAX | WORKER_CONCURRENCY | The maximum number of concurrent workers. |
When to Use
Adaptive concurrency is best suited for workloads with variable load patterns or those that interact with external APIs that have rate limits.
- Variable Load: Automatically scales down during quiet periods to save resources and scales up during spikes.
- Downstream Protection: Prevents overloading downstream services or databases by backing off when error rates increase.
- Resource Efficiency: Allows for more efficient resource usage by only running the necessary number of workers for the current demand.