diff --git a/server-admin/queues.md b/server-admin/queues.md index 8427b95..cc58131 100644 --- a/server-admin/queues.md +++ b/server-admin/queues.md @@ -74,6 +74,17 @@ There is no `QueueAccepted` condition. A run's `queued` status and queue entry a The scheduler stores only the latest queued condition. If the same reason and message are already present, the scheduler refreshes `checkedAt` only when the new observation is at least one minute newer than the stored observation. Older observations do not replace newer ones. +## Scheduled Runs and Queues + +A DAG assigned to a queue defined in `config.yaml` — by its own `queue` field or +by one inherited from `base.yaml` — has its scheduled runs enqueued rather than +started as the schedule fires. The queue's +`max_concurrency` therefore bounds every run of the DAGs that share it, and a +run waits in `queued` until a slot is free. + +A DAG with no `queue` field uses a local queue, and the scheduler starts its +runs directly. + ## Catchup Runs and Queues Catchup runs (missed run replay) are dispatched through the queue system. When the scheduler detects missed cron intervals for a DAG with `catchup_window` set, it enqueues each interval as a queue item with a deterministic run ID. The queue processor then executes them in order. diff --git a/writing-workflows/queues.md b/writing-workflows/queues.md index 519f32f..45994c3 100644 --- a/writing-workflows/queues.md +++ b/writing-workflows/queues.md @@ -15,7 +15,7 @@ steps: run: echo "Processing batch" ``` -The scheduler places this DAG into the `batch` queue. The queue's `max_concurrency` (defined in `config.yaml`) determines how many DAGs in this queue can run at the same time. +The scheduler places this DAG into the `batch` queue. The queue's `max_concurrency` (defined in `config.yaml`) determines how many DAGs in this queue can run at the same time. Scheduled runs, catch-up runs, retries, and runs added with `dagu enqueue` all wait for a free slot in that queue. When a run is waiting in `queued`, the scheduler may attach a runtime condition that explains the latest observed reason it has not started yet, such as a queue concurrency limit or a distributed worker selection issue. See [Queued Runtime Conditions](/server-admin/queues#queued-runtime-conditions) for the exact reason names and messages. @@ -34,7 +34,7 @@ See [Base Configuration](/server-admin/base-config) for how base config merging ## Behavior Without a Queue -When a DAG does not set `queue` (and no base config default exists), it runs in a local queue named after the DAG itself. Local queues have a fixed concurrency of 1, meaning only one instance of that DAG runs at a time. +When a DAG does not set `queue` (and no base config default exists), it runs in a local queue named after the DAG itself. Local queues have a fixed concurrency of 1, meaning only one instance of that DAG runs at a time, and the scheduler starts its runs directly instead of enqueueing them. ## Overriding at Enqueue Time diff --git a/writing-workflows/scheduling.md b/writing-workflows/scheduling.md index 8bcbda6..7edf09c 100644 --- a/writing-workflows/scheduling.md +++ b/writing-workflows/scheduling.md @@ -306,7 +306,9 @@ If `queues.enabled` is `false`, the scheduler logs a warning per DAG that has `c ### Dispatch via enqueue -Catchup runs are dispatched through the queue system, not started directly. For each missed interval, the scheduler: +Catchup runs are dispatched through the queue system, not started directly. So +are the scheduled runs of a DAG assigned to a queue defined in `config.yaml`, +whether the DAG names that queue itself or inherits it from `base.yaml`. For each missed interval, the scheduler: 1. Generates a deterministic run ID from the DAG name and scheduled time 2. Checks if a run with that ID already exists (`FindAttempt`): if so, skips it