docker service update --force silently reorders service mounts; subsequent identical docker stack deploy rolls the service again

[KOSTOGE]

Description

docker service update --force mutates the order of Spec.TaskTemplate.ContainerSpec.Mounts even when no mount-related flag (--mount-add, --mount-rm) was supplied. A subsequent, completely unchanged docker stack deploy then restores the Compose declaration order, SwarmKit sees that as a TaskSpec change, and the service is rolled a second time even though nothing user-visible changed.

Source reading (current master) explains the mutation:

• In cli/command/service/update.go, updateService() calls updateMounts() unconditionally, unlike most other properties which are guarded by "did the corresponding flag change" checks.
• updateMounts() rebuilds the mounts slice and explicitly sorts it by Source, then Target, even when neither --mount-add nor --mount-rm was specified.
• So docker service update --force svc does two things: rebuild/sort cspec.Mounts, and increment TaskTemplate.ForceUpdate. The user only asked for the second.
• SwarmKit compares an existing task against the desired service spec using reflect.DeepEqual on the TaskSpec (moby/swarmkit/manager/orchestrator/task.go). Mounts is a slice, so element ordering participates in the comparison. Same mounts, same targets, same options, different order equals a different TaskSpec.

Observed in production on a multi-node cluster (nightly service update --force jobs caused every subsequent routine stack deploy to roll unchanged worker tasks) and reproduced deterministically on a single-node swarm with Docker 29.2.0, see steps below.

Reproduce

1. Save this as stack.yml. Volume names are chosen so Compose declaration order (z, a, m) differs from the CLI's Source sort order (a, m, z):

services:
  test:
    image: alpine:3.22
    command: ["sleep", "infinity"]
    volumes:
      - z-volume:/data/z
      - a-volume:/data/a
      - m-volume:/data/m
    deploy:
      replicas: 1

volumes:
  z-volume:
  a-volume:
  m-volume:

2. Deploy and record mount order plus task ID:

docker swarm init
docker stack deploy -c stack.yml mountbug
docker service inspect mountbug_test \
  --format '{{range .Spec.TaskTemplate.ContainerSpec.Mounts}}{{println .Source "->" .Target}}{{end}}'
docker service ps mountbug_test --format '{{.ID}} {{.CurrentState}}'

Result: mounts in Compose order z -> /data/z, a -> /data/a, m -> /data/m; running task whdbju5xst40.

3. Force update and inspect again:

docker service update --force mountbug_test
docker service inspect mountbug_test \
  --format '{{range .Spec.TaskTemplate.ContainerSpec.Mounts}}{{println .Source "->" .Target}}{{end}}'

Result: mounts silently rewritten to sorted order a -> /data/a, m -> /data/m, z -> /data/z. One rollout happened here, which is expected since --force was requested (task whdbju5xst40 replaced by y68o9cbonah0).

4. Wait for convergence, then deploy the identical file again with zero changes:

docker stack deploy -c stack.yml mountbug
docker service inspect mountbug_test \
  --format '{{range .Spec.TaskTemplate.ContainerSpec.Mounts}}{{println .Source "->" .Target}}{{end}}'
docker service ps mountbug_test --format '{{.ID}} {{.Name}} {{.CurrentState}}'

Actual result: mounts revert to Compose order z, a, m and a new task appears:

vg8iojw090s6 mountbug_test.1 Running less than a second ago
y68o9cbonah0 mountbug_test.1 Shutdown 2 seconds ago
whdbju5xst40 mountbug_test.1 Shutdown 46 seconds ago

That second rollout was caused by nothing but mount slice ordering.

Expected behavior

Per the documented purpose of --force ("can be used to perform a rolling restart without any changes to the service parameters"), docker service update --force SERVICE should only increment TaskTemplate.ForceUpdate and must not modify ContainerSpec.Mounts when no mount flag was supplied. A subsequent identical docker stack deploy should then be a no-op instead of replacing tasks.

A command that is not modifying mounts should not rewrite their representation. One possible fix appears to be calling updateMounts() only when --mount-add or --mount-rm actually changed, matching the guard pattern used for other properties in the same function. We are not attached to that specific implementation; the unconditional mutation is the problem. Canonicalizing order inside SwarmKit's comparison would also mask the symptom, but the CLI rewriting untouched fields seems like the root cause.

docker version

Client:
 Version:           29.2.0
 API version:       1.53
 Go version:        go1.25.6
 Git commit:        0b9d198
 Built:             Mon Jan 26 19:25:17 2026
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Desktop 4.60.1 (218372)
 Engine:
  Version:          29.2.0
  API version:      1.53 (minimum version 1.44)
  Go version:       go1.25.6
  Git commit:       9c62384
  Built:            Mon Jan 26 19:26:07 2026
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v2.2.1
  GitCommit:        dea7da592f5d1d2b7755e3a161be07f43fad8f75
 runc:
  Version:          1.3.4
  GitCommit:        v1.3.4-0-gd6d73eb8
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    29.2.0
 Context:    default
 Debug Mode: false
 Plugins:
  ai: Docker AI Agent - Ask Gordon (Docker Inc.)
    Version:  v1.17.2
    Path:     /usr/local/lib/docker/cli-plugins/docker-ai
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.31.1-desktop.1
    Path:     /usr/local/lib/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v5.0.2
    Path:     /usr/local/lib/docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.47
    Path:     /usr/local/lib/docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Docker Inc.)
    Version:  v0.3.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-desktop
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.31
    Path:     /usr/local/lib/docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.4.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-init
  mcp: Docker MCP Plugin (Docker Inc.)
    Version:  v0.38.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-mcp
  model: Docker Model Runner (Docker Inc.)
    Version:  v1.0.8
    Path:     /usr/local/lib/docker/cli-plugins/docker-model
  offload: Docker Offload (Docker Inc.)
    Version:  v0.5.42
    Path:     /usr/local/lib/docker/cli-plugins/docker-offload
  pass: Docker Pass Secrets Manager Plugin (beta) (Docker Inc.)
    Version:  v0.0.24
    Path:     /usr/local/lib/docker/cli-plugins/docker-pass
  sandbox: Docker Sandbox (Docker Inc.)
    Version:  v0.11.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-sandbox
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.19.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-scout

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 32
 Server Version: 29.2.0
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Discovered Devices:
  cdi: docker.com/gpu=webgpu
 Swarm: active
  NodeID: xahervt3xgh0xsneob1wdv0c9
  Is Manager: true
  ClusterID: yqee4mscjtz1ljm98pn2k8mtj
  Managers: 1
  Nodes: 1
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 192.168.65.3
  Manager Addresses:
   192.168.65.3:2377
 Runtimes: io.containerd.runc.v2 nvidia runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: dea7da592f5d1d2b7755e3a161be07f43fad8f75
 runc version: v1.3.4-0-gd6d73eb8
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.6.87.2-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 26.76GiB
 Name: docker-desktop
 ID: 2126ddc4-92da-457b-8681-359faad472e7
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///var/run/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false
 Firewall Backend: iptables

Additional Info

Production correlation supporting the repro: two independent services showed the same pattern where a nightly docker service update --force job reordered mounts into Source-sorted order and the next routine docker stack deploy rolled all worker tasks back to Compose order. A third service without the nightly force-update did not exhibit the behavior, which rules out node-level causes. The single-node repro above needs no GPU, NFS, external storage, or multiple nodes.

Relevant code paths:

• cli/command/service/update.go: unconditional updateMounts(flags, &cspec.Mounts) call inside updateService(), and the sort at the end of updateMounts() by Source then Target.
• moby/swarmkit/manager/orchestrator/task.go: task dirty check via reflect.DeepEqual(serviceTaskSpec, t.Spec).

One thing we deliberately did not propose as the fix: making stack deploy sort mounts too. Mount order can matter for nesting scenarios (see moby/moby#22327 for historical tmpfs ordering failures), so the safer principle is that commands not modifying mounts should not rewrite them.

Duplicate search: we searched docker/cli and moby issues for combinations of "service update", "--force", "mount order", and "stack deploy" and found no existing report of this exact interaction.