guest: pre-create nested mount points under read-only mounts

[Jie Chen (jiechen0826)]

Problem

Containers fail to start with read-only file system (EROFS) when a volume is mounted into a subdirectory of another volume that is mounted read-only. For example:

• an emptyDir at /mnt/data (readOnly) plus another volume at /mnt/data/subdir
• a read-only configMap at /etc/coredns plus a custom config volume at /etc/coredns/custom

Root cause

runc applies spec.Mounts in order and remounts a bind mount read-only as soon as it processes it. When it then processes a child mount nested under it, runc must create the child's mount point under the now read-only parent, which fails with EROFS.

On a normal Kubernetes node the kubelet pre-creates these subdirectories on the host before handing the spec to the runtime. Inside an LCOW UVM the guest (GCS) owns the mount setup, so runc creates the mount point at runc create time and hits the read-only parent.

Reordering the mounts does not help: mounting the child before the parent makes the parent bind mount shadow the child.

Fix

ensureNestedMountTargets (new internal/guest/runtime/hcsv2/mount.go), called from Host.CreateContainer just before the spec is written for runc. For each mount whose destination is nested under a read-only bind mount, it pre-creates the mount point inside that parent's writable source (a directory, or an empty file when the child source is a file), so it already exists once runc makes the parent read-only. This mirrors what the kubelet does on a host node.

It is best effort: if a mount point cannot be pre-created it logs and continues, so runc's own error still surfaces and unaffected containers are unchanged.

Testing

• Unit tests for the parent-selection and path logic.
• Root-gated functional tests using real bind mounts: one reproduces the EROFS failure (control), and one verifies the fix makes the nested mount point usable under a read-only parent.

[Harsh Rawat (rawahars)]

Which type of parent mounts are targeted here?
There can be SCSI VHD mounts which are attached in RO config to the UVM. If nested mounts are created under the same then that can potentially fail.

[Jie Chen (jiechen0826)]

Which type of parent mounts are targeted here? There can be SCSI VHD mounts which are attached in RO config to the UVM. If nested mounts are created under the same then that can potentially fail.

Good catch, thanks. This only targets read-only bind mounts whose source is a writable directory (the CRI volume mounts like emptyDir/configMap). For a read-only SCSI VHD the source is itself a read-only filesystem, so there's nothing to pre-create and runc would fail the same way regardless. I've added a guard that skips any parent whose source isn't writable, so read-only SCSI VHD and dm-verity parents are skipped up front. Does that cover your concern?