Skip to content

Move ls to run-worker, one level only - #219

Open
bethac07 wants to merge 1 commit into
masterfrom
issues/214
Open

bethac07 wants to merge 1 commit into
masterfrom
issues/214

Conversation

@bethac07

Copy link
Copy Markdown
Collaborator

Resolves #214

Currently available as bethcimini/distributed-cellprofiler:2.2.1_4.2.8_fixatt4 ; had one small-scale test but would be great to have a larger test before we pull!

@ErinWeisbart

Copy link
Copy Markdown
Member

Thanks Beth!
Do you mind explaining briefly what the other fixes we tested were (since I know not all of them worked) so that we have a record here in case this needs to be revisited again for some reason?

@bethac07

Copy link
Copy Markdown
Collaborator Author

Sure!

Previously, inside cpworker, we os.listdired the bucket root and then subprocess lsed one level down. Exactly WHY that very strange combo is lost to time inside my brain, though possibly a more diligent search of 2016-era issues/commits might turn up if other things were tried.

Fully removing any ls of the mounted system was the only thing that caused bucket mounting to break(per my understanding - I made the containers but did not personally test them). Keeping either an os.listdir OR subprocess call to ls inside the cpworker script at only a single level worked to mount the bucket. So did a two-level ls in run_worker (aka ls /home/ubuntu/bucket/*/) . But since the cpworker script runs every job, and the run_worker script runs only on container mount, this solution (ls /home/ubuntu/bucket/ ) generates the minimal "touches" (and therefore costs) that still preserve functionality.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

See if we can minimize ls'ing for S3FS bug

2 participants