Skip to content

A pipe runs every stage to completion, even when the next stage uses only part of the result #387

Description

@shellicar

Limiting a pipe's output does not limit the work the pipe does. Each stage produces its full result before the next stage runs, so a later stage that uses only part of an earlier stage's output still pays for that earlier stage in full.

For example, a Find piped into Head(1) over a large directory walks the whole tree before Head runs: Find produces every file under the starting path, and only then does Head take the first. On a home directory with hundreds of thousands of files, returning one item takes tens of seconds to minutes, or runs out of memory. The same shape in the shell, find ~ | head -1, returns almost immediately, because head takes one line and the pipeline stops.

To reproduce: run a Find over a large tree into Head(1), and compare the wall-clock time against find <same path> | head -1.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions