One Parallel Search

Consider:

$ rg "unsafe" crates/

The production path is short enough to trace completely, but each step changes who owns the operation.

1. Select parallel mode

main parses low-level flags into HiArgs. run selects search_parallel unless the effective thread count is one. Sorting or searching one explicit file forces that count to one before this decision.

Source: crates/core/main.rs:78

2. Construct shared templates

search_parallel constructs:

• a HaystackBuilder for application-level file eligibility;
• a BufferWriter for atomic-ish file-sized output commits;
• atomic matched and searched flags;
• optional statistics behind a mutex; and
• one configured SearchWorker that will be cloned per traversal worker.

The initial SearchWorker owns a matcher, reusable Searcher scratch buffers, and a printer whose destination is a private output buffer.

Source: crates/core/main.rs:166

3. Build one visitor per OS thread

WalkParallel::run invokes its factory once per traversal thread. Each factory call clones SearchWorker, producing thread-local matcher, searcher, printer, and scratch state.

args.walk_builder()?.build_parallel().run(|| {
    let mut searcher = searcher.clone();
    Box::new(move |result| { /* this thread's callback */ })
});

There is no lock around Searcher. Mutable reusable buffers stay local to one worker, and synchronization is reserved for genuinely shared results.

4. Discover and filter entries

Each traversal worker processes directory Work. It reads a directory, extends the inherited ignore matcher, rejects ignored paths, maximum-size violations, and custom-filter failures, then pushes accepted children onto its local deque.

Source: crates/ignore/src/walk.rs:1769, generate_work

5. Convert an entry into a haystack

The application callback receives Result<DirEntry, ignore::Error>. HaystackBuilder reports traversal errors and rejects non-files, while still preserving special treatment for explicit paths and stdin.

This is a second filtering boundary: ignore answers “should traversal expose this entry?” while HaystackBuilder answers “should ripgrep search it?”

6. Search one file synchronously

SearchWorker::search configures binary detection, then chooses stdin, preprocessor, decompressor, or ordinary path search. Ordinary path search calls Searcher::search_path, which can use a memory map, a whole-file buffer for multiline mode, or an incremental rolling line buffer.

Source: SearchWorker::search, Searcher::search_path

The worker blocks while reading this file. Parallelism comes from other OS threads searching other files, not from suspending this operation.

7. Commit output

The printer writes all results for the file into its worker-local buffer. Only after the search completes does BufferWriter::print serialize that buffer to stdout. Files may appear in nondeterministic order, but their lines do not become arbitrarily interleaved with another worker’s file.

Source: crates/core/main.rs:189

State at the end

The worker clears and reuses its buffer for the next file. Shared atomics retain whether anything was searched or matched. When all local deques are empty and all workers have become inactive, a quit message propagates and scoped threads are joined before search_parallel returns.