Claude Code Grep vs LSP: 2,694 Sessions, 0 LSP Calls

On 4 September a post titled “Grep beats LSP? Why coding agents ignore your fancier tools” got 97 points and 68 comments on Hacker News. The question under it was why an agent with a language server available keeps reaching for text search. I run a fleet of headless Claude Code sessions on a Mac mini, and every one of them leaves a transcript, so I had a dataset for the question before I had an opinion.

The answer from 2,694 sessions is blunt. Zero LSP calls. Twenty-six calls to the built-in Grep tool. 3,410 Bash commands containing grep or rg. But the count is the least interesting part, because when I went looking for why, the choice turned out to be made by two sentences in the harness rather than by the model, and I could flip it with one flag.

What 30 days of transcripts say

Every session Claude Code runs writes a JSONL file under ~/.claude/projects/. I parsed every file modified in the last 30 days, kept records timestamped after 2026-08-09, and excluded the session writing this post. That gave 2,711 files, 2,694 distinct sessions and 18,198 tool_use blocks. The sessions are almost all headless: 2,670 have entrypoint: sdk-cli, 17 are interactive.

Bash dominates with 11,614 calls. Read has 1,426, Edit 1,235, Write 776. The Grep tool has 26. Glob has 5. Nothing with lsp in its name was called once, and no session used ast_grep either. Of the 11,614 Bash commands, 3,410 contain grep or rg. That number needs unpacking, because my regex catches everything. 1,125 of them are | grep filters on the output of something else, like launchctl list | grep damdam. 2,285 run grep against files directly, and 343 of those are recursive searches over a directory. Only 2 of the 3,410 use rg, even though ripgrep is installed on this machine and is what the Grep tool is built on.

Splitting by permission mode is where the pattern shows up.

Permission mode	Sessions	tool_use	Bash	Bash with grep	Grep tool	Read
auto	2,137	4,501	1,630	246	26	969
bypassPermissions	359	13,643	9,936	3,164	0	457
default	137	48	42	0	0	0
acceptEdits	6	6	6	0	0	0
not recorded	55	0	0	0	0	0

The 359 bypass sessions are my publishing fleet; 327 of them ran in this repository. They account for three quarters of all tool calls and every one of the 3,164 greps came through Bash. Not one Grep tool call. The auto sessions belong to three other projects on the same machine and used Read 969 times against 1,630 Bash calls, a completely different ratio. All 26 Grep tool calls live there, in 18 sessions.

The two sentences that disagree

I pulled the strings out of the 2.1.263 binary to see what the model is actually told. The Grep tool’s own description is unambiguous:

Content search built on ripgrep. Prefer this over `grep`/`rg` via Bash
A powerful search tool built on ripgrep
  Usage:
  - ALWAYS use Grep for search tasks. NEVER invoke `grep` or `rg` as a Bash command.
    The Grep tool has been optimized for correct permissions and access.

And here is the text that Claude Code attaches to the conversation when you start it with --permission-mode bypassPermissions or --dangerously-skip-permissions, which is how every slot in my fleet runs:

While bypass permissions mode is active:
Do your work through the Bash tool wherever it can accomplish the job: read files
with cat, head, or sed -n, search with grep and find, and make file changes with
sed, heredocs, or short scripts, rather than using the dedicated Read, Edit, or
Write tools. Fall back to a dedicated tool only when Bash genuinely cannot do the job.

Both strings are in the same binary. One says never grep from Bash; the other says search with grep from Bash. In my transcripts the second one wins 3,164 to 0. The minified source around it shows the selection logic: the bypass branch always gets the strict version, an auto-mode branch can get a strict or a “relaxed” version that ends with “The choice is yours”, and other modes get nothing. I cannot see from a transcript which variant the 2,137 auto sessions received, so I am not reading their 26-to-246 ratio as a clean control. The string occurs twice in the binary and zero times in the plugins, hooks or settings on this machine, so this is Claude Code’s own text, not something I configured.

Nine probes, one flag

A census cannot separate “the model prefers grep” from “the harness told it to”. A probe can. I made a three-file repository where render_post is defined once, called twice, imported once and mentioned in a comment and a README. Then I asked the same question nine times with the same model:

claude -p 'Find every place in this repository where the function render_post
  is defined or called. Reply with file:line pairs only.' --model sonnet \
  --permission-mode bypassPermissions
# vs
  --permission-mode default --allowedTools "Bash,Grep,Glob,Read"
  --permission-mode acceptEdits --allowedTools "Bash,Grep,Glob,Read"

Bash was allowed in all nine runs, so nothing was forced. The only difference between the arms is the mode and the paragraph that comes with it.

Mode	Run 1	Run 2	Run 3
bypassPermissions	Bash: grep -rn "render_post" . then Read ×2	Bash: grep -rn "render_post" <repo>	Bash: grep -rn "render_post" <repo>
default	Grep: render_post	Grep: render_post	Grep: render_post
acceptEdits	Grep: render_post	Grep: render_post	Grep: render_post

Three for three one way, six for six the other. Same model, same prompt, same files, same permission to run a shell. The tool choice that the Hacker News thread attributed to training or habit was, in this harness, a function of one command-line flag. One detail I will not overread: three of the six Grep-tool runs left out the import line src/build.py:1 in their answer, and all three bypass runs included it. Nine runs is not evidence of anything, but it is the kind of difference the agentconnect study measured properly.

Where the LSP was the whole time

The zero deserves its own confound. Claude Code has had a native LSP tool since 2.0.74, published to npm on 2025-12-19, and the changelog since then is a trail of fixes: 2.1.0 stopped the tool being enabled with no server configured, 2.1.47 stopped findReferences returning results from node_modules, 2.1.162 made workspaceSymbol return anything at all. But the docs are explicit that Claude Code keeps the tool inactive until you install a code intelligence plugin, and the plugin does not ship the server; you install pyright or typescript-language-server yourself. On this machine claude plugin list shows three plugins and none of them is an LSP plugin. The only language server on the PATH is the clangd shim that Apple’s command line tools install, for a fleet that writes Python and shell.

So the native LSP tool was never on the menu. What was on the menu is a set of MCP tools from a plugin I do use, oh-my-claudecode, which exposes lsp_goto_definition, lsp_find_references and ten siblings. They appear as deferred tools, the kind a session has to load by name through ToolSearch before it can call them, and I covered how late that attachment happens in headless runs already. The listing is present in 2,563 of the 2,694 sessions. ToolSearch was called 338 times in those sessions. The query string contained “lsp” zero times; 246 of the 338 were select:WebSearch,WebFetch. Whether those MCP tools would even have found a server to talk to is a question no session got far enough to ask.

That is the honest shape of my zero. It is not “the model saw a working language server and chose grep”. It is “nobody wired one, the harness said grep, and the model did what it was told”.

What the study measured that I could not

The article behind the HN thread, by Pengcheng Xu at agentconnect, did the experiment my transcripts cannot. With grep and a live pyright-backed LSP both available and no steer either way, three Claude models picked the semantic tool 0% to 6% of the time on simple localization tasks and 45% to 57% of the time when asked to find every caller. Forcing LSP first on the simple tasks dropped success from 100% to 89%. On reference-completeness the LSP path had precision 1.00 against grep’s 0.76, but recall stayed near 0.66 in both arms, so it removed false matches without finding more true ones. On a clean TypeScript repository it added no F1 and cost 16% more tokens; on a noisy one it added 0.246 F1 and saved 12%.

The part I keep coming back to is the output-shape experiment. Their LSP tool initially returned bare locations, and the agent then opened each file. When they changed the response to include the matching source line, the same backend went from pass@1 0.67 to 0.83 on rename tasks and follow-up reads fell from 15.2 to 3.2 per episode. Grep had been returning path:line:content all along. Anthropic’s own guidance on writing tools for agents says the same thing from the other side: the context a tool returns is part of its design. The author flags that this is a pilot with two to three rollouts per cell and no textDocument/rename tested; the task definitions are on GitHub.

The HN comments split the way you would expect. Of 68, 21 mention LSP and 15 grep or ripgrep; 7 talk about tokens, 4 about the harness or system prompt, 3 about ast-grep. One commenter wrote that they “frequently have a hard time getting Claude to use LSP at all”, another that a skill instructing the model to use the LSP was what finally worked, and a third described watching Claude Code do “lots of find and grep and ripgrep” and only afterwards asking it which tool would have been easier. A separate thread the day before, on armature’s 16,893-session study of which third-party services agents pick, is about a different kind of tool choice, but it is the same lesson at scale: the choice is measurable, and it is shaped by what the runtime puts in front of the model.

What I am changing, and what I am not

I am not installing an LSP plugin for the fleet. The work these sessions do is publishing: query a database, render HTML, check links, write a log line. The searches are for strings in Markdown and shell scripts, which is grep’s home ground, and the study’s own result is that semantic navigation earns its tokens on noisy code and reference-completeness tasks, neither of which describes this repository. If that changes I now know the number that would tell me: grep precision on the reference tasks, not language.

What I did change is how I read my own transcripts. I have written before about the harness being everything around the model, and about a tool-choice habit that persisted 172 outputs to disk because sessions picked tail -N in Bash over a tool with a smaller read-back. I had been attributing those choices to the model. Some of them are, but the first thing to check is which paragraph the mode attached, because that paragraph out-votes the tool’s own description 3,164 to 0. If you run --dangerously-skip-permissions and are wondering why Claude Code never uses Grep or Read, that is your answer, and a CLAUDE.md line asking it to use Grep is arguing with a system-level steer. If you run auto or default mode, the same model on the same repository will reach for the Grep tool, as six probes out of six did here.

The slot prompts and the launchd runner that produce these transcripts are in the Playbook; the census script is forty lines of Python over the JSONL and I have described the fields in the note below.

Sources and method for this post: transcript files under this machine’s ~/.claude/projects/ with a modification time in the last 30 days, parsed on 2026-09-08 for tool_use blocks in assistant records with a timestamp on or after 2026-08-09, with this session’s own transcript excluded; permission mode is the permissionMode field on the session’s records. “Bash with grep” means the command string matches grep, rg, egrep or fgrep as a word anywhere, and the recursive and pipe-filter splits come from a second pass over the same commands. Harness text is from strings on the 2.1.263 binary installed at ~/.local/share/claude/versions/; the nine probes ran with Sonnet between 12:04 and 12:08 KST on 2026-09-08 and their tool sequences were read from their own JSONL files. Hacker News comment counts come from the Algolia items API for stories 49560260 and 49557206; the study numbers are quoted from the agentconnect article and its repository as published. Release dates are from npm view @anthropic-ai/claude-code time. I have no LSP plugin installed and did not measure a working language server; this post is about the choice, not about which tool is better.