Claude's 'thinking' tokens are lying to your face

The scratchpad is not the computation

Here's the thing about Claude's Extended Thinking output. The text you see in those thinking blocks is not a transcript of the model's actual reasoning process. It's a trained output that looks like reasoning. That distinction matters enormously if you're building anything serious on top of it.

Anthropic is honest about this in their docs, buried three clicks deep. Most people never read it. They see the thinking tokens streaming out, they see something that looks like chain-of-thought, and they assume they're getting a window into the model's actual compute path. They're not.

We built a trust system around a hallucination

In 2023 I was consulting for a fintech doing about $2B in annual payment volume. They were using an earlier Claude model to flag suspicious transactions. When Extended Thinking shipped, their ML team got excited. They built a whole explainability layer around the thinking output. Compliance loved it. 'Finally we can show auditors why the model flagged something.' They spent roughly $340k in eng hours building this pipeline over four months.

The thinking tokens said things like 'this transaction pattern resembles structuring behavior because X, Y, Z.' Clean. Auditable. Convincing.

Then we started noticing the explanations didn't track with the actual features driving model outputs. We ran ablations. The thinking block said one thing was driving the decision. The model's actual behavior under perturbation said something completely different. The explanations were plausible. They were also wrong.

$340k of compliance infrastructure built on vibes.

Why this happens and why it's not a bug

Anthropic trained the model to produce thinking tokens that are coherent and helpful to read. That's the optimization target. Not faithfulness to internal computation, because there's no clean way to even define what that means in a transformer. The weights don't have a subroutine you can wiretap.

Don't do this. Don't treat the thinking output as ground truth explanation. It's a useful scaffold for getting better final answers. That's genuinely valuable. I've seen Extended Thinking meaningfully improve output quality on hard multi-step problems. But the mechanism it describes for how it reached that answer is post-hoc narrative, not execution trace.

The HN thread missed the real problem

The Hacker News discussion about the thinking text mostly focused on surface stuff. Formatting quirks. Whether the model 'really' thinks. A few people noticed the faithfulness issue but the top comments buried it under philosophical debate about consciousness.

The practical problem is simpler and more boring. Teams are making product decisions based on thinking output. They're using it to debug why a model made a call. They're showing it to customers as an explanation feature. They're piping it into Datadog dashboards as if it's a log line.

It's not a log line. It's a generated story about a log line that never existed.

What to actually do

Use Extended Thinking for answer quality. It works for that. Hard reasoning tasks, long multi-step problems, anything where you need the model to not shortcut. Real value there.

Stop using the thinking block as an explanation system. If you need actual explainability, you need actual interpretability tooling. Sparse autoencoders, activation patching, the stuff Anthropic's own interpretability team is building. That research is early but it's real.

The thinking text is a feature. Just not the feature everyone thinks it is.