If a chatbot occasionally gives correct answers, why does the "bullshit" framing say you still shouldn't simply trust it?

Because a true output is produced by the very same truth-indifferent process as a false one — correctness is incidental, so a right answer is no evidence the next one will be.

This is the sharpest practical payoff of the bullshit framing. A human bullshitter who doesn't care about truth will still say true things sometimes — "good bullshit often contains some degree of truth; that's part of what makes it convincing." The same holds for an LLM: when it's right, it's right by accident of the prediction, not because it checked or knew.

So:

• A correct answer does not indicate the system was tracking the truth.
• It tracks truth, at best, "indirectly and incidentally."
• Therefore every output deserves the same wariness — you can't rely on past accuracy as a guarantee.

This is why "verify the output" is non-negotiable, and why being lulled by a streak of good answers is the trap.

Tip: Trust is earned by a reliable connection to truth. A bullshitter (human or machine) has none — so their hits don't certify their next claim.