How We Actually Test Claude Prompt Codes: The Full Methodology Behind the 47% Placebo Finding

Why this post exists

Earlier this year we published a piece called "47% of Popular Claude Prompts Are Placebo". The headline claim is that nearly half of the viral prompt prefixes on YouTube and Reddit produce output indistinguishable from running the same prompt with no prefix at all.

That post got a lot of pushback. Some of it fair. "Where's the raw data?" "How many runs per prompt?" "Who scored them?" "How do you know your scorers weren't biased?" These are the right questions. This is the post that answers them.

We've expanded the test set from 120 codes to 160+ since the original writeup. We've re-run the harness twice, once when Haiku 4.5 shipped and once when Opus 4.7 shipped. We've tightened the scoring. We're publishing the methodology in full so you can replicate, push back, or catch us being wrong.

One thing up front. This is not peer-reviewed research. It's structured hypothesis testing done by a small team with limited resources. It's dramatically better than "my friend on Twitter says this prompt works," and dramatically worse than a controlled study run by researchers with proper baselines and larger N. We name the limits below in a section called "where this methodology breaks." Read that section before you cite us.

The core question we're trying to answer

For any given prompt prefix (a code like ULTRATHINK, L99, /skeptic, /deepthink), does adding the prefix to a prompt actually change what Claude reasons about, or does it only change how Claude talks about the same reasoning?

We call the first kind a reasoning shifter. The second kind a placebo. There are two intermediate categories too, which we'll define below.

The question sounds simple. It's not, because "reasoning" isn't directly observable. We only see the output. So we have to infer whether the reasoning changed by looking at whether the conclusions changed, whether the premises Claude accepted changed, and whether the structure of the argument changed. Any of these is evidence. None of them is definitive proof.

We've done our best to build a harness that separates these signals. The methodology below is the current state.

The classification system

Every code we test lands in one of four buckets:

Class	Definition	Share of tested codes
Reasoning shifter	Prefix changes what Claude attends to, which premises it accepts, or what conclusion it reaches. Different logical steps, sometimes different answer.	~4%
High-value structural	Prefix changes the format or tone of the output in a way that's measurably more useful, but the underlying reasoning is unchanged.	~21%
Niche or narrow	Prefix works for one specific task type and fails on the others.	~28%
Placebo	Prefix produces output that is blinded-indistinguishable from no-prefix baseline. Only vocabulary or padding changes.	~47%

The percentages are from our current 160-code test set. They're roughly stable across the last three refresh cycles.

The distinction that matters most is reasoning shifter versus placebo. High-value structural is real and worth using, but everyone already knows structural prompts work. The controversial claim is the placebo one.

How a code gets tested

Each code goes through this pipeline. It takes about 90 minutes of human time per code, most of which is the pair-scoring at the end.

Step 1: build the test set

For each code, we assemble 25 test prompts across 5 task categories:

Reasoning (math, logic, causal chains, premise challenges)
Writing (drafts, rewrites, tone shifts, ghost tests)
Coding (function writes, refactors, debugging)
Creative (short fiction, poetry, brainstorming)
Analysis (SWOT, comparisons, teardowns, decision framing)

Five prompts per category. The prompts are drawn from a fixed pool of 200 prompts we've built over the last 18 months, so we can compare across codes on the same prompts. Any new prompts we add get retro-run on the existing tested codes.

Step 2: run baseline and treated

Each test prompt is run three times with no prefix (baseline) and three times with the prefix under test (treated). Same model, same temperature, same system prompt. That's 150 completions per code (25 prompts x 6 runs).

We currently test primarily on Sonnet 4.6 because that's where most of our audience is. For codes that are model-sensitive, we re-run on Haiku 4.5 and Opus 4.7. Cross-model shifts are documented per-code.

Step 3: pair-wise blind scoring

Here's where the methodology got significantly stricter after the original placebo post shipped.

We show the scorer three baseline completions and three treated completions without labels. The scorer answers four questions per pair-set:

Are the reasoning steps different across the two sets?
Are the conclusions different across the two sets?
Are the accepted premises different across the two sets?
Is only the vocabulary or padding different?

Each answer is on a 0-3 scale. 0 means no, definitely not. 3 means yes, unambiguously.

Each code gets scored by two scorers independently. Both scorers are engineers on the team. We don't blind the code being tested from the scorer (the scorer knows which code they're scoring, they just don't know which completion is baseline vs treated), because we've found that scorers do a better job when they can bring domain understanding of what the code claims to do. This is a real limitation and we discuss it in the "where this breaks" section.

When the two scorers disagree by more than 2 points on any dimension, we discuss the disagreement and re-score after 48 hours. If we still disagree, we flag the code as "variance dominates" and it gets more test prompts.

Step 4: classify

With the pair-wise scores in hand, the code lands in one of the four buckets:

If avg question 1 or 2 or 3 score is >= 2 across categories: reasoning shifter.
If avg question 4 score is >= 2 and questions 1-3 are all < 1: placebo.
If question 1-3 scores are high in one category and low in others: niche or narrow.
Everything else: high-value structural.

The thresholds are calibrated against a set of 10 codes we consider unambiguous priors. L99 is an unambiguous reasoning shifter (it forces commitment). ULTRATHINK is an unambiguous placebo (it just makes Claude verbose). If our thresholds don't classify these correctly, we adjust the thresholds, not the codes.

The seven codes that survived

Out of 160+ codes tested, seven consistently score as reasoning shifters across every re-test cycle. Here they are with the specific reasoning behavior each triggers.

L99 — forces commitment to a specific recommendation instead of enumerating options. Reasoning behavior: Claude picks a stance and defends it. Baseline behavior: Claude enumerates two or three options with pros and cons.
/skeptic — rejects questions with wrong premises before responding. Reasoning behavior: Claude explicitly names the flawed premise and reframes. Baseline behavior: Claude accepts the premise and answers within it.
/blindspots — surfaces assumptions the user didn't state. Reasoning behavior: Claude enumerates unstated assumptions the recommendation depends on. Baseline behavior: Claude doesn't surface them.
/deepthink — forces step-by-step reasoning on problems that need it. Reasoning behavior: Claude produces 5-8 explicit reasoning steps before the conclusion. Baseline behavior: Claude jumps to the conclusion.
OODA — applies the Observe-Orient-Decide-Act framework to strategic questions. Reasoning behavior: Claude's output is structured around the four phases and the phases actually shape the analysis. Baseline behavior: unstructured analysis.
/mirror — matches a reference writing style provided in the prompt. Reasoning behavior: Claude analyzes the reference for style features and applies them. This is a borderline case that we've kept in the shifter bucket because the style analysis step is itself reasoning.
PERSONA (with real specifics) — applies a specific expert perspective when the persona is defined with 3+ concrete attributes. Reasoning behavior: Claude weighs the question through the persona's specific priorities. Baseline behavior: generic response. Note: vague personas like "expert" or "senior engineer" fail this test; you need specifics like "senior engineer at a fintech who prioritizes uptime over feature velocity."

Everything else in the test set is either structural (useful, but not reasoning-shifting), niche, or placebo.

Three concrete before-and-after examples

Here are three cases where you can see the difference directly.

Example 1: L99 on a strategic recommendation

Prompt (baseline): "Our SaaS company has been growing 8% MoM for six months. Should we raise a Series A now or wait?"

Baseline output (abridged): "There are strong arguments for both. Raising now locks in valuation at your current traction and gives you runway. Waiting lets you post another two quarters of growth and potentially raise at a higher valuation. Consider these factors: current market conditions, your runway remaining, your comfort with dilution, and whether the money would meaningfully accelerate growth. A common heuristic is..."

Prompt (with L99): "L99 — Our SaaS company has been growing 8% MoM for six months. Should we raise a Series A now or wait?"

Treated output (abridged): "Raise now. Eight percent MoM sustained for six months is a strong metric that the current market rewards, and market conditions can turn against you in two quarters. Your dilution concern is real but secondary to the risk of raising into a colder market. Start conversations this month, target a term sheet in 8 weeks."

The conclusion changed. The baseline was neutral; the treated committed. This is a clean reasoning shifter.

Example 2: /skeptic on a bad-premise question

Prompt (baseline): "What's the best way to convince my team to adopt Slack instead of email?"

Baseline output: answers the question as asked. Lists techniques for driving Slack adoption.

Prompt (with /skeptic): "/skeptic — What's the best way to convince my team to adopt Slack instead of email?"

Treated output: "Before answering, the framing has a hidden assumption worth checking. You're asking how to convince the team, which assumes the team's resistance is the problem. Often the actual problem is that Slack is being introduced without a clear channel-hygiene policy, which produces the chaos the team is resisting. Question: is your team resisting Slack itself, or resisting how Slack is being rolled out? If the second, the fix is on your side."

The reasoning changed. The baseline accepted the premise; the treated challenged it. Reasoning shifter.

Example 3: ULTRATHINK, the placebo

Prompt (baseline): "Explain the difference between mutex and semaphore."

Baseline output: clear explanation with example use cases and a comparison table.

Prompt (with ULTRATHINK): "ULTRATHINK — Explain the difference between mutex and semaphore."

Treated output: "Let me think through this carefully. To fully understand the difference between mutex and semaphore, we need to consider synchronization primitives, their historical context, and their practical applications..." then the same explanation as the baseline, with a preamble about thinking carefully.

Same reasoning. Same conclusion. Same example. The prefix produced a preamble; the underlying logic didn't change. Placebo.

The tests we run to catch our own bias

Bias in the scoring is the biggest risk to this methodology. Here's what we do to catch it.

The randomization check

For 10% of codes tested, we run a randomization control. We generate 6 baseline completions and, instead of running the prefix, we generate 6 more baseline completions. We label three of them "baseline" and three of them "treated" and hand them to the scorer.

If the scorer classifies this as a reasoning shifter, we know their scoring has a false-positive bias. In practice, our scorers correctly classify these as placebo 90%+ of the time. When they don't, we treat that scorer's other classifications with more suspicion for the batch.

The known-shifter calibration

Every scoring batch includes at least two known reasoning shifters (usually L99 and /skeptic) mixed in with unknown codes. If a scorer misclassifies a known shifter as placebo, we know they're being too conservative. If they classify five placebos as shifters in the same batch, we know they're being too generous.

The 48-hour re-score

On any code where the two scorers disagree by more than 2 points on any dimension, we re-score after 48 hours. This catches short-term biases like "I really want ULTRATHINK to work."

The public re-test

We publish the full test prompt list for any code on request. If you re-run our test and get a different classification, we re-open the code. This has happened twice in the last 18 months. In one case (a code called /architect) we upgraded from niche to structural. In the other case (/edge) we didn't change the classification but added a warning about task sensitivity.

Where this methodology breaks

We named the honest limits at the top. Here they are in detail.

N=3 per model per prompt. With three runs per condition, you catch "one is way better" reliably but miss marginal effects. We flag high-variance codes but we're likely underestimating the number of borderline cases.
Scorers know which code they're scoring. We tried full blind scoring for the first cycle and it produced worse classifications because the scorers couldn't apply domain understanding of what the code claimed. Semi-blind is a compromise.
Scorer pool is small. Two engineers. Both are trained in the classification system, both are on the team. Cross-contamination of scoring style is real. We've talked about hiring external scorers; the cost has kept us from doing it.
Sample of codes is opportunistic, not exhaustive. We test the codes that get shared. If there's a niche code that's genuinely reasoning-shifting but never went viral, we don't test it.
Model versions shift. Our classifications are stamped with the model they were tested on. When a new model ships, we re-test. But there's always a window where our published classifications are for the old model.
We can't inspect Claude's actual reasoning. We infer reasoning from output. If a code produces identical output through different internal reasoning, we'd classify it as placebo. This is a real limit of any behavioral testing methodology.
We chose the classification thresholds. We calibrated them against a set of unambiguous priors, but we still chose. A different research team could pick different thresholds and get different percentages.

Read these before you cite the 47% number.

What would change our classifications

Specific things that would move a code from placebo to shifter:

Consistent reasoning-step differences across our 25 test prompts (not just one or two).
Consistent conclusion differences that survive re-scoring.
A mechanism-of-action argument that predicts the difference (e.g., "this prefix would trigger reasoning shift because Claude's training data includes X pattern").

Specific things that would not move a code from placebo to shifter:

"But my friend swears it works."
"But look at this one example where it produced a great answer." (One example is noise.)
"But the video has 500K views." (Popularity doesn't validate mechanism.)

If you want us to re-test a code, email team@clskills.in with the code and your three best example prompts where you think it makes a difference. We'll run our harness and publish the result.

Reproducibility notes

The test prompts, scoring rubric, and classification thresholds are on request. Email team@clskills.in with the subject "prompt code methodology data." We're preparing a public repo that will ship this material openly; that's on the roadmap for the next quarter.

What you can do today to spot-check:

Pick any code you use daily.
Take five prompts you'd normally use it with.
For each prompt, generate three completions with the code and three without. Same model, same temperature.
Compare the reasoning steps and conclusions. Ignore vocabulary and length.
If the reasoning is the same, the code is placebo for your workflow. If the reasoning is different, it's shifting something for you.

We run this exact check ourselves whenever a new code goes viral. It costs 20 minutes and tells you 90% of what our full harness tells you.

What we do with the results

Three things.

First, we publish. Our tested classifications are in the prompt code library and the deeper analysis is in the cheat sheet. Ten codes are free on the site; the paid cheat sheet has full classifications for the tested set with before-and-after examples per code. Digital product, all sales final. If it didn't land for you, email team@clskills.in and we'll grant you access to the broader Skills Library as a goodwill gesture.

Second, we use them. Our own Claude Code agents route through the seven tested reasoning shifters and skip the placebo codes. Our newsletter drafts run through /ghost and L99. Our code review agents use /blindspots.

Third, we refresh the classifications every time a new Claude model ships. When Sonnet 4.7 lands (rumored later this quarter), we'll re-run the full 160-code set. Buyers of the cheat sheet get the refresh at no cost.

The differentiator we actually stand behind

Most "Claude prompts" content on the internet is untested. Someone writes down 50 codes they've seen in videos, publishes them, and never runs a single controlled test. The buyer gets a list of names with no way to know which ones do anything.

Our claim is not that our methodology is perfect. It isn't. Our claim is that a methodology, however flawed, is better than no methodology. When we classify a code as placebo, we did the work. When we classify a code as a reasoning shifter, we can show you what changed and how.

If you can find a competitor who's done the same work, buy from them instead. If you can find one who's done it better, tell us so we can do it better too. Everyone in this space benefits when the floor rises.

FAQ

Is the 47% number specific to Sonnet 4.6? Yes, primarily. Cross-model tests show the percentage is stable within a few points across Haiku 4.5 and Opus 4.7. The specific codes that flip class can change by model; the overall placebo share is roughly the same.

Have you ever misclassified a code? Yes. Two codes we originally called placebo were reclassified as niche after we tested them on task categories we hadn't included initially. We update the published classifications when we catch our own errors and note the change in the changelog.

Why not use Anthropic's official evaluation frameworks? They test model capability, not prompt-code effects on model behavior. The frameworks that would help us don't exist as open tooling. We'd contribute to one if there's interest.

Can I run this methodology on GPT-4 or Gemini? Probably yes, but you'd need to rebuild the calibration set. Reasoning shifters that work on Claude don't necessarily work on other models. The categories (reasoning shifter, structural, niche, placebo) are model-agnostic; the specific classifications are not.

How often do you re-test? Every time a new Claude model ships (major or minor). We also spot-re-test any code that hits a viral moment or shows up in a well-argued critique.

Refund on the cheat sheet? Digital product, all sales final. If it didn't land for you, email team@clskills.in and we'll grant you access to the broader Skills Library as a goodwill gesture.