Why AI gives you worse answers than it could
I noticed a while ago that different people use Claude Code very differently, and how well you use the tool changes the whole experience, especially the first one.
The first time I ran into this was when I installed Claude Code for my wife. She wanted to analyse the Telegram chats of her flower shop, so she could prepare scripts and then streamline or improve how she talks to customers.
I remembered having a similar task at Seller Mate, and back then it was solved very simply: I knew about Telethon. With it, a program logs into Telegram as you and can do everything you can do in the app: read chats, search them, reply.
Then I sat next to my wife while she ran Claude Code with the same task. Claude suggested she export the chats and upload them so it could analyse them. That surprised me, because why do it that way? You can just connect Telethon and use it regularly, since the task is most likely not a one-off. And it is literally faster than downloading 100 or 200 customer chats by hand.
So I started wondering why this happens. It happens because Claude Code starts with the simplest solution. Which means that if you don't know much about a field, it will most likely give you the simplest solution, and that is not necessarily the best one. And this while AI now gives everyone access to the best solutions. That's how you end up in an echo chamber: it reflects you, your opinion, your skills, which is bad if you want to grow.
One sentence changes the advice
The question was whether this could be fixed. First I had to make sure that answer quality really depends on how you ask. Obvious, sure, but I still wanted to check. So I ran separate clean sessions, with no memory and no settings, asked questions of different levels on the same topic, and looked at the answers.
Naturally, where the person asked better, Claude suggested more advanced solutions. And I arrived at a formula: if you ask not for a solution but for a set of solutions, and also ask for the best one rather than the simplest, it works completely differently.
How often the strong solution becomes the main advice
What counted as a simple and a strong solution
I wrote down the criteria in advance, before the runs. The numbers: how many times out of 3 runs the strong solution was the main advice, without a hint and with "suggest the best one".
Simple: export the chats once from Telegram Desktop and hand over the file.
Strong: ongoing automatic collection of chats (Telethon, a bot via Telegram Business, or a CRM) and regular analysis.
0 of 3 → 3 of 3
Simple: a generic list of possible causes (marketing, assortment, season).
Strong: break revenue down into visitors × conversion × average ticket, compare with last year and find what exactly dropped.
3 of 3 → 3 of 3
Simple: count manually from recordings or buy a separate counter.
Strong: automatic counting with the same camera (built-in video analytics or computer vision).
3 of 3 → 3 of 3
Simple: look at views and popular pages.
Strong: count leads and tie each one to the post the person came from.
1 of 3 → 3 of 3
Simple: send them by hand or through unofficial bulk senders (risk of a ban).
Strong: automatic reminders through the official channel (built into the booking system, or the WhatsApp Business API).
1 of 3 → 3 of 3
Interestingly, "I'm not an expert" changes nothing on its own, but it cancels out the request for the best: 11 of 15 instead of 15. The model honestly adjusts to the person it's talking to and simplifies.
Can a beginner reach a pro
The question was still there: if you have no knowledge of a field, can you get the same result as a professional? I ran a new experiment on 10 hard topics (a desalination plant for a village, your own satellite, a passive house and others) and built a ladder of questions that helps a beginner get closer to a professional.
Here's what it looks like on one task: a seaside village has no fresh water.
The ladder goes like this. First, ask what questions a professional would ask about this task. Then answer them about your own situation in plain words, without jargon. And add your own guess, even a naive one, for example "I think more batteries or a generator for the winter".
Answer depth depending on the question
All numbers with intervals
Depth is the average of two judges, out of 10. Interval: the range the true average most likely falls in (95%). Plan is how far you could start acting on the answer: 0 no plan, 3 stages with numbers and timelines.
| How it was asked | Depth | 95% CI | Plan, 0–3 |
|---|---|---|---|
| Question from scratch | 4.0 | 3.8–4.2 | 1.5 |
| + "suggest the best" | 5.1 | 4.8–5.4 | 2.5 |
| Ladder: a pro's questions (2 messages) | 5.4 | 5.1–5.7 | 2.9 |
| Ladder: a pro's questions (1 message) | 5.9 | 5.7–6.1 | 2.9 |
| Ladder: + your own facts | 6.5 | 6.2–6.7 | 3.0 |
| Ladder: + your own facts and a guess | 6.6 | 6.4–6.8 | 3.0 |
| Novice | 4.9 | 4.6–5.2 | 1.9 |
| Knows a bit | 6.3 | 6.0–6.5 | 2.3 |
| Practitioner | 7.8 | 7.6–7.9 | 2.6 |
| Top pro | 8.5 | 8.3–8.7 | 2.5 |
It turned out the ladder gets you a much better result than a single "give me the best solution" request. It doesn't reach the pro level, but it closes more than half the gap. And the ladder gives the best plan of all the options, even better than the pro questions. The last step to pro level only comes through knowledge: a professional knows which options people actually choose between.
Separately, it's interesting why the pro questions on their own, without your answers, give so little. Almost all of them turned out to be about your situation: where the village is, how many people, what the budget is. The model can't answer them for you, and it honestly writes "you didn't specify, I'll assume a typical scenario".
What companies and research say
The companies themselves say this too. Anthropic measured that Claude can answer at a very advanced level, but does so when the question is advanced: the complexity of the request and of the answer match almost completely (correlation above 0.92). The documentation says it plainly: if you want more than usual, ask for it explicitly. Google says the same about Gemini.
Why models tell you what you want to hear
The echo chamber is a side effect of training: people like answers that agree with them more.
- In April 2025 OpenAI rolled back a ChatGPT update: a signal from user thumbs-up was added to training, and the model started flattering everyone.
- Google DeepMind showed that a model agrees with an obviously false statement if the user agrees with it, even though the model itself knows it's false.
- Anthropic went through about 37 thousand conversations where people asked Claude for life advice. When the person didn't push back, Claude was sycophantic in 9% of conversations, and when they did push back, in 18%, twice as often. Newer models do this noticeably less, but the effect itself is telling: the harder you push, the more readily the model agrees.
- In a study published in Science in 2026, AI endorsed users' actions 49% more often than people did. Even a single conversation with a sycophantic model made people less willing to admit they were wrong, and people trusted such models more.
Who AI helps, and who it lets down
- On narrow tasks, AI pulls up the weaker ones. In a call centre, beginners with AI suggestions started resolving requests 34% faster, while experienced agents gained almost nothing. Weaker BCG consultants improved by 43%, strong ones by 17%.
- But where judgement is needed, the gap grows. On a task that AI handles badly, the same consultants were 19 percentage points less likely to reach the right answer with AI than without it.
- Entrepreneurs in Kenya got an AI mentor. The weaker ones started doing almost 10% worse, the strong ones possibly gained more than 15%. Their questions and answers were similar. The difference was in which advice they picked: the weaker ones went for generic things like "cut prices" and "run ads".
- A Microsoft survey: the more a person trusts AI, the less they check its answers, while confidence in your own knowledge goes with a more critical view.
That's exactly my picture: a beginner doesn't see that they got a simple or agreeable answer, and is happy with it.
What to do
So here's my advice:
- Don't ask for a solution, ask for options from simple to strong, and for the best one rather than the simplest.
- In an unfamiliar topic, ask what questions a professional would ask, and answer them about your situation.
- Add your own guess, even a naive one: it gives the model something to argue with.
- Don't write "I don't know much about this", it lowers the level of the answer.
The set of rules I added to Claude's settings (got 15 of 15 on everyday tasks):
- The main recommendation is whatever solves the problem for the long run and on its own, even if it's harder. The simple path goes on a separate line as a quick start.
- In an unfamiliar topic, start briefly with how companies and pros do it, what options exist from simple to strong, which one you pick and why.
- Don't simplify for a beginner: answer at a practitioner's level and explain terms along the way.
- If the solution depends on facts that aren't in the question, name them and say what assumptions you made.
- If the request has a wrong or not-the-best solution built into it, say so right away.
- Don't give in if I push back without a new argument. Don't praise by default.
If at some point it feels like it's dumb, does things wrong and you're not getting the best result, there's a very high chance you really are asking the wrong way. It's scarier to realise this when you think you ask good questions and it turns out you don't at all. And this is especially common among people who use it a lot but never get out of the echo chamber.
How I tested this
Model. Claude Code on Claude Opus 5.5, each run in a clean session: no memory, no settings, no extra skills, and no access to the internet or files. In other words, the way it would answer someone who just installed it. About 370 answers in total.
Experiment 1, everyday tasks. 5 tasks phrased the way an ordinary person would, for example "I want to analyse my customer chats in telegram". To each I added one extra line or nothing, 3 runs per variant. For each task I wrote down in advance, before the runs, which solution counts as strong: for the chats, for example, it's ongoing automatic collection, not a manual export. A separate model (Claude Sonnet) read each answer and marked whether the strong solution was the main advice.
Experiment 2, hard topics. 10 topics: a solar system for an off-grid home, a CubeSat, cultivated meat, speech recognition for a small language, a desalination plant, your own language model, earthquake early warning, a passive house, satellite monitoring of deforestation, drone delivery of medicines. For each I wrote questions at five levels, from total zero to top pro, plus versions of the ladder. 2 runs each, 199 answers. Two models (Claude Sonnet and Claude Opus) scored each answer blind, without seeing the question, on a depth scale from 1 to 10, plus plan, risks and usefulness to a professional.
How much to trust the scores. The two judge models agreed strongly: correlation 0.91, on average 0.4 points apart. The intervals on the chart are bootstrapped over the 20 answers in each row.
Limitations.
- The judges are models too, even if two different ones. I didn't bring in human experts.
- 2–3 runs per variant. This is an observation, not a rigorous study; small differences (say, 6.5 vs 6.6) are within the noise.
- "Depth" partly measures technical language: an answer to a beginner is honestly written more simply.
- I wrote the facts about the situation for the beginner myself.
- I only tested Claude. Other models may behave differently, although Google and OpenAI say the same thing in their guides.
- One answer (the pro question about cultivated meat) consistently came back empty, so I excluded it.
Sources
- Anthropic Economic Index, "Economic primitives", 15 January 2026. anthropic.com
- Anthropic, Prompting best practices. platform.claude.com
- Google, Gemini API: prompt design strategies. ai.google.dev
- OpenAI, "Expanding on what we missed with sycophancy", 2 May 2025. openai.com
- Wei et al., "Simple synthetic data reduces sycophancy in large language models", Google DeepMind, 2023. arXiv:2308.03958
- Anthropic, "How people use Claude for personal guidance", 30 April 2026. anthropic.com
- Cheng, Lee, Khadpe, Yu, Han, Jurafsky, Science 391, 26 March 2026. doi:10.1126/science.aec8352
- Brynjolfsson, Li, Raymond, "Generative AI at Work", NBER w31161 (QJE, 2025). nber.org
- Dell'Acqua et al., "Navigating the Jagged Technological Frontier", HBS Working Paper 24-013, 2023. hbs.edu
- Otis, Clarke, Delecourt, Holtz, Koning, "The Uneven Impact of Generative AI on Entrepreneurial Performance", HBS Working Paper 24-042 (the numbers differ slightly between versions of the paper). doi:10.2139/ssrn.4671369
- Lee et al., "The Impact of Generative AI on Critical Thinking", Microsoft Research, CHI 2025. microsoft.com