Research papers, myth by myth, Part 12: 'Nobody really knows how people use AI, and it is mostly replacing jobs'

Ask ten people what AI is doing to work and you will hear ten confident answers: it writes all the code now, it is replacing call centres, it is a toy for students. Ask them where the numbers come from and the room goes quiet. Surveys ask people what they think they do. Forecasts guess what models might do. Very little measures what people actually type into these systems, because those conversations are private.

The myth: “Nobody really knows how people use AI, and it is mostly replacing jobs.”

The bottom line: there is aggregate evidence now, gathered without anyone reading conversations, and so far it shows more people working with AI than handing whole tasks to it. The first Anthropic Economic Index (published February 2025) classified 57% of task-related use as augmentation and 43% as automation. The split has moved since, in both directions, and it describes one provider’s users, so it settles less than either side of the argument would like.

What do I need to know first?

Start with something familiar: a national census. The statistics office collects a form from every household, which is deeply personal. It does not publish the forms. It codes the answers into categories, counts them, and publishes tables. Before release, it checks for any table cell so small that it might describe two or three identifiable households, and suppresses or rounds it. You learn a great deal about the country and nothing about your neighbour.

That is the design of the system behind all three papers in this part:

The household form is a conversation with an AI assistant: private, and never shown to a person.
The census questions are facets: fixed questions a model answers about each conversation, such as its topic, its language or how many turns it had.
Coding answers into categories is clustering: grouping conversations whose facets mean similar things.
Suppressing small cells is a threshold: a group is dropped unless it contains enough distinct accounts and conversations.
The published table is the aggregate view, which is all a human analyst sees.

Two more terms for the work question. Automation means the AI does the task with little human involvement. Augmentation means the person and the AI do it together. Picture a friend and your CV. “Write my CV” is automation. “Check my CV for mistakes”, “explain what recruiters look for” and “let’s improve this draft together” are augmentation. Anthropic splits these into five patterns: directive (hand over the whole task) and feedback loop (do it, see the result, report back) count as automation; task iteration, learning and validation count as augmentation.

What did the researchers do?

Paper 1: Clio (December 2024)

Alex Tamkin and colleagues published Clio: Privacy-Preserving Insights into Real-World AI Use, with an Anthropic write-up. The name stands for “Claude insights and observations”, and the authors compare it to Google Trends: aggregate patterns, no individual searches. (A note on the page says that as of August 2026 the tool is called Anthropic Insights.)

The pipeline, step by step, with Claude doing every step:

Extract facets. For each conversation, Claude answers questions such as “what is the topic?”, and is told to leave out private details.
Cluster. Each summary becomes an embedding, and k-means groups similar ones.
Apply thresholds. Clusters with too few unique accounts or conversations are thrown away, so a topic specific to one person never surfaces.
Describe and audit. Claude writes a title and summary for each cluster, again told to exclude private details, then a separate auditing prompt removes any cluster whose summary still contains them.
Build a hierarchy that analysts can browse, from broad areas down to narrow topics.

The paper explains why it uses layers instead of a formal guarantee: rich text summaries make differential privacy hard to apply, so it takes “a statistical and empirically validated approach”, with the layers acting as defence in depth.

how Clio turns private conversations into an aggregate view: Claude extracts facets while leaving out private details, groups similar conversations into clusters, drops clusters with too few accounts or conversations, names and audits the rest, and only then does a human analyst see anything

Paper 2: the Anthropic Economic Index (February 2025)

The Anthropic Economic Index launched on 10 February 2025 with the paper Which Economic Tasks are Performed with AI? Evidence from Millions of Claude Conversations. The team pointed Clio at about one million Claude.ai Free and Pro conversations from one week in December 2024 and asked a different question: which job task does each conversation represent?

For the list of tasks they used O*NET, the US Department of Labor’s catalogue of about 20,000 work tasks. A model first screened out conversations unrelated to any occupation, then walked down a hierarchy of tasks to pick the best match. For privacy, any task with fewer than 5 unique accounts or 15 conversations was left out. They released the resulting dataset publicly.

Paper 3: Values in the wild (April 2025)

Saffron Huang, Esin Durmus and colleagues published Values in the Wild (write-up: Values in the wild), later a COLM 2025 conference paper. Same privacy-preserving system, a third question: when a conversation calls for a judgement, which values does Claude express? They sampled 700,000 Claude.ai Free and Pro conversations from one week of February 2025, mostly with Claude 3.5 Sonnet, kept only the subjective ones, and analysed 308,210 conversations, about 44% of the sample.

What did they find?

Clio: what people actually ask

Across one million conversations, the biggest single cluster was web and mobile application development, at over 10%. Teaching and learning came to more than 7%, and business strategy and operations nearly 6%. Below that sat thousands of small clusters, including dream interpretation, analysis of football matches, disaster preparedness, crossword hints, Dungeons and Dragons, and counting the r’s in “strawberry”. Usage also varied by language; conversations in Japanese discussed elder care and ageing populations more often than average.

The privacy evidence is the part I would read first. In a sample of 5,000 conversations, about 10% of the raw conversations contained private information by the auditor’s rubric. By the time cluster summaries reached an analyst, the audit found none. Clio also reconstructed the topic mix of a synthetic test set with 94% accuracy. And it found things single-conversation checks miss, such as a network of automated accounts generating search-engine spam where no individual conversation broke the rules.

The Economic Index: tasks, not jobs

Here is the occupational picture, with one worked comparison. Of every 100 task-related conversations in December 2024, 37.2 were in the “computer and mathematical” group (software changes, debugging, network troubleshooting). Of every 100 US workers, about 3.4 are in that group (using the May 2023 US employment figures included in the released dataset). So coding-type work was roughly eleven times more common in Claude conversations than in the workforce. The second group, “arts, design, sports, entertainment and media”, was 10.3%, mostly writing and editing. At the other end, “farming, fishing and forestry” was 0.1%.

share of Claude.ai conversations by occupational group in the first Economic Index sample, December 2024, next to each group's share of US workers: computer and mathematical tasks are 37.2% of conversations but 3.4% of workers, while transport, food service and farming barely appear

Two findings bear directly on the “replacing jobs” half of the myth:

Depth was shallow and wide. About 36% of occupations used AI for at least a quarter of their tasks, but only about 4% used it for three quarters or more. The write-up says plainly that there was no evidence in this dataset of jobs being entirely automated.
Augmentation led. 57% of task use looked like augmentation and 43% like automation.

A small piece of arithmetic explains a number you will see in later reports. Some conversations fit neither pattern. Later reports restate the first sample as 55% augmentation and 41% automation, with the rest unclassified. If you drop the unclassified 4 and rescale, 55 / 96 is 57% and 41 / 96 is 43%, the original headline. Same data, two ways of counting.

What changed in later reports

Anthropic has repeated the analysis, which is where it gets interesting:

September 2025 (AI’s role in the US and global economy), August 2025 data: “directive” use, where people hand over the whole task, rose from 27% to 39% since December 2024, and “for the first time, automation (49.1%) has become more common than augmentation (47%)”. The same report added business traffic from Anthropic’s own API (programming interface) for the first time: 77% of those conversations showed automation and 12% augmentation.
January 2026 (New building blocks for AI use), November 2025 data: augmentation back ahead on Claude.ai at 52% to 45%, though the authors add that over a longer time frame automation’s share is still slowly rising. Computer and mathematical tasks were about a third of Claude.ai conversations and nearly half of API traffic.
March 2026 (Learning curves), February 2026 data: augmentation “increased slightly” in both Claude.ai and API traffic, and automation fell sharply in the API data. The write-up gives no single headline split.
June 2026 (Cadences): the first results from an Economic Index Survey linked to usage data, so researchers can compare what people say about their jobs with how they use the tool.

augmentation versus automation in each Anthropic Economic Index sample: augmentation led in the first samples, automation edged ahead in August 2025, augmentation led again in November 2025, and business API traffic is mostly automation

Values in the wild: what the assistant stands for

Claude’s most common values were service values. In the paper’s table, as a share of subjective conversations: helpfulness 23.4%, professionalism 22.9%, transparency 17.4%, clarity 16.6% and thoroughness 14.3%. People’s own most common values were more varied: authenticity, efficiency and clarity led. The paper found 3,307 distinct AI values in all, grouped into five families: practical, epistemic, social, protective and personal.

The response types are the most telling numbers. In the subjective conversations, Claude strongly supported the user’s values in 28.2%, mildly supported them in 14.5%, acknowledged them neutrally in 9.6%, reframed them in 6.6%, mildly resisted in 2.4% and strongly resisted in 3.0%. Those six add up to 64.3%, the share where people expressed values at all. Strong resistance happened when, for example, a user asked for unethical content or expressed moral nihilism. The authors also note that some value mirroring may be plain sycophancy, and say they cannot tell which from this data.

One link back to Part 11: rare clusters with values such as “dominance” and “amorality” turned out most likely to be jailbreak attempts, which suggests this kind of monitoring can spot attacks that pre-release tests missed.

So is the myth true?

The first half is false and the second half is not supported by this evidence.

“Nobody really knows” is no longer true. There is large-scale, privacy-preserving, repeated measurement of real use, with open datasets and stated methods. It is aggregate and it is from one provider, but it exists.

“Mostly replacing jobs” does not match what these reports measure. In most Claude.ai samples so far, augmentation has led automation, but narrowly, and automation briefly led in August 2025. Business API traffic leans heavily towards automation. Most importantly, these studies measure tasks in conversations, not jobs gained or lost. A rising automation share is a signal worth watching; it is not a count of redundancies.

What did the paper not show?

The authors are unusually direct about the limits, and they matter more here than in most parts of this series.

One provider, one kind of user. Clio and the first index use Claude.ai Free and Pro only, not Team, Enterprise or (until September 2025) the API. Clio’s paper says its insights “may not generalize to other AI systems”. The index write-up adds that Claude is marketed for coding, so coding may be overrepresented, and that the data are not “a representative sample of AI use in general”.
Short windows. Each analysis is roughly one week of conversations.
Work or hobby? Someone asking for editing help might be writing a report or a novel. A model filters out non-work conversations, but cannot be certain.
What happens after the chat is invisible. A memo written by Claude looks like automation, even if the user then rewrites half of it, which would be augmentation. Clio’s paper lists “limited to conversational data” as a fundamental limit: it sees conversations, not real-world effects.
Claude grades Claude. The model doing the classifying is the model being studied. The values paper names the risk of bias towards finding behaviour close to its own principles.
Privacy is empirical, not proven. Clio relies on tested layers, not a formal guarantee, and Anthropic’s write-up says the system “might not catch certain kinds of private information”.
Rare events disappear by design. Thresholds that protect privacy also hide anything that happens only a few times, including a single serious misuse.

Why does this matter if you build or use AI?

Look at usage in aggregate, not transcript by transcript. If you run an assistant for your own users, the same pattern (facets, clusters, thresholds, an audit, then a human view) gives you product insight without a team reading private chats.
Watch the automation share in your own logs. If users mostly hand over whole tasks, quality checks on outputs matter more; if they mostly iterate, response speed and editing tools matter more.
Treat a provider’s index as context, not your data. Your users, domain and channel will differ from Claude.ai’s.
Monitor values and behaviour after launch. Pre-release evaluation catches what you thought to test; aggregate monitoring catches what you did not.

How do I lock it in?

Term              What it is                                         Census picture
Clio              Claude summarises and clusters chats privately     the statistics office
Facet             a fixed question answered per conversation         a question on the form
Clustering        grouping chats with similar facets (k-means)       coding answers into categories
Threshold         drop groups with too few accounts or chats         suppress tiny table cells
Audit             a model removes clusters with private details      checking tables before release
Aggregate view    the only thing an analyst sees                     the published table
O*NET task        one of ~20,000 US work tasks                       an occupation code
Augmentation      person and AI do the task together                 friend helps with your CV
Automation        AI does the task, little human input               friend writes your CV

My view: the Economic Index is most useful as a time series, and the swings are the story. Seeing directive use climb from 27% to 39% in eight months, then augmentation edge back ahead, tells me far more than the February 2025 headline alone. I would read any single split as weather and the trend over several reports as climate.

One honest gap: nobody outside Anthropic can audit Clio’s privacy layers or rerun the classification on the raw conversations, by design. The released datasets let you check the arithmetic but not the pipeline. I would like to see a comparable index from another provider, run on a different user base, before treating any of these splits as true of AI use in general.

What did we learn?

Clio lets a lab study millions of conversations by having a model extract facets, cluster them, drop small clusters and audit summaries, so people only ever see aggregates.
The first Anthropic Economic Index found 37.2% of task-related conversations in computer and mathematical work, against about 3.4% of US workers.
That first index classified 57% of task use as augmentation and 43% as automation, with AI spread thinly across many tasks rather than taking over whole jobs.
Later reports show the split moving: automation briefly ahead in August 2025, augmentation ahead again by November 2025, and business API traffic mostly automation.
Values in the wild found Claude’s most common values were helpfulness, professionalism, transparency, clarity and thoroughness, with strong resistance to users’ values in 3.0% of subjective conversations.
All of this comes from one provider’s users over short windows, measures tasks rather than jobs, and has Claude classifying its own conversations.

What comes next?

This is the last part for now. The series overview lists every myth and paper in one place, and I will add new parts as important papers come out.

Sources

Clio: Privacy-Preserving Insights into Real-World AI Use (Tamkin et al., arXiv, December 2024) and Clio: Privacy-preserving insights into AI use (Anthropic, published December 2024)
Introducing the Anthropic Economic Index (Anthropic, published February 2025)
Which Economic Tasks are Performed with AI? Evidence from Millions of Claude Conversations (Handa et al., arXiv, February 2025)
Anthropic Economic Index dataset (Hugging Face; the occupation chart uses release_2025_02_10)
Economic Index: AI’s role in the US and global economy (Anthropic, published September 2025)
Economic Index: New building blocks for AI use (Anthropic, published January 2026)
Anthropic Economic Index report: Learning curves (Anthropic, published March 2026)
Anthropic Economic Index report: Cadences (Anthropic, published June 2026)
Values in the wild (Anthropic, published April 2025) and the paper Values in the Wild: Discovering and Analyzing Values in Real-World Language Model Interactions (Huang et al., COLM 2025)
O*NET OnLine (US Department of Labor)

Opinions are my own. Every fact in this post comes from the public documentation linked above.