Inside a quarter-million AI research queries
An AI literature report can put together dozens of references. In eight weeks of public ScholarQA logs, we found a wide gap between the sources assembled and the paper links clicked.
84%
of report threads had no recorded paper-link click.
1 in 50
cited papers received a paper-link click, counting each paper once per report.
ScholarQA | 2 July–27 August 2025 | recurring example-like query excluded. How we counted ↓
A rare and open dataset
The Allen Institute for AI (Ai2), the nonprofit behind Semantic Scholar, released the Asta Interaction Dataset in February 2026: 259,038 query records collected between February and August 2025. It links questions with cited or listed papers, section expansions and paper-link clicks. Participation was opt-in. User identifiers are withheld.[1][2]
Asta then · 2025
Earlier versions of Asta cover Paper Finder, a ranked search interface, and ScholarQA, which generated cited reports with expandable sections.
Asta now · October 2026
Today’s Asta brings paper finding, literature synthesis and data analysis under one research assistant. These logs describe earlier tools, not current performance.[3]
Many AI tools produce literature-review-style summaries with citations. Access to their underlying query-and-click records is much rarer. That makes this release particularly valuable: it lets outsiders investigate actual interactions beyond demonstrations and aggregate usage claims. Its openness lets us inspect, question and extend the findings. It is a useful form of accountability for AI research tools.
What Ai2 found
Longer requests. ScholarQA queries averaged about 37 words, compared with just over five on conventional Semantic Scholar search.
More than search. Users assigned expert roles, pasted manuscripts and asked for help recovering half-remembered papers.
Evidence viewing rose. Viewing inline evidence increased by 27% from users’ first to fourth queries. This measures interaction with supporting excerpts in ScholarQA, rather than visits to paper pages.[4]
What our analysis adds
Our counts exclude the dominant repeated example-like question.
Brief searches and substantial drafts coexist. Median query length was 11 words for ScholarQA and 7 for Paper Finder; 1,361 queries exceeded 500 words.
Citations are sometimes requested after the writing. Some users pasted finished prose and asked for references Without changing the original text in any way
.
Reports can remain active for months. A report on what is topic modeling and its uses in libraries
had a section expansion 155 days after creation.
| Eight-week window | ScholarQA | Paper Finder |
|---|---|---|
| Threads analyzed | 36,678 | 46,381 |
| Threads with a paper-link click | 15.6% | 38.2% |
| Median papers cited or listed* | 28 | 69 |
| Median papers clicked, among clicking threads | 2 | 2 |
*Among threads with recorded outputs. Paper links lead to Semantic Scholar records.
How we counted
ScholarQA paper-link tracking began months after query collection. Counting earlier reports in the denominator would understate the click rate. Both cohorts therefore contain threads created 2 July–27 August 2025, when paper-link tracking was available for both tools; clicks are observed through the dataset’s end. “One in 50” is 20,098 clicked out of 1,003,850 distinct report–paper pairs, not a typical report or user. Broader example filtering and fixed follow-up checks left the pattern similar. Different tasks and users prevent us from attributing the gap to interface design. User identifiers are not public, so Ai2’s user-level revisit findings cannot be recomputed from these files.
The report as a working document
AI generated reports may become supporting documents in research. Something that researchers keep, revisit and use while developing questions or drafting text.
If a summary becomes a reference point for subsequent work, its selection of evidence and its wording matter well beyond the first reading. Researchers may carry its claims into their own writing. That makes checking the connection between a claim and its source an essential part of using the report.
Checking what a citation actually supports
A real paper can be cited beside a claim it does not support. Checking a citation means going to the source and asking: does this study provide evidence for what the report says?
Some users in the dataset asked the AI to add references to finished prose without changing the text. That instruction treats the claims as settled before the evidence is checked. It can encourage the tool to find papers that look relevant, while leaving an exaggerated or unsupported sentence intact.
Ask instead: Do these sources support each claim? Flag unsupported wording and suggest revisions.
A paper on the same topic may support a narrower conclusion or contradict the claim. Check the relevant passages yourself, then revise your sentence to reflect what the evidence actually shows.
Caveat: The logs cannot tell us how often this checking happened. Readers could inspect excerpts inside ScholarQA or consult papers elsewhere, and a click alone does not demonstrate checking. “One in 50” measures recorded paper-link clicks, not the probability that a source was verified.
Three habits researchers can adopt
When you use an AI literature report to plan a study or draft a paper, build these three habits into your workflow.
1
Check the claim
Before reusing an important statement, follow its citation and locate the supporting passage. Consult the relevant methods and limitations, and revise your wording if the evidence is narrower than the summary suggests.
2
Check the coverage
Search independently for missing perspectives and contrary findings. A long bibliography does not establish a comprehensive review.
3
Keep an evidence trail
Save the prompt, generation date, sources and notes on limitations alongside any report you keep. Link your writing to the sources you checked, so you and your coauthors can trace the basis of each claim.
