A literature review done by hand is weeks of searching, reading, and tracking citations. Most of that effort is mechanical — finding the relevant papers, following the citation graph, and spotting which work is actually influential — and it can be automated into a pipeline that leaves you with the reading and the judgment, not the bookkeeping. This guide consolidates the moving parts from our earlier research-automation posts: where to get scholarly data reliably, how to run citation analysis at scale, and how to wire it all into a repeatable pipeline.
Where to get the data (and why not to scrape Google Scholar)
Google Scholar has the broadest coverage, but it has no official API and aggressive anti-automation defenses, so a pipeline built directly on it is fragile and gets you blocked fast. The durable sources are the ones designed to be queried. The Semantic Scholar API is free, structured, and — critically — returns citation edges and influential-citation counts alongside metadata and abstracts. OpenAlex is a fully open catalog of works, authors, institutions, and citations with generous limits. Crossref resolves DOIs and reference lists. Between them you can go from a topic to a set of papers to the full citation neighborhood without scraping anything; reserve Scholar for the occasional coverage gap, not as your backbone.
Citation analysis at scale
Once you hold the citation graph, the analysis is graph work. Rank papers by citation count within your topic to find the anchors; use citation velocity (citations per year since publication) to separate durable, compounding work from a paper that spiked once and faded; and follow forward citations from a foundational paper to surface the newest work building on it — the frontier of the field. Two structural measures do the clustering a human reviewer spends days on by hand: co-citation (papers frequently cited together tend to belong to the same conversation) and bibliographic coupling (papers that cite the same sources are working on the same problem). Run either over your set and the field’s sub-topics fall out automatically. Semantic Scholar’s “influential citations” flag is a useful shortcut for weighting citations that actually matter versus perfunctory ones.
The research pipeline, end to end
Assembled, the pipeline is five stages. (1) A seed query resolves to an initial paper set. (2) Expand one or two citation hops out — backward to foundations, forward to recent work — to capture the neighborhood. (3) Deduplicate on DOI and enrich with metadata, abstracts, and author/institution data. (4) Rank by influence and cluster by co-citation to surface the anchors and the sub-conversations. (5) Export a reading list ordered by influence with the citation graph attached, so you read in the right order instead of alphabetically.
Run that on a schedule and it becomes a living review that updates as new work lands, rather than a snapshot that’s stale the month after you finish — genuinely useful for a fast-moving field where the relevant literature changes quarter to quarter. The NexGenData catalog on Apify includes actors that turn scholarly and citation sources into clean JSON if you’d rather not build and maintain the collection layer yourself.
Handling the messy parts: dedup, disambiguation, coverage
Two problems bite every scholarly pipeline. Author disambiguation — the same name across institutions, or one person under name variants — is why you should key on persistent IDs (ORCID, OpenAlex author IDs) rather than name strings. And deduplication across sources: the same paper appears in Semantic Scholar, OpenAlex, and Crossref with slightly different metadata, so normalize on DOI first and title-similarity second before you count anything, or your citation totals will double. On coverage, no single source is complete — Semantic Scholar and OpenAlex each miss different corners — so for a serious review, query at least two and union the results.
A worked example
Reviewing, say, retrieval-augmented generation: seed with the two or three canonical papers, pull their forward citations to get everything built on them, then rank that set by citations-per-year to surface what’s both recent and already influential. Cluster by co-citation and you’ll see the field split into its sub-conversations — retrieval methods, evaluation, long-context alternatives — each as its own reading list. Schedule the job monthly and new high-velocity papers appear at the top automatically, so your review never goes stale. What used to be three weeks of manual tracking becomes a maintained, ranked, clustered bibliography you can hand to a team.
We cover AI agents, automation, and the tools that make them work. Our mission is to make AI accessible to everyone.