All articles
Data integrity
10 min readNovember 1, 2026

Why We Built SearchVault on Authorized Data (And Why SERP Scraping Is Dying)

Google's crackdown on SERP scraping has made scraped ranking data unreliable. Here is why SearchVault is built entirely on authorized Google Search Console data.

Twelve months ago, checking where a client ranks on Google was routine. You fired a few thousand automated queries at the search engine, recorded the positions, and charted the line. Today, that line is increasingly fiction. Google has spent 2025 and 2026 systematically dismantling every mechanism that made SERP scraping work, and the industry built on top of it is running out of road.

At SearchVault, we made a decision early: we would build entirely on authorized Google data, the data Google gives you about your own site through Search Console. This post is about why. It is also an honest look at what scraping's collapse means for agencies, in-house SEO teams, and anyone who has ever signed off on a rank tracker.

Google's war on scrapers (2025-2026)

To understand why scraping is dying, it helps to see how quickly Google moved. In less than two years, it deployed technical, data, and legal weapons that together have made programmatic access to results pages unreliable, expensive, and legally dangerous.

January 2025: SearchGuard closes the door

Google deployed its SearchGuard system in January 2025, and it changed the game overnight. SearchGuard is a JavaScript-based anti-bot layer that sits on the search results page itself. It performs browser fingerprinting, analyzes mouse movement, tracks keyboard rhythm, and issues rotating cryptographic tokens that an automated client must solve before it can see results.

The era of "just send a GET request" ended on that date. Scrapers that worked for a decade either failed outright or had to emulate a human well enough to pass a system designed specifically to detect emulation. The cost and complexity of basic rank tracking jumped immediately, and they have only gone up since.

September 2025: ten times the requests

Then came September 2025, when Google removed the &num=100 parameter. For years, that parameter let tools request 100 results for a single query, which kept request volume manageable. Removing it forced scrapers to issue ten times as many requests to achieve the same coverage.

That is ten times the cost, ten times the rate-limit exposure, and ten times the surface area for fingerprint detection. The change did not just make scraping more expensive. It made it dramatically easier for SearchGuard to identify and throttle the machines doing it.

September 2026: poisoned SERPs

The most damaging capability in Google's arsenal arrived in September 2026: poisoned SERPs. For detected scrapers, Google now serves structurally normal results pages with correct layouts, real titles, and real URLs, but with systematically wrong rankings.

A poisoned SERP looks legitimate. It parses perfectly. And its numbers are fiction. This is the hardest blow yet, because it attacks the final assumption scrapers had left: that even if access was hard, the data you finally extracted was at least the real results. That assumption is gone.

An 80 percent drop in reliable retrieval

The combined effect has been severe. Across the major SERP API vendors, successful data retrieval has fallen by up to 80 percent. Tools that still function are increasingly returning the data Google chose to give them rather than the data Google shows users.

Worse, when vendors try to compensate, they often reconstruct "plausible" rankings statistically. The practical result is that two clients can look at the same keyword and see different positions, both of them wrong, and neither tool can explain why.

The courts enter: Google vs. SerpApi

Google has also moved the fight into the courts. In December 2025, it sued SerpApi, one of the largest SERP scraping providers, under the Digital Millennium Copyright Act, arguing essentially that SerpApi's customers circumvented technical protection measures to access Google's results.

SerpApi won its motion to dismiss in July 2026, which was welcome news for an industry that had braced for the worst. But the legal signal is unambiguous. Google will keep spending whatever it takes to shut scraping down, and one favorable ruling does not change the direction of travel. Lawsuits were the last escalation available; none of the earlier ones failed to arrive.

The scheduled shutdown: Custom Search JSON API

Most telling of all, Google is not only attacking scrapers. It is also shutting down the sanctioned routes. The Google Custom Search JSON API, one of the few officially licensed ways to query Google's index programmatically, is being retired on January 1, 2027.

There is no lawsuit, no motion to dismiss, and no appeal. The date is on the calendar. For any tool or workflow that depends on querying Google on behalf of someone else's keywords, the message is simply that the end is scheduled.

What scraped rankings were really telling you

It is worth being honest about this, because the SEO industry deserves better than nostalgia for a broken data source.

Scraped position data was never what it looked like. You were never looking at what users saw. You were looking at what an automated browser could observe: no personalization, no local variation, no consent flows, no AI-generated results assembled per session, nothing of the context that shapes a real search experience. Scraping measured a synthetic user, and the synthetic user does not exist.

Position was always an approximation. Now it is an approximation downstream of a system actively trying to deceive the instrument collecting it. When the data source itself is designed to lie to your tooling, no amount of parsing sophistication brings the truth back.

That is why we believe the future belongs to first-party data, and specifically to Google Search Console.

Why first-party data is the future

Search Console, or GSC, is the authorized alternative, and it is the only ranking signal that comes from Google itself. Its advantages are not marginal. They are structural.

Authorized by Google, zero legal risk

When a property owner verifies their site in Search Console, they are using the API Google built for exactly this purpose. There is no circumvention, no terms-of-service violation, no legal exposure, and nothing to defend in court.

Access cannot be poisoned, rate-limited, or litigated out of existence. The terms are written by Google, SearchVault operates entirely inside them, and so can every agency and in-house team that uses us, which is not something any scraping vendor can say today.

Real clicks, real impressions, real positions

GSC is not an estimate of what Google serves. It is a record of what Google actually served and what people actually did with it: real clicks, real impressions, real positions, taken from Google's own logs.

When Google reports that a query earned 4,820 impressions, that is the count from the logs. When it reports an average position of 11.4, that average is computed over real users in real sessions. No reconstruction, no statistical guesswork, no synthetic browser. This is the ground truth scraping always aspired to and never reached.

AI Overview impressions, included

The gap between scraped data and first-party data widened further in 2025, when Google began exposing AI Overview impressions inside Search Console.

For the first time, you can see how often a query surfaced your site inside Google's AI answer, as a separate count from the classic blue links. Scrapers cannot get this. An AI answer is assembled for a specific person in a specific session; a headless browser in a data center does not see what a human sees. In the AI search era, first-party data is not just more reliable. For many queries, it is the only window into visibility at all.

No provider instability, no per-query cost

Scraping vendors break when Google changes the results page, which in 2025 and 2026 meant breaking often. Authorized integrations do not carry that risk, because Google commits to the Search Console API the way it never committed to the open results page.

The cost model is healthier too. Per-query pricing starts to look absurd when a single client's keyword set runs to thousands of queries a day, and it collapses entirely when Google starts billing in legal fees. Authorized data is fixed-cost, predictable, and auditable.

The honest caveats

None of this means GSC data is perfect, and we will not pretend otherwise in our own product copy.

GSC aggregates by day. It reports average position, which rounds away the interesting extremes of a ranking distribution. It samples under some conditions, and it limits historical depth. And most importantly, it retains only 16 months of history; if you do not store the data yourself, it silently disappears.

Those are real limits. They are the price of data that is true. And every one of those limits is solvable with engineering, which is exactly what SearchVault was built to do.

How SearchVault differs from traditional rank trackers

The differences between SearchVault and a conventional rank tracker are not feature-set differences. They are architectural.

Built entirely on authorized Google data. There is no scraping layer anywhere in the stack. Not for rankings, not for competitors, not for anything. Every number in the product traces back to data Google authorized the property owner to access. We do not rent Google's doorstep; we read the meter.

Permanent data retention. SearchVault captures and stores GSC data continuously, so the 16-month window becomes permanent, compounding history. The longer you run, the more valuable the dataset becomes, which is the exact opposite of a scraping tool whose data becomes less trustworthy every quarter. Long-run trend analysis, year-over-year comparisons, and multi-year decay curves become possible for the first time.

AI visibility risk scoring. We run pattern analysis over GSC data to score each site's exposure to AI search displacement: impression shifts, click-through collapse in query clusters, and accelerating zero-click behavior. The goal is to flag an account weeks or months before revenue impact is visible in the headline numbers, and it only works because the underlying data includes the AI Overview impressions that scrapers cannot see.

Actionable intelligence, not raw rank numbers. A position is not a strategy. SearchVault is organized around what to do: opportunities sitting at striking distance, cannibalization between your own pages, content decay, data health, and measured impact of actions taken. Raw rank numbers were always a proxy. We try to skip straight to the decisions.

We are also explicit about what we are not. SearchVault does not claim to track exact keyword-level positions in the way the old rank trackers did, because that goal was always a mirage, and the instrument used to measure it is now actively poisoned. We track the numbers that are real.

The future: AI search visibility, zero-click analysis, GSC-based competitor intelligence

The next phase of this transition is already visible.

AI search visibility as the core KPI. Impressions inside AI Overviews, generative search engagement, and share of answers are becoming the metrics that matter. They are measurable today, but only through authorized channels.

Zero-click analysis. A growing share of searches end with the user satisfied without a click. That looked like a reporting hole under the old model. Seen through first-party data, it is an insight: if your brand is showing in the answer but nobody clicks through, that is a positioning signal, not a missing number. Measuring presence, clicks, and the gap between them is the new analytics discipline.

GSC-based competitor intelligence. We are investing heavily in competitor intelligence that works exclusively on authorized data: gap models built from your GSC performance, public signals, and the properties you manage, without ever querying Google on a competitor's behalf. It is a slower path than scraping their keywords, and it is the only one that still has a future.

None of this is speculative within SearchVault. The data layer is already first-party end to end, the retention engine is already compounding, and the AI visibility signals are already part of the product. If you want to see how it feels, the how it works page is a good place to start, and our GA4 integration write-up shows how we think about connecting first-party datasets into revenue.

Conclusion

The last two years have been a stress test for the SEO industry, and the conclusion is clear. Scraping delivered approximate numbers for decades, and Google has now made it both unreliable and indefensible in a single campaign.

The tools that survive will be the ones that stopped renting a view of Google's front door and instead built on the authorized, first-party data that Google provides to property owners. That transition is not a sacrifice. It is an upgrade, and it is the only upgrade available that does not depend on winning a war against Google.

We made our choice before the war started. If you manage large accounts, the question is no longer whether your scraped data will fail, but when, and what you will have left when it does.

SearchVault is built on authorized data, and it is requesting access from serious teams doing serious SEO.

See it with your own Search Console data

Request access and we will set up an account built around the properties you already own.