IndexNow, instant indexing and crawl budget strategy in a real time web

IndexNow tells search engines what changed, and Google is not among them

A search engine only knows your page has changed when its crawler comes back and looks. For a shop whose prices move every day, that wait is the gap between what the page says and what the search result says. IndexNow closes part of that gap. It is a small, open protocol: when a page is added, updated or deleted, your site sends the address to a search engine straight away, instead of waiting to be visited.

Two facts frame everything in this article. First, IndexNow works for Bing, Yandex, Seznam, Naver, Yep and a few others, and one submission reaches all of them.¹ Second, Google does not take part. It said in 2021 that it would test the protocol, and its own documentation today still sends you to sitemaps and Search Console instead.²³ So the honest strategy has two halves. Tell the engines that listen, the moment something changes. And for Google, make every visit its crawler pays you count, which is what people mean by crawl budget.

I take the two halves in that order, with one invented shop running through the whole piece.

Worked example: the shop in this article

Shannon Cycle Supply is an invented bike parts shop in Limerick. Nobody real, every number made up. It sells about 14,000 products at www.shannoncycles.example, and its owner, Siobhán, reprices and restocks constantly. On a normal day around 600 product pages change: a price moves from €34.99 to €32.99, a set of brake pads goes out of stock, a new tyre range arrives. What she gives out about is simple: search results show yesterday's price, and shoppers who click leave annoyed.

How search engines find changes without being told

A crawler is software that fetches pages, the way a browser does, but for a search engine. It learns about addresses from links on pages it already knows and from sitemaps, which are files that list a site's URLs. Then it decides when to come back to each one. A popular page gets revisited often. A quiet page might wait weeks.

Google is frank about the pace. Its guidance on asking for a recrawl says crawling "can take anywhere from a few days to a few weeks", and that a request "does not guarantee that inclusion in search results will happen instantly or even at all".³ That is the nature of pull: the engine chooses when to look.

The cost runs both ways. A crawler that cannot know whether a page changed has to fetch it to find out. Bing put this plainly when it launched IndexNow: a direct signal reduces "the need for exploratory crawl to test if the content has changed".⁴ Exploratory crawl is knocking on doors to check whether anyone moved house. Usually nobody did.

Worked example: how long Siobhán's price change takes to show

Siobhán raises a pump from €19.99 to €24.99 on a Monday morning, with no signal from her side.

1. The page changes on her server at 9am Monday.
2. A crawler last fetched that page eight days ago. Product pages deep in a category are not popular enough to be fetched daily.
3. Its schedule says the next visit is due in about a week. Nothing tells it to come sooner.
4. Until then, the search result shows €19.99. Anyone who clicks finds €24.99, and that surprise loses the sale and a bit of trust with it.

The point is not the exact number of days, which I made up. The point is that, under pull, Siobhán has no lever at all.

What IndexNow is, in one request

IndexNow gives her the lever. It is an open protocol created by Microsoft Bing and Yandex and launched on 18 October 2021.⁴ The idea fits in one sentence: when a URL on your site is added, updated or deleted, you send it to a participating search engine, and that engine passes it on to every other participant.

The sharing is written into the protocol, not left to goodwill. To take part, a search engine must agree that URLs submitted to it are "automatically shared with all other participating search engines", and it must pass them on in a timely way.¹⁵ So you submit to one endpoint, and the rest hear about it.⁶

Who is listening today? The participants list published by indexnow.org names Bing, Yandex, Seznam (the Czech search engine), Naver (the South Korean one), Yep, the Internet Archive and Amazon's crawler, Amazonbot.⁷ Google is not on it.

Ownership is proved with a key. You make up a key, put it in a text file on your own site, and include it with every submission. The engine fetches that file to check that the sender controls the site.¹ No account, no password.

The diagram runs top to bottom. Your part ends at the second step. Everything after it happens on the engines' side, and the last two steps are the ones people forget: a submission is a strong hint, not an order. Bing's documentation says it straight: "Using IndexNow does not guarantee that web pages will be crawled or indexed by search engines."⁸ Yandex says the same of its own endpoint.⁹

Worked example: Siobhán's pump, with IndexNow

Same Monday, same pump. This time the shop sends the pump's URL to IndexNow at 9.02am, two minutes after the change is saved. Bing checks her key file, accepts the URL and passes it to Yandex, Seznam, Naver and the others. When Bing fetches it is still Bing's call, but Bing says submitted URLs are generally prioritised.⁸ On Google, nothing has changed at all.

Setting it up: the key file, one URL, many URLs

Three pieces, and each has to match the documented API, because the engines reject anything that does not.

The key and the key file

The key must be between eight and 128 characters long, using only the letters a to z and A to Z, the digits zero to nine, and the dash.¹¹⁰ A random 32 character hexadecimal string is the usual choice. Here is Siobhán's, invented for this example:

a3f9c2e17b4d48e6a0b5c9d2e8f1a7c4

The recommended place for it is a text file in the root of the site, named after the key itself, with the key as its only content, saved as UTF-8:¹

https://www.shannoncycles.example/a3f9c2e17b4d48e6a0b5c9d2e8f1a7c4.txt

If you cannot write to the root, you can host the file elsewhere on the same host and point to it with a keyLocation parameter. The catch is that a key file only covers URLs at or below its own folder. A file in /catalog/ can vouch for /catalog/ URLs but not for /help/.¹

One URL, with a GET request

For a single page, a plain GET request is enough. The url parameter must be URL encoded:¹¹⁰

curl "https://api.indexnow.org/indexnow?url=https%3A%2F%2Fwww.shannoncycles.example%2Fproducts%2Ftrack-pump&key=a3f9c2e17b4d48e6a0b5c9d2e8f1a7c4"

api.indexnow.org is the shared global endpoint. You could equally send it to www.bing.com/indexnow or yandex.com/indexnow. The result is the same, because of the sharing rule.⁶

Many URLs, with a POST request

For more than one page, you send a JSON body. The documented fields are host, key, an optional keyLocation, and urlList, an array of up to 10,000 URLs per request, with the content type set to JSON in UTF-8:¹¹⁰

curl -X POST "https://api.indexnow.org/indexnow" \
  -H "Content-Type: application/json; charset=utf-8" \
  -d '{
    "host": "www.shannoncycles.example",
    "key": "a3f9c2e17b4d48e6a0b5c9d2e8f1a7c4",
    "urlList": [
      "https://www.shannoncycles.example/products/track-pump",
      "https://www.shannoncycles.example/products/brake-pads-disc",
      "https://www.shannoncycles.example/products/gravel-tyre-40mm"
    ]
  }'

Every URL in urlList must belong to the host you name, or the request fails.¹

What the answer means

The response code tells you what happened, and it is worth logging every one:¹⁸¹⁰

• 200 means the URLs were received.
• 202 means received, with the key still being checked (normal for a new key).
• 400 means the request was malformed.
• 403 means the key file could not be found or does not contain the key.
• 422 means a URL does not belong to the host, or the key breaks the format rules.
• 429 means too many requests, which the engines read as potential spam.

A 200 confirms receipt, nothing more.

Worked example: wiring it into Siobhán's shop

Siobhán's shop runs on Ruby on Rails. Whenever a product's price, stock or text changes, the product's URL goes into a short queue. A background job, a task that runs outside the web request so the shop never waits on it, collects the queue every ten minutes and sends it in one POST:

require "net/http"
require "json"

class IndexNowJob < ApplicationJob
  ENDPOINT = URI("https://api.indexnow.org/indexnow")
  HOST = "www.shannoncycles.example"

  def perform(urls)
    urls.uniq.each_slice(10_000) do |batch|
      body = { host: HOST, key: ENV.fetch("INDEXNOW_KEY"), urlList: batch }
      response = Net::HTTP.post(ENDPOINT, body.to_json,
                                "Content-Type" => "application/json; charset=utf-8")
      Rails.logger.info("IndexNow #{response.code} for #{batch.size} URLs")
    end
  end
end

Walk through one day. Around 600 product pages change, some more than once, a price drop in the morning and a sell out in the afternoon. uniq removes the duplicates inside each ten minute window. There are 144 windows in a day, so most carry a handful of URLs, far below the 10,000 limit, and each_slice makes one request per window. That is the shape the protocol was built for.

If Siobhán's shop ran on a hosted platform instead, she might not need any code at all. Bing lists platforms with IndexNow built in, including Shopify, Wix Studio and Duda, plugins for WordPress such as Yoast and Rank Math, and Cloudflare's Crawler Hints setting for sites behind its network.⁸

What to send, and what to leave out

The engines say clearly how they want it used. Three rules cover nearly everything.

Send real changes only. Bing asks for URLs "added, updated, or deleted" since you started using IndexNow.⁸ Yandex asks you to "report only new, changed, or deleted pages".⁹ The IndexNow FAQ adds that you should skip "every small layout or cosmetic change" and focus on what matters, such as content, pricing and availability.⁶ A new footer is not news.

Send deletions and redirects too. When a product is discontinued and its page returns a 404 or 410, or when a page moves with a redirect, submit that URL. The FAQ says so directly, because it lets the engines drop or update the old address sooner.⁶

Do not resubmit the whole site. The FAQ says IndexNow "is not designed for submitting every URL on your site at once", with an exception for a recent migration or redesign.⁶ For a page that changes often, it suggests waiting at least five minutes between submissions of the same URL, and Yandex's reference asks for intervals of ten minutes if you must resubmit.⁶¹⁰ Every submitted URL also counts toward your site's crawl quota with those engines, so noise has a cost.⁶⁸

None of this replaces a sitemap. Bing's own advice in July 2025 was to combine the two: a complete sitemap for coverage, IndexNow for fast submission of each changed URL.¹¹ The sitemap is the full inventory. IndexNow is the change notice.

Worked example: Siobhán's first week, and the mistake I would stop

Siobhán's previous developer, keen to help (fair play to him), had a nightly task that submitted all 14,000 product URLs to IndexNow "to be safe". Of those 14,000, about 600 had actually changed that day. So roughly 13,400 submissions, about 96% of the batch, announced changes that had not happened, each one counting against the shop's crawl quota. A signal that cries wolf every night stops being a signal.

The fix is the job above: only the URLs whose price, stock or text changed, collected every ten minutes, plus discontinued products the moment they start returning 410, plus old URLs the moment they redirect. The nightly blast goes. The sitemap stays, with an honest lastmod date on every product.

Where Google stands today

A good few articles out there are too hopeful here. This is what can actually be stood up.

In November 2021, a month after the launch, Google told Search Engine Roundtable that it "will be testing the potential benefits of this protocol".² That is the last public statement I can find from Google about IndexNow. It is not on the indexnow.org participants list.⁷ Google's own page on asking it to recrawl a URL names two routes, the URL Inspection tool in Search Console for individual pages and a sitemap for many, and does not mention IndexNow.³ Five years on, a test that never turned into support is a no in practice, whatever happens next.

Google also removed its own push style shortcut. Until 2023 you could ping Google's sitemap endpoint to say a sitemap had changed. In June 2023 Google deprecated that endpoint, and the deprecation is now complete: requests to it return a 404, and code that still calls it does nothing useful.¹² If your shop or plugin still pings Google, it is shouting into an empty room.

What Google does use is the sitemap's lastmod date, the timestamp of a page's last change. Google uses it "if it's consistently and verifiably" accurate, and it should reflect "the last significant update to the page", not a new copyright year in the footer.¹³ Its 2023 announcement was blunter: report false dates and "eventually we're not going to believe you anymore".¹² Google ignores changefreq and priority entirely.¹³

There is a Google Indexing API, but it is not a general shortcut. Google says it can only be used for pages with a JobPosting or a BroadcastEvent embedded in a VideoObject, in other words job adverts and livestreams.¹⁴

Worked example: what Siobhán does for Google

For Google, Siobhán's lever is the sitemap. Her product sitemap regenerates every hour. The lastmod on each product comes from the moment its price, stock or description last changed, not from when the sitemap file was built. The sitemap is listed in robots.txt and submitted in Search Console. For the occasional urgent page, a recall notice for a faulty brake lever say, she uses URL Inspection by hand.

Then she makes Googlebot's visits count. That is crawl budget, and it is the rest of this article.

Crawl budget is real, but only for some sites

Google defines crawl budget as "the set of URLs that Google can and wants to crawl", and it treats every hostname as a separate site with its own budget.¹⁵ Two things set it.

The crawl capacity limit is how hard Google can crawl without hurting your server. Every site starts at a conservative default. If your pages respond quickly and reliably, the limit rises. If they slow down, throw 5xx server errors or return 429, it falls.¹⁵

Crawl demand is how much Google wants to crawl. It rises with the number of URLs Google knows about on your site (its "perceived inventory"), with how popular they are, and with how often they go stale. Google calls inventory "the factor that you can positively control the most", because every duplicate or useless URL it knows about is something it will try to crawl.¹⁵

Here is the part that is often skipped. Google's guide is written for large sites, roughly a million pages changing weekly, or medium sites of 10,000 pages or more changing daily, or sites with many URLs stuck in "Discovered, currently not indexed" in Search Console. Google calls these rough thresholds, not exact ones. If your new pages are usually crawled the day you publish them, Google says you do not need the guide at all: keep the sitemap current and check the indexing report.¹⁵ A 40 page brochure site is grand without any of this. Crawl budget is not its problem, and anyone selling you a crawl budget audit for one is selling you a problem.

Worked example: does Siobhán's shop qualify?

Siobhán has 14,000 products and about 600 change every day. That puts her in Google's second group, 10,000 or more pages changing daily. Her Search Console indexing report also shows around 2,100 product URLs as "Discovered, currently not indexed": Google knows they exist and has not got round to fetching them. Her site is the kind where crawl budget is well worth an afternoon.

Read your server logs to see where the crawl goes

Opinions about crawl budget cost nothing to hold and a lot to act on. Your server's access logs settle the matter. Every request a crawler makes is a line in that log: the time, the URL, the status code and the user agent, the name the client gives itself.

First, make sure a request claiming to be Googlebot really is, because anyone can type "Googlebot" into a user agent. Google's method is a reverse DNS lookup on the IP address, a check that the name ends in googlebot.com, google.com or googleusercontent.com, and then a forward lookup to confirm the name points back to the same IP. Or you match the IP against the ranges Google publishes.¹⁶ Skip it and scrapers in Googlebot costume pollute every number that follows.

Second, sort the verified requests by URL type and status code. Then one ratio tells you how much of the crawl is going to waste:

$$w = \frac{c_{\text{waste}}}{c_{\text{total}}}$$

Here $w$ is the waste share, $c_{\text{waste}}$ is the number of verified crawler requests in the period that went to URLs you never want in the index (filter combinations, internal search results, redirect chains, soft 404 pages), and $c_{\text{total}}$ is the total number of verified crawler requests in the same period. It is an ordinary rate, events over opportunities, explained in Foundations: ratios and rates.

Worked example: a month of Googlebot on Siobhán's shop

I take one month of Siobhán's logs, keep only requests verified as Googlebot, and only HTML pages. There are 180,000 of them. Sorted by URL type, they look like this.

The horizontal axis is the type of URL Googlebot requested. The vertical axis is the number of requests in the month. The takeaway is the first bar. The single largest consumer of Googlebot's time is filter combinations, URLs like /tyres?width=40&colour=black&sort=price, which are near copies of category pages. The product pages Siobhán actually wants fresh get less than a third of the crawl.

Now the ratio, step by step. The waste is the filter combinations, internal search, soft 404 pages and redirect chains added together: $c_{\text{waste}} = 81{,}000 + 12{,}000 + 12{,}000 + 9{,}000 = 114{,}000$. The total is $c_{\text{total}} = 180{,}000$.

So $w = 114{,}000 / 180{,}000 = 0.633$. Nearly two thirds of Googlebot's visits went to pages Siobhán never wanted in search. The same logs show filter pages taking well over a second to respond, and slow responses keep the capacity limit low.¹⁵ She is wasting a budget that is already being held down.

Fix the waste at the source, not with noindex

Each fix comes from Google's own guidance and matches a bar in the chart.

Filter combinations. Google's guidance on faceted navigation, the filter and sort options on category pages, says crawlers "will typically access a very large number" of these URLs, and that time spent there is time not spent on new, useful pages. If you do not need filtered pages in search, the strongest option is to disallow them in robots.txt. A canonical link pointing back to the main category helps less and more slowly.¹⁷

Internal search results. They are endless and duplicate the category pages. Google's guide says to block unimportant URLs like these in robots.txt.¹⁵

Soft 404 pages. A soft 404 is a page that says "not found" or "no longer available" but returns a 200 status, so the crawler keeps coming back. Google says to eliminate them, and to return a real 404 or 410 for pages that are gone for good.¹⁵

Redirect chains. A link that redirects to a URL that redirects again costs a request per hop. Google's guide says to avoid long chains.¹⁵ Point internal links and old redirects straight at the final URL.

Speed and caching. Faster responses raise the capacity limit, and supporting HTTP caching lets the server answer a crawler's "has this changed?" with a short 304 Not Modified instead of the whole page.¹⁵

Two things Google tells you not to do. Do not use noindex to save crawl budget, because Google still has to fetch the page to see the tag. And do not use robots.txt to shuffle budget around temporarily, only to block URLs you never want crawled. Google adds a detail worth reading twice: freed budget only moves to other pages if Google was already hitting your site's capacity limit.¹⁵

The diagram runs top to bottom, one URL type at a time. The first question splits everything into wanted and unwanted. Wanted URLs need nothing clever: speed, a sitemap entry and an honest date. Unwanted URLs get the bluntest true answer available, gone, moved or blocked, in that order of preference. One month's logs are the measurement, the next month's are the check.

Worked example: Siobhán's shop, a month later

Here is what I would change, in order, and the invented result.

1. Filter and sort parameters on category pages go into robots.txt as disallowed. Each category keeps one unfiltered, crawlable version.
2. /search goes into robots.txt.
3. Discontinued products, which used to show "sorry, no longer available" with a 200, now return 410 and are submitted to IndexNow the day they go.
4. About 900 old redirects that hopped twice now point straight at the final page.
5. Filter page responses get cached, and product pages return 304 when nothing has changed since the crawler's last visit.

The next month's verified Googlebot log has 135,000 HTML requests. The robots.txt change went live a few days into that month, so some filter URLs were still fetched before it.

The axes are the same as before: URL type along the bottom, requests in the month up the side. The takeaway is that the chart has turned upside down. Product pages are now the tallest bar by far. The new waste is $c_{\text{waste}} = 3{,}000 + 2{,}000 + 1{,}000 + 0 = 6{,}000$, so $w = 6{,}000 / 135{,}000 = 0.044$, down from 0.633.

How much more attention did the product pages get? The percentage change:

$$\Delta\% = \frac{v_{\text{new}} - v_{\text{old}}}{v_{\text{old}}} \times 100$$

Here $\Delta\%$ is the percentage change, $v_{\text{new}}$ is the new value (product page requests after the fixes) and $v_{\text{old}}$ is the old value (product page requests before). The formula, and how it differs from a change in percentage points, is in Foundations: percentages.

Step by step: $v_{\text{new}} = 104{,}000$ and $v_{\text{old}} = 52{,}000$, so $\Delta\% = (104{,}000 - 52{,}000) / 52{,}000 \times 100 = 100\%$. Product pages got twice the visits, from fewer requests in total. Total crawl went down, which is fine: there is less junk to look at.

Two honest notes. This works because Siobhán's shop was in Google's target group and held at its capacity limit. On a small, fast site the same changes would tidy the logs and little else, because freed budget only moves when you were at the limit.¹⁵ And over the following weeks, the 2,100 "Discovered, currently not indexed" products should start to come down. That is the number I would watch, not the crawl count.

How I run this, and what to expect

On keferboeck.com I do the same thing at a smaller scale. When I publish or update an article, a short script sends the new and changed URLs to IndexNow, and the sitemap carries the real date of the change. It is the job above with fewer URLs and no prices.

What to expect from IndexNow, honestly: a better chance of quick pickup on Bing and the engines that share with it, and no change at all on Google. How much that is worth depends on how much of your traffic comes through Bing's index, which is part of the story in how to rank in ChatGPT and AI search engines.

What to expect from crawl budget work: on a large or fast changing site, a crawl that goes where the money is, and fewer pages stuck waiting. On a small site, a tidy log and not much else, so do not pay for it. And if your crawl problem is really thousands of near identical pages you made on purpose, that is a different conversation, about programmatic SEO done with structure.

Put together, the strategy for a real time web is short. Tell the engines that listen, the moment something changes, and only then. Keep the sitemap honest for the one that does not. And read your logs before you believe anyone about crawl budget, including me.

Footnotes

1. IndexNow.org, Documentation, https://www.indexnow.org/documentation

2. Barry Schwartz, Google is testing IndexNow, Search Engine Roundtable, 10 November 2021 (secondary source, reporting Google's statement), https://www.seroundtable.com/google-indexnow-32396.html

3. Google, Ask Google to recrawl your URLs, Google Search Central documentation, https://developers.google.com/search/docs/crawling-indexing/ask-google-to-recrawl

4. Microsoft Bing Webmaster Tools team, IndexNow: Instantly Index your web content in Search Engines, Bing Webmaster Blog, 18 October 2021, https://blogs.bing.com/webmaster/october-2021/IndexNow-Instantly-Index-your-web-content-in-Search-Engines

5. IndexNow.org, Search engines: requirements for participating search engines, https://www.indexnow.org/searchengines

6. IndexNow.org, Frequently asked questions, https://www.indexnow.org/faq

7. IndexNow.org, Participating search engines (searchengines.json), https://www.indexnow.org/searchengines.json

8. Microsoft Bing, IndexNow: Get started, Bing documentation, https://www.bing.com/indexnow/getstarted

9. Yandex, IndexNow, Yandex Webmaster Help, https://yandex.com/support/webmaster/indexing-options/index-now.html

10. Yandex, IndexNow API reference, Yandex Webmaster Help, https://yandex.com/support/webmaster/indexnow/reference.html

11. Fabrice Canel and Krishna Madhavan, Keeping content discoverable with sitemaps in AI powered search, Bing Webmaster Blog, 31 July 2025, https://blogs.bing.com/webmaster/July-2025/Keeping-Content-Discoverable-with-Sitemaps-in-AI-Powered-Search

12. Gary Illyes, Sitemaps ping endpoint is going away, Google Search Central Blog, 26 June 2023, https://developers.google.com/search/blog/2023/06/sitemaps-lastmod-ping

13. Google, Build and submit a sitemap, Google Search Central documentation, https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap

14. Google, Indexing API quickstart, Google Search Central documentation, https://developers.google.com/search/apis/indexing-api/v3/quickstart

15. Google, Crawl budget management for large sites, Google Search Central documentation, https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget

16. Google, Verifying Googlebot and other Google crawlers, Google Search Central documentation, https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot

17. Google, Managing crawling of faceted navigation URLs, Google Search Central documentation, https://developers.google.com/search/docs/crawling-indexing/crawling-managing-faceted-navigation