Concepts
IndexNow and Bing submissions
What Raylit sends to IndexNow, how auto and manual submissions work, and how rate limits are handled.
IndexNow is an open protocol that lets a website tell search engines a page is new or has changed, instead of waiting to be recrawled. Raylit submits your pages to IndexNow, which shares them with Bing: the index ChatGPT, Copilot and Perplexity read from.
What gets sent
Each crawl flags pages that are First Seen or Content Changed (see page states). Flagged pages show as Pending in the domain's submissions table.
Raylit does not submit a URL that is noindex (a robots or bingbot meta tag, or an X-Robots-Tag header) or whose canonical URL points somewhere else. A redirect is not submitted either; the chain is stored on the URL, and the destination can be submitted when that page is indexable. Sending those URLs would ask Bing to index an address the site told it to ignore, which is the copy assistants would then cite.
Raylit does not send one request per page. It waits about a minute after a page is flagged so that the pages crawled meanwhile go out together, then submits them in batches of up to 10,000 URLs per request.
Auto submit
With auto submit on, pending pages go out on their own. With it off, they stay pending until you submit them. You can switch it in the domain settings at any time; pages already waiting are held as pending when you turn it off.
Manual submissions
From the submissions table you can select pages and submit them yourself, up to 100 pages at a time. Manual submissions count towards your plan's monthly allowance, shared across all your domains:
- The allowance resets at the start of each calendar month.
- Pages are counted when you submit them, whatever IndexNow answers.
- The Starter plan does not include manual submissions.
Submission statuses
| Status | Meaning |
|---|---|
| Pending | Waiting for the next batch, or for you to submit it while auto submit is off. On a domain that is not verified yet, the same rows are labeled Waiting for verification and nothing is sent until the key file passes. |
| Submitted | IndexNow accepted the page. |
| Failed | IndexNow rejected the request, or it could not be reached. |
| Rate limited | IndexNow asked Raylit to slow down. The submission is retried automatically. |
The most common failures are:
- 403 Forbidden: IndexNow could not read your key file, or it does not contain the key. If Raylit's own check passes, a firewall or bot protection is usually blocking IndexNow's fetcher. See Hosting the key file. If you verify with a redirect, that redirect must be a 301 or 302 to the hosted key URL. See Verify with a redirect.
- 422 Unprocessable: the URLs do not belong to the domain, or the key does not match.
- Server errors and timeouts are retried a few times before the submission is marked as failed.
Rate limits
IndexNow limits how often a domain can submit. When it answers with HTTP 429 (Too Many Requests), Raylit marks the submission as Rate limited and stops sending for that domain until the cooldown ends:
- If IndexNow says how long to wait, Raylit waits that long (up to an hour).
- Otherwise it waits a minute, doubling the wait on each further 429, up to an hour.
- Submissions waiting out a rate limit keep retrying for up to a day.
You do not need to do anything: pending pages go out once IndexNow accepts submissions again. Batching pages into few large requests keeps rate limiting rare.