Concepts
How crawling works
How Raylit discovers your pages, how often it crawls and what each page state means.
A crawl is how Raylit learns what is on your website. Each crawl discovers your pages, fetches them, and compares them with the previous crawl to decide which ones Bing should hear about.
Discovering pages
Every crawl starts from your sitemap: https://your-domain/sitemap.xml, or a custom sitemap address you set in the domain settings. Raylit follows sitemap indexes into nested sitemaps and collects the pages they list.
If the sitemap yields no pages at all, Raylit falls back to following links from your home page instead.
When following links, Raylit skips pages your robots.txt disallows and links it cannot fetch. Pages listed in your sitemap are always crawled, whatever robots.txt says, so leave out of your sitemap any page you don't want submitted. A sitemap page that cannot be fetched is recorded as Not Live.
Each crawl stops collecting new pages once the account is already monitoring as many pages as the plan allows. That limit is one allowance for the whole account, shared across every site; see Plans and limits.
When crawls run
Active domains are crawled once a day on every plan. Raylit checks for due domains once a day, so a crawl can start up to a day after it comes due.
- Crawl now on the domain page queues a crawl straight away. Only one crawl runs per domain at a time.
- Pause takes a domain off the schedule; Resume puts it back. You can still start a one-off crawl while a domain is paused, for example to check a fix.
Each domain's crawl history keeps your most recent finished crawls; how many depends on your plan.
Page states
After a crawl, every page has one of these states, describing what the crawl found at its URL.
| State | Meaning | Sent to IndexNow? |
|---|---|---|
| First Seen | The page was found for the first time. | Yes |
| Content Changed | The page's content changed since it was last sent. | Yes |
| Minimal Change | Only a tiny part of the page changed, such as a date or a counter (2% of its words or less). | Only while IndexNow has not accepted the page yet |
| Not Changed | The content is the same as last time. | Only while IndexNow has not accepted the page yet |
| Redirect | The URL redirects somewhere else. | No |
| Canonical Different | The page's canonical URL points to another page. | No |
| No Index | The page asks search engines not to index it, with a noindex robots meta tag or X-Robots-Tag header. |
No |
| Not Live | The page returned an error status or could not be fetched. | No |
Redirect, Canonical Different, No Index and Not Live pages are not meant to be indexed under their own URL, so Raylit never sends them. When such a page comes back (it is live again, or its noindex is removed), the next crawl picks it up like any other page.
AI View
AI View is the text an assistant can read from a page. Every plan includes it. The domain page summarizes how many pages are readable, and each page has a status. See AI View for the Human view and AI view toggle, JavaScript flags, and how the word count is kept in one place.
Technical details
Each crawled URL also keeps the signals Bing and AI assistants use to decide whether that address is the one to cite. The domain's Pages table can filter on redirects, a canonical that points elsewhere, and noindex. Open a page for the full panel.
| Signal | What Raylit stores | Why it matters |
|---|---|---|
| Final HTTP status | The status of the last response, after same-host redirects. | IndexNow should hear about a URL that returns a page. An error status means assistants have nothing to cite there. |
| Redirect chain | Every hop and its code. A hop on another host is listed and not requested. | Submitting a redirect asks Bing to index an address that only forwards elsewhere. Raylit skips it and can submit the destination when that page is indexable. |
| Canonical URL | The canonical URL, flagged when it points at a different URL. | A canonical elsewhere means this page is a duplicate. Submitting it splits signals, and an assistant can retrieve the address the site said not to use. Raylit skips it. |
| Meta robots and X-Robots-Tag | The tag and the header, and whether they say noindex or nofollow for every crawler or for Bing. | noindex asks Bing not to index the page, so Raylit does not submit it. nofollow does not block the submission; it only asks crawlers not to follow links on the page. |
| Hreflang count | How many alternate + hreflang links the page declares, including x-default. |
Each one is a language or region version. Assistants use them to choose which version to show. |
llms.txt
At the end of each crawl Raylit requests https://your-domain/llms.txt and https://your-domain/llms-full.txt, using the same limits as the rest of the crawl: public http and https only, on the domain's own host.
The domain page reports /llms.txt as Missing, Found with issues, or Valid. A valid file has an H1 title, a blockquote summary, H2 sections of Markdown links, stays within 100 KB, and its same-host links resolve. Raylit checks a sample of those links, not every one, and does not request links on other hosts. /llms-full.txt is optional; the page says whether it was found.
The same page offers a starter llms.txt built from the titles of pages already crawled. Copy it onto your site at /llms.txt and replace the summary with your own. The starter updates as crawled pages change. Nothing in Raylit has to be configured for the check.