Website indexing in Yandex, Google and Bing
Indexing is checked page by page: whether Yandex (Russia's largest search engine), Google and Bing have taken it into search, and if not, why. Here a script checks 18 pages every morning and submits changed ones for recrawl.
Published September 29, 2026
How to check your site's indexing yourself: eight steps
- Sitemap. Check that sitemap.xml lists every page that should be in search, and only those. Remove deleted URLs from it.
- Search consoles. Add the site to Yandex Webmaster (the search engine's site console), Google Search Console and Bing Webmaster Tools, and submit the sitemap to each. All three are free.
- Status of each page. In Yandex Webmaster, open “Pages in search”: excluded pages are listed there with the reason. In Search Console and Bing, use URL Inspection.
- Daily log. Once a day, record the status of each page in a table. Without history you cannot tell a page that dropped out from a page that was never in search.
- The reason, not the fact. If a page is not in search, look at the reason. In our case, a “low-value or low-demand page” was fixed by a title matched to real queries in Yandex Wordstat, the search engine's keyword-demand tool, not by new text.
- Submit after an edit. After you edit a page, submit it for recrawl in Yandex Webmaster and Bing and announce the change via IndexNow. Do not submit the whole site every day: the quota is capped.
- Dropped means an event. If the count falls, open the event history in Yandex Webmaster. A removal record means the page dropped out; no record means the sample fluctuated.
- Script. Once you have more than a dozen pages, ask an AI model to write a script that runs steps 3–7 through the search engine APIs every morning and sends a summary to your messenger.
Which approach fits which site
- A site under 20 pages — the consoles and a weekly check are enough; a script pays off when edits go out every day.
- A blog or catalog with hundreds of pages — check new and changed pages one by one, and the rest by the overall counter.
- A site on a website builder — the platform usually generates the sitemap itself; check that it is submitted to the consoles and read page status there.
Below is how it works here: 18 pages, three search engines and 24 days of daily snapshots.
Where the project is now
The first search numbers were collected on September 1, 2026; the per-page index snapshot and the submission of changed pages started on September 3. From then on the design changed only in response to failures: each rule below appeared on a day when a number in the report did not match what was actually happening. Since September 29 the morning report is built on the site's server and arrives even when the laptop is off. The site and the dashboard are still updated from the laptop, because that is where pages are checked in a browser before deploy.
What was needed
The requirement was simple: every morning, know for each page of the site whether Yandex, Google and Bing have taken it into search, and if not, why, without opening three consoles. A page leaves search without notice, and the only way to catch it is a snapshot taken before and after. The constraint that shaped the design: search engines do not keep per-page history. Google returns only today's status of a URL. The Yandex sample includes pages Yandex has not decided on yet. Bing does not say what is in its index and only shows whether its crawler has opened a URL. So the history is kept locally, one row per page per day, and any “dropped” conclusion is checked against an event the search engine itself logged.
| Method | What we took from it | Canonical name · source |
|---|---|---|
| One owner for the data: the URL list comes only from the live sitemap | Yandex kept the tag pages deleted on Sep 15 in search for another week, and without this rule the dashboard would have shown 20 pages out of 18. The count covers only the URLs the site itself gives to search engines. | Single source of truth; the DRY principleHunt & Thomas, “The Pragmatic Programmer”, 1999 |
| Failure isolation: if one engine does not respond, the others are still captured | Yandex, Google and Bing are queried separately, each with its own retries: Google up to three with a minute between them, Bing tries a URL again after a failure. A failure in one engine does not stop the snapshot of the others, and a snapshot failure does not stop the site deploy. | Bulkhead and graceful degradationNygard, “Release It!”, 2007 |
| Loud failure: a failure of the numbers, the build or IndexNow stops the deploy | If the search numbers were not collected, the site did not build, the check failed or IndexNow rejected the request, the deploy stops and the owner gets the message “site untouched”. A site quietly deployed with stale numbers is worse than one not deployed at all. | Fail fastShore, “Fail Fast”, IEEE Software, 2004 |
| A gap is not zero: the engine did not respond, the cell shows “?” | No response means no measurement, not zero pages in the index. The report says “no response” and does not compute the difference from the day before. | Separating missing values from zeroRubin, “Inference and missing data”, Biometrika, 1976 |
| Measure before explaining: “dropped” only on a Yandex event | A page disappearing from the Webmaster sample is an observation, not a conclusion. A page counts as dropped only if the Yandex event feed records its removal with a reason; without an event it is “not taken yet”. | Falsifiability of a hypothesisPopper, “Logik der Forschung”, 1934 |
| Re-runs without duplicates: a second run in a day overwrites that day's rows | The snapshot can be re-run any number of times: today's rows are replaced, not appended. A morning network failure is fixed by running it again, with no manual cleanup of tables. | Idempotent operationsHelland, “Idempotence Is Not a Medical Condition”, ACM Queue, 2012 |
| Submit the difference: only changed pages go to search engines | The deploy compares files on the server by checksum, and only the URLs of pages that actually changed are submitted to Yandex, Bing and Google. Daily quotas go to edits, not repeats. | The rsync algorithm: checksum comparisonTridgell & Mackerras, “The rsync algorithm”, 1996 |
How it is built
Five Python scripts and five bash scripts, with no database and no queues: the history lives in two CSV tables, one with a row per page per day and one with a row per day. The laptop runs the cycle on a schedule, but not before 8 a.m. and only when the lid is open and there is a network connection. At night macOS wakes the machine for a few seconds, and without this rule the automation would hit the server at 3 a.m. At 9:00 Moscow time the site's server collects the same numbers itself and sends a report to Telegram. For each search engine the report gives a verdict and a list of what is wrong; on Mondays it adds a weekly summary.
The site does not depend on the snapshot. The snapshot is for the owner.
What holds the design together
There are six rules, and each goes back to a day when the report did not match reality. Four of them are about telling a lost page from a failed measurement: with 18 pages, a single false alarm costs more than a single missed one.
Decisions and their grounds
Below are six decisions that shaped the design itself, not its settings. Each had an alternative that was rejected, and each row names it.
| Decision | Grounds |
|---|---|
| Three search engines, not just Yandex | On September 3 Bing had not opened a single page of the site, and Google held 7 of 14. Watching only Yandex would have hidden the fact that the other two engines barely saw the site. |
| Status of each page, not an overall counter | The Yandex overall counter also includes old URLs outside the sitemap. On Sep 26 an index update removed four deleted tag pages, and the overall “4 dropped” raised an alarm where everything was going to plan. The per-page snapshot keeps current pages apart from stale ones. |
| Exclusion reason from the event feed | Webmaster gives the reason only in a removal event, and only for the latest event per URL. The script takes the most recent event for each page and marks a page with no events as “not taken into search yet”: that is a queue, not a loss. |
| Direct submission on top of IndexNow | IndexNow is sent on every deploy, but Bing went a week without opening a single page, and Google reads the sitemap on its own schedule. So changed pages are also submitted directly: to the Yandex recrawl queue, to Bing as a URL plus the ready page content, and to Google by resubmitting the sitemap. |
| Report on the server, deploy from the laptop | The report is needed every morning, and the laptop is sometimes closed. The deploy needs page checks in a real browser at six widths, and keeping that on the server for a single job is not worth it. |
| AI visibility check outside the automation | Checking mentions of the site in AI model answers costs about 100 RUB (about $1.20) per run and is started by hand. The report shows the age of the last measurement and sends a reminder when it is more than a week old. |
What the design costs
The limits come before the result: it helps to know what the design costs before seeing the numbers. The main one is in the first row: the “in Yandex search” number is an estimate from a sample, not an official count.
| Decision | How it limits |
|---|---|
| The Yandex count is based on the Webmaster sample | Before an index update the sample shows pages Yandex has not taken yet; after the update they disappear. So the “in search” column can dip without any real loss, and every drop is checked against the event feed and the official counter. |
| Google: you cannot request indexing of a URL | Through the API Google returns a page's status but does not accept a request to crawl it: that is only a button in the console. The script can do one thing here: resubmit the sitemap on deploy. |
| Bing does not report index status per URL | For a page, Bing returns only the date of the last crawl. The “in index” number comes from its overall statistics; per page you can only see whether the crawler opened the URL. |
| Daily search engine quotas | Google's documentation allows 2,000 URL inspections a day; for our site, Yandex allows 150 recrawl requests a day and Bing 100 submissions. With 18 pages there is plenty of headroom. On a site with tens of thousands of pages, a daily snapshot of every URL will not fit, and you have to sample. |
| The site is updated only from the laptop | When the laptop is off, the report still arrives with fresh numbers, but the dashboard on the site shows yesterday's until the laptop wakes up. |
| The Yandex token lasts six months | Every six months, access to Webmaster and Yandex Metrica, its web analytics service, is renewed by hand with one command. The Google and Bing keys do not expire. |
What came out
Over the 24 days of snapshots from September 3 to 29, 2026, the table collected 430 rows, one per page per day. The clearest episode involves two articles that Yandex excluded on September 9 as low-value or low-demand. The report named the reason the same day. The first article returned to search on September 15, three days after its title was rewritten to match real Wordstat queries. The second returned on September 24: its title was edited again on the 17th, and on the 22nd all five pages outside search were submitted for recrawl. Since September 24 all 18 sitemap pages have been in Yandex search. The running cost is $0, because the Webmaster, Search Console, Bing and IndexNow interfaces are free. Writing such a pipeline would take a programmer at least 65 hours; the calculation is below.
Effort is code size divided by programmer productivity. For projects of about a thousand lines, the industry norm is 2,500 to 25,000 lines per person per year. At the fast end, the pipeline's 892 working lines come to about 65 hours, or eight working days, which is about $2,100 at Upwork rates for a web developer. The norm covers not only writing the code but also debugging and testing, the work done here together with an AI model.
- T
- programmer hours; ≈65
- S
- lines of code excluding blank lines, comments and docstrings; 892
- P
- lines per person per year, the fast end of the norm for projects of about 1,000 lines; 25,000
- 1,824 h
- a person-year: 12 months of 152 hours
- Sample
- 18 pages of one site, 24 days of snapshots from Sep 3 to Sep 29, 2026. There are three gaps: no snapshot was taken on Sep 6, 11 and 18. The sitemap changed: 14 URLs until Sep 9, 21–22 until Sep 15, and 18 after that.
- Yandex count
- The “in search” column follows the Webmaster sample and fluctuates without real losses: on Sep 20–22 it showed 13, and there were no removals. So a loss is declared only on a Yandex event, and the official Yandex counter is shown next to it.
- Cause and effect
- It cannot be proven that the recrawl is what brought the articles back, since the titles were changed on the same days. The timing is known: 7 days from the second title edit to search, and 2 days from the recrawl.
- Cost of the work
- Programmer productivity varies tenfold: by the same table, 892 lines take 65 to 650 hours. The COCOMO II average is about 400, and the 1981 basic COCOMO gives 324. The article uses the low end: this is a lower-bound estimate, not a quote.
- Not measured
- How many visitors the return of pages to search brought: impressions and clicks are collected for the site as a whole, not per page.
The pipeline does what it was built for: every morning, for each of the 18 pages, it is known where the page stands in Yandex, Google and Bing, and if a page is not in search, the reason is named. Two articles that Yandex excluded as low-value returned to search, and since Sep 24, 2026 all 18 sitemap pages have been in Yandex search. Google holds 16 of 18; two new research pages are waiting to be crawled. Bing remains the weak spot: by its own statistics, 8 pages are in its index. The running cost is $0. Writing such a pipeline would take a programmer at least 65 hours, about $2,100 at web developer rates.
Kaufman Lab is an AI architecture lab. We design processes and methodologies. Code ages with every new generation of models; methodology does not, so the projects focus on methodology. Any current AI model can build a system from this design.
What went wrong
Each pitfall below is in the commit log with a date, and each changed the design, not just one line. They have one thing in common: a number in the report looked plausible and did not match what was happening.
False “dropped” on an updatedropped only on a Yandex event
Pages outside Yandex searchdemand-based titles and recrawl
What else can be added to the code
The pipeline is built so that new features sit on top of the snapshot and the report without reworking the core: per-page history is already accumulating, and the search consoles are already connected. Below are the next steps that would strengthen it, paid services included, and what each one gives the site owner.
What people usually ask
The questions below are about indexing itself: what it costs, why pages drop out and how to get them into search faster.
What is a daily site indexing check?
A daily site indexing check is a snapshot that shows every morning, for each page, whether Yandex, Google and Bing have taken it into search, and if not, why. It is useful for site owners whose inquiries come from search. Kaufman Lab built such a snapshot for 18 pages: it names the reason for each exclusion and submits changed pages for recrawl. It does not track rankings for individual queries and does not edit content.
How much does a site indexing check cost?
The running cost is $0: the Yandex Webmaster, Google Search Console, Bing Webmaster Tools and IndexNow interfaces are free, and the AI subscription is not allocated to projects. There are two market figures. Tracking indexing by hand is an SEO expert's job, about $3,800 a month at Upwork rates. Writing such a pipeline would take a programmer at least 65 hours at the industry productivity norm, about $2,100 at web developer rates on Upwork.
Why does Yandex exclude a page as low-value or low-demand?
In this project the cause was the title, not the length of the text: the articles had two to three thousand words each. Yandex excluded two articles on September 9, 2026, when their titles did not match what people search for. After the titles were rewritten for Wordstat queries, the first returned to search three days later, and the second seven days after the second edit.
How do you speed up indexing of new pages?
Tell search engines about the page right away instead of waiting for a crawler to find it through the sitemap: send the URL via the IndexNow protocol (Yandex and Bing accept it), add the page to the Yandex Webmaster recrawl queue and resubmit the sitemap in Google Search Console. Here this happens automatically on every deploy, and only for changed pages. After the recrawl on Sep 22, 2026, the pages entered Yandex search with an index update two days later.
How is “indexed” in Google different from “in search” in Yandex?
For each URL Google returns one of several statuses: indexed; discovered, not yet crawled; crawled, not indexed; unknown to Google. Yandex distinguishes “page in search” from “excluded” with a reason, and makes its decisions at index updates that happen every few days. Between updates the numbers do not change, however often you check. So the same page can be in Google's index and not yet in Yandex search.
Why does the page count in Yandex Webmaster not match the sitemap?
Yandex counts every URL it keeps in search, including deleted ones: they leave not immediately but at one of the next updates. Here four deleted tag pages stayed in search for a week and left on Sep 26, 2026. So count your own number only over sitemap URLs, and show the difference from the Yandex overall counter as a separate line.
What if Bing does not crawl the site?
Submit pages directly through Bing Webmaster Tools, as a list of URLs plus the ready page content; our site's quota is 100 URLs a day. That is what was done here: on September 3, 2026 Bing had not opened a single page, by September 29 its crawler had opened 7 of 18, and 8 are in its index. If a week passes without a single crawl, the report reminds you to contact Bing support from the console.
How do you check whether a page is indexed in Google?
Use the URL Inspection tool in Google Search Console: it shows whether the URL is indexed, and if not, whether it has been discovered or crawled and why it was not indexed. The URL Inspection API returns the same data, up to 2,000 URLs a day per site according to Google's documentation. That is how each of this site's 18 pages is checked every morning: as of Sep 29, 2026, 16 are indexed, one research page is discovered but not yet crawled, and Google does not know the second one yet.
How many pages a day can you submit for recrawl in Yandex?
As many as the site's daily quota allows: Webmaster shows it in the recrawl section and returns it through the API. For this site, as of Sep 29, 2026, it is 150 URLs a day. According to Yandex help, the crawler visits a submitted page within three days, but a visit does not mean the page will enter search: that decision is made at an index update. So submit changed pages, not the whole site every day.
Why did a page drop out of Yandex search?
Webmaster gives the reason in the removal event: low-value or low-demand, page not found, redirect, non-canonical URL, server response error. First make sure the page really dropped out: the sample-based counter fluctuates while Yandex has not decided on a new page. If the event history has no removal, the page has not entered search yet, and that is a queue, not a loss.
- Boehm B. W. Software Engineering Economics. Prentice-Hall, 1981 — the basic COCOMO model. archive.org · person-month of 152 hours: COCOMO II Model Definition Manual, USC, 2000. pdf
- McConnell S. Code Complete, 2nd ed. Microsoft Press, 2004 — table 27-2, productivity by project size. codinghorror.com
- Nygard M. T. Release It! Design and Deploy Production-Ready Software. Pragmatic Bookshelf, 2007 — failure isolation, timeouts, fail fast.
- Shore J. Fail Fast. IEEE Software, 21(5), 21–25, 2004. doi:10.1109/MS.2004.1331296
- Helland P. Idempotence Is Not a Medical Condition. ACM Queue, 10(4), 30–46, 2012. doi:10.1145/2181796.2187821
- Tridgell A., Mackerras P. The rsync algorithm. Technical Report TR-CS-96-05, Australian National University, 1996. hdl:1885/40765
- Rubin D. B. Inference and missing data. Biometrika, 63(3), 581–592, 1976. doi:10.1093/biomet/63.3.581
- Hunt A., Thomas D. The Pragmatic Programmer. Addison-Wesley, 1999 — single source of truth, the DRY principle.
- Popper K. Logik der Forschung. Springer, 1934 — falsifiability.
- IndexNow. Documentation — the protocol for notifying search engines, accepted by Yandex and Bing. indexnow.org
- Yandex. Webmaster API: recrawl submission and quota; help page “Low-value or low-demand page” (both in Russian). yandex.ru/dev · yandex.ru/support
- Google, Microsoft. URL Inspection API and its limits; Bing Webmaster API: SubmitUrlBatch, SubmitContent. developers.google.com · limits · learn.microsoft.com