AI Competitor Analyst

The service is built, has run on live data and left verifiable artifacts behind. The code is two files: run.py (835 lines) does the measurement, and dashboard.py (588 lines) builds a niche map on top of it without downloading the pages again.

The production run took place on August 21, 2026, in the niche "website promotion, Moscow": 30 sites on the list, 175 Firecrawl credits, about 45 minutes. The duration is reconstructed from the timestamps of the raw files, since the log has none. The same evening, at 18:23, the dashboard was built from the texts already downloaded, for zero Firecrawl credits and 29 extra model calls.

Still to do: the run on the niche named on the service card (legal services, a sample of 50). Two items remain open. One is a beta run by a newcomer who works from the files alone, without the author's help. The other is a mismatch between documentation and code: the instructions tell a person to set up a key for Yandex Cloud, Yandex's cloud platform, while the working script calls OpenRouter.

The goal: see what the whole niche tells a client, not just the three best-known competitors. Contractors price this work per competitor. Demis Group, a Russian marketing agency, publishes a price list of 26,000 RUB for three competitors, 53,000 RUB for five and 72,000 RUB for seven (about $300, $620 and $840). Checking fifty sites by eye at ten minutes each takes a full working day without a break.

The constraint that shaped the whole design: a model confirms what it was asked about, not what is on the page. Asked whether a page answers the client, it agrees out of politeness almost everywhere, and the measurement ends up measuring its agreeableness. The literature calls this sycophancy (Sharma et al., 2023).

Hence one rule for its answers: no "yes" without a verbatim quote, and no quote without a number the code has found in the page text. In the August 21, 2026 run, 20 of 29 sites passed these checks, and for all twenty the quote sits in its own column of the table, ready to be opened and checked by hand.

The pipeline is linear and calls a paid service exactly twice: for company search and for site crawling. Everything else is our own code in one file, with requests as the only dependency.

Search returns 76 companies for 36 credits. Then comes the only manual step: a person strikes out aggregators, duplicates and off-niche companies by eye, and 76 becomes 30. Each site is opened in a real browser: the home page plus up to five inner pages, with links to prices, plans and price lists taken first; otherwise the top menu eats all five slots. In parallel, a free probe with a plain HTTP request records the status code and the load time in milliseconds.

The text goes to the model twice: "does the page answer the client's question", with a mandatory quote, and "what type of contractor is this", with a label from three values. Three code checks stand between the model's answer and a table row: the quote is not empty → the quote contains an amount or a timeline → the numbers from the quote are found on the page. Price, request form and calculator are detected by code from a dictionary, without the model.

All downloaded texts go into raw/: 155 pages, 52.2 MB in 58 files. So the dashboard, with its map of 24 attributes and the price distribution, is built from saved texts and spends no credits, and recalculating after a change to a check costs nothing.

Six rules, each stated against the alternative it rejects. In three of them the code does not trust the model; in the other three it does not trust itself.

All six can be checked against the run's artifacts in a minute: the quote column opens next to the site, the denominator is in the run passport, and the three check states are visible in the table itself.

Eight decisions, each backed by a measurement or a hard limit of an external service. None was made for tidiness: each either removes a specific error or stays within an external service's limit.

The decision with the biggest consequences is the pause between requests. On its own it stretches the run to forty-five minutes, and it has its own row in the limits section.

The main cost is time. The 6.5-second pause between requests stretches a run of 30 sites to about 45 minutes. That figure is not measured: it is reconstructed from the timestamps of the raw files. A sample of 50 sites at the same speed would take an hour, and that is a calculation, not a second run. Only a paid plan speeds this up. Of the total, 155 downloaded pages at 6.5 seconds each add up to about 17 minutes of pure waiting (155 × 6.5 s). Page rendering takes most of the rest; the model and text parsing barely add to the duration.

The second cost is strict definitions. Requiring a number in the quote cuts off sites that answer the client in words: at least one of the nine "no" answers in the run was given at exactly this step, and the reason is recorded in the log. The cap of six pages means that a price hidden on page seven does not make it into the measurement. On average, 5.3 pages were read per site.

Separately, the run's artifacts contain three price counters that give different numbers. This is not an error: they are three different definitions, and here they are in full. 27 of 29 in the report means "mentions a price": the page contains a word from the dictionary (the Russian for "plans", "price list", "cost of services", "% of the amount") or a number with a ruble sign. 25 of 29 in the dashboard means "price as a number": a regular expression only, matching a number with ₽ or a Russian abbreviation for rubles anywhere in the saved text. 21 of 29 in the price distribution is the starting monthly price for promotion: the number was named by the model or by a verified quote, and the code found it on the page, within a range of 3,000 to 3,000,000 RUB (about $35–35,000).

The difference in one sentence: the first two counters answer "is there a figure with a ruble sign on the page", while the third answers "can this figure be turned into a monthly price". The arithmetic checks out. Of the 25 sites with a number, five do not yield a monthly price (the number is the price of another service or a figure from a case study). One site is added back: it says "from 100,000", but its ruble sign is a letter drawn by an icon font, so the regex missed it while the model read it. 25 − 5 + 1 = 21.

The third counter has two sources, and they overlap. The model named a starting price in 13 profiles of 29, the verified quote gave one for 18 sites, and 10 sites are in both. So profile only: 3 sites; profile and quote: 10; quote only: 8; 3 + 10 + 8 = 21.

This page gives almost no percentages, and that is a cost too: 29 sites opened, fewer than forty, and under the project's method a sample below forty is reported in absolute numbers. Where a ratio such as "×10" or "+20%" appears, it is calculated from the sites' own price lists, not as a share of the sample.

The run on August 21, 2026: niche "website promotion, Moscow", 30 sites on the list. 29 opened, and 20 answered the client's question and backed it with a quote. Spending: 175 Firecrawl credits out of 1,000 free per month. The 139 for crawling are in the run passport; the 36 for search come from the service diagram and were added during review. Duration: about 45 minutes, and that is not a measurement. It is reconstructed from the raw files' timestamps, 16:18 → 17:03, because the log has no timestamps at all.

The niche broke down into 24 attributes, and the distribution is a result in itself. Case studies appear on 27 sites of 29, case studies with numbers on 20: the other seven have no result of the form "+N%" or "N times" on the page, which is how this attribute is defined. 7 of 29 promise guaranteed results, while 2 of 29 back that with money ("pay for results"), so the promise is three and a half times more common than the commitment. Not one of the 29 sites offers to work without prepayment: it is the only attribute on the map that never appeared.

Prices could be extracted for 21 sites. The entry price is 15,000 RUB (about $175), the median 45,000 RUB (about $525), the top 150,000 RUB (about $1,750): a tenfold gap. The cheapest and the most expensive of the sites that named both a price and a timeline promise results in the same 3 months. The price differs tenfold; the promise is the same. The comparison rests on the eight sites that named both values, and it is a calculation from their own price lists, not a measurement of outcomes.

Full-service work costs 20% more than plain SEO: a median of 48,000 RUB (about $560) against 40,000 RUB (about $470) for SEO-only contractors, with 10 priced sites in the first group and 11 in the second. This is a ratio of two medians, not a share of the sample.

Ten failures, all found while reviewing the run's own artifacts, not during the build. Eight of the ten concern accounting: the recorded spending, time, search coverage and run passport do not match what actually happened. The other two are about definitions: a figure is labeled as something other than what it measures.

The one with the biggest consequences is the run passport. It declares the method as "firecrawl search + scrape → yandexgpt-lite", while the code uses a different model and a different provider. The file itself was rewritten on August 25, four days after the run, whereas the table and the log were written on August 21 at 17:03:58. A passport rewritten after the fact stops being a passport.

All downloaded texts live in raw/, and the dashboard is built from them without new credits, so a new attribute or a new report is one more pass over the archive, not a new crawl of the niche. Below is what can be added next, including paid services.

Ten questions people ask most often about this measurement. The first one defines the subject and is written to stand on its own when quoted out of context.

Every answer contains a number from the August 21, 2026 run or a published market price.