Topics without competitors
A free niche is a gap: people search for a topic, but little is written about it. The calculation compares query history with published content and flags such topics. We measured 82,560 phrases and counted demand for 3,011 niches.
Published Aug 25, 2026 · figures checked Sep 1, 2026
How to do this yourself: seven steps to a topic with demand and no competitors
- A wide list of phrases. Collect everything people ask about your subject, not just a dozen keywords.
- Demand. Look up the monthly search count for each phrase in Yandex Wordstat, the keyword-demand tool of Yandex, Russia's largest search engine. It is free; all you need is a Yandex account.
- Supply. Count how many pieces on the topic have already been published and how fresh they are. Search results and Dzen, a large Russian blogging platform, show this in a couple of minutes.
- The gap. Look for topics with demand and little content. That is a free niche.
- Filter the noise. Remove seasonal spikes and one-off news: they look like demand, but they are not.
- One denominator. Always compute shares over the same list. In our case the same measurement gave “+110 topics” over the full registry and “+43” over the data mart.
- Test on history. Take past months and check whether the same choice would have worked then.
Which approach fits which goal
- Blog or channel: 20–30 topics and a monthly recount are enough.
- Services and sales: look at phrases with “price”, “order”, “cost”. Demand is lower, but these people are closer to buying.
- New market: there is no demand yet, so the numbers will not help. Talking to customers works here.
Below is how we built this calculation: 82,560 measured phrases and a test on historical data.
Where the project is now
The chain was completed on August 13, 2026: collection into bronze, recalculation into silver and gold, and a data mart of niches with verdicts, all on daily timers. Four days later, on August 17, the owner decided to stop collection: another project needed the Yandex Cloud quota for 30 days. The shutdown followed a fixed order: first the five timers, then the process, and only with SIGTERM, so that the collector closed its snapshot itself and gzip wrote the end-of-stream marker. The project is parked intact, not abandoned: the collector checkpoint, the registry of loaded snapshots and the log of measured phrases all survive the pause.
What was needed
There was one requirement: find topics with plenty of demand and little content, and base it on data rather than an editor's intuition. Backlink platforms answer a different question, “what does my competitor have that I don't”, and by design they cannot see demand where there is no supply at all. One constraint shaped the whole design: an observation cannot be repeated after the fact. Dzen counters are cumulative, so the signal is the delta between snapshots. Yandex serves its regional breakdown only for a short window and keeps no history, and daily detail is kept only briefly. So the work splits into two unequal parts: collection, which cannot be redone, and analysis, which can be rerun as often as needed.
| Method | What we took from it | Canonical name · source |
|---|---|---|
| Reconnaissance before building: map the terrain before writing code | Before the link was written, we studied how Ahrefs, Semrush, Exploding Topics and Spate solve the phrase ↔ topic problem. The concentration thresholds are not eyeballed: FREE_HHI 0.15 and TIGHT_HHI 0.25 are the standard Herfindahl–Hirschman thresholds used in competition analysis, converted to shares. | Herfindahl–Hirschman indexHirschman, “National Power and the Structure of Foreign Trade”, 1945 · Herfindahl, “Concentration in the Steel Industry”, 1950 |
| Data has one owner: bronze is immutable | A snapshot is written append-only, with a timestamp, the parser version and a run ID. Silver and gold are rebuilt from it in full; after a rebuild they are reconciled phrase by phrase. | Medallion architecture; append-only logDatabricks, platform documentation |
| Measure before explaining: a failing check before any hypothesis | A command that tells “bad” from “good” on a specific symptom is built before any theory. Three failed attempts to explain the same thing mean it is time to stop: the design is in question, not a detail. | Falsifiability of a hypothesisPopper, “Logik der Forschung”, 1934 |
| Every claim carries its status: measurement, hypothesis or decision | Every number cites a file and a line; every decision says who made it and when. The word “hypothesis” stays in the text if there was no check. A number without a source is not used, even when it is correct. | Data provenance; architecture decision records (ADR)Nygard, “Documenting Architecture Decisions”, 2011 |
| Split by testability: what can be recomputed comes first | The question “does the signal predict demand growth” is answered on data already collected, without a single quota call. The question “is the verdict useful” needs four weeks of forward data, so it is handled separately. | Walk-forward validation, point-in-time correctnessPardo, “Design, Testing and Optimization of Trading Systems”, 1992 |
| Acceptance by thresholds: the criterion is declared before the run | The look-ahead leakage checks were written in advance: shuffling outcomes within a point must give 1.00, and shifting the outcome point by 12 weeks must change the result. Both held. | Preregistration of criteria; permutation testNosek et al., PNAS, 2018 · Fisher, “The Design of Experiments”, 1935 |
| Exact match instead of trust: a rule instead of an estimate | The data mart's answers were reconciled with silver rebuilt from bronze: 70,802 phrases, zero mismatches. A row-count comparison is not accepted, because a file that lost a character in the middle passes it too. | Data reconciliationthe lab's own rule, no external source |
How it is built
One Python process, DuckDB next to it, systemd on top: no queues, no containers, no vector stores. The sources are isolated. Wordstat and Dzen each have their own module folder, their own raw data and their own vocabulary; cross-imports are forbidden, and the link between them lives only in the analytics layer on top of the snapshots. A phrase is linked to a topic by its distinctive lemma: the content lemma of the topic title that is rarest across the whole taxonomy. “Championship” and “world” appear in a hundred topics, “judo” in one. An error in silver costs one recalculation; an error in bronze is permanent, because there is nothing to rebuild it from.
An error in silver costs one recalculation; an error in bronze is permanent, because there is nothing to rebuild it from.
What holds the design together
Six rules, each the product of a failure or a measurement rather than taste. The fifth comes from a paper on crawler refresh policies and rests on two proven theorems. The other five are our own, and each has a dated entry in the log.
grep -c 429 returned 27 “quota hits”, and all 27 turned out to be the substring inside “impressions 1442917” and counters 4290–4299. There were no real errors, yet a hypothesis built on that number survived for half a day.Decisions and their grounds
Each decision carries a date and a status: measurement, hypothesis or owner's choice. Below are the seven that changed the platform's design rather than a setting. Each has a rejected alternative and a named cost.
| Decision | Grounds |
|---|---|
| Two sources, not one | Wordstat tells what people search for; Dzen, how much has already been written about it. Each dataset on its own is available to anyone. Together they give a view that backlink platforms do not build by design. |
| Storage in three layers: bronze → silver → gold | The split follows the cost of an error, not the technology. Raw data is immutable and irreplaceable, the database is derived and recomputed for free, and the data mart can be redone at will. |
| No series versioning; an update appends the tail | A measurement on 200 phrases on Aug 11, 2026: 89,600 weekly points compared, 99 mismatches (0.11%), all of them in the last two weeks; zero across the other 446 weeks. SCD Type 2 does not pay off for such a correction, and bronze already keeps the version history. |
| Phrase peak over all history, niche peak over two years | A maximum distributes over a union, so a phrase's peak survives monthly compression exactly. A niche's peak is the maximum of the sum of its carriers' series, and roll-ups over different dimensions do not commute: once time is rolled up into months, the carriers can no longer be summed at the weekly grain. |
| Concentration thresholds come from competition-analysis practice | FREE_HHI 0.15 and TIGHT_HHI 0.25 were listed as eyeballed until a search on Aug 11, 2026 showed that they are the standard Herfindahl–Hirschman concentration thresholds converted to shares. The grounds existed; the reference had been lost. The 0.5 dominance threshold has no external grounds, and the docs say so. |
| Data does not go into the repository | 2.6 GB of snapshots and databases live on the server, and a copy of the irreplaceable part sits on a local machine; checksums of the unpacked streams match byte for byte. Git holds the code, the canon and the protocol: 168 files. |
| The pause is a decision, not downtime | Another project needs the Yandex Cloud quota for 30 days. The shutdown order is mandatory: timers first, then the process. In reverse order, the daily timer would restart collection at 06:00. |
What the design costs
The limits come before the result on purpose: the cost of the design should be known before the reader sees the figures. None of them is a defect. Each was accepted deliberately and costs exactly what is named on the right.
| Decision | What it limits |
|---|---|
| A cap on vocabulary depth | 56,080 depth-4 phrases will never be measured under the current cap. They cost no calls, but they inflate the registry and skew any share computed “over the registry”: 37% of the registry on Aug 17, 2026, and 42% in the Aug 14, 2026 measurement, when the vocabulary held 133,308 phrases. The share moves with the denominator, which is why it always carries a date. |
| A carrier must contain the topic's distinctive lemma | Niche coverage was 29% in the Aug 14, 2026 measurement: demand counted for 2,780 of 9,651 niches. In the Aug 17 data mart snapshot it is already 31% (3,011 of 9,722). Over a third of the quiet niches are class B1: the lemma is in the vocabulary, but only in short phrases. The fix is collecting long phrases, which means quota. |
| Monthly compression of history older than two years | A niche's peak is computed over a 104-week window, and 457 niches out of 2,853 get the “takeoff” label, against 351 with the old all-history peak. The old figure cannot be restored: silver no longer holds eight years of weekly sums. |
| Daily detail and geo are collected in advance | 45,973 calls out of 167,488, about 27% of all quota spent, have never been read by the analytics layer (measured on Aug 17, 2026). There was a reason: collect today what cannot be recovered later. But it is a bet, and it costs a quarter of the grant. |
| The backtest checks demand, not the verdict | The niche verdict includes competition (HHI and the leader's share), which the backtest does not contain at all. The 1.38 lift answers “will demand for this phrase grow” and says nothing about whether the niche is worth writing for. |
| The signal is an edge, not a sure thing | 43 of 65 points have a lift above 1, with a median of 1.15, a worst point of 0.68 and a best of 1.78. This is computed over 65 decision points out of 76: in the other eleven the signal picked no phrase, so there was nothing to compute the lift on. In two months out of three the signal helps; in one it does not help or even hurts. |
| A single copy of the data on the server | If the server dies, the snapshots die with it: they are not in the repository and should not be. A copy of the irreplaceable part is kept by a separate script on a local machine; on Aug 17 it was 977 MB in 110 files. |
What came out
The core was first measured on August 14, 2026, and at no cost: on data already collected, without a single quota call. The backtest universe is 60,448 phrases with at least three years of history. There are 76 decision points, the last week of every month from January 2020 to April 2026, and 3,820,805 decisions. Features at each point are computed strictly from the weeks before it, the outcome from the following twelve. More important than the lift itself is that naive momentum gives 0.93. “What grew will keep growing” works worse than a random choice, so the signal carries information beyond trend continuation instead of rediscovering it.
Lift is the share of phrases whose demand grew by more than 10% among those flagged by the signal, divided by the same share among all phrases at the decision point. A value of 1 means the signal is no better than a random choice. The ratio is computed within each point and averaged across points, weighted by the number of flagged phrases. The base rate drifts from month to month, and pooling all points together would give a lift above 1 even on random outcomes: that is Simpson's paradox, not a signal.
- L
- lift, dimensionless; 1.38 over 3,820,805 decisions
- Δ
- change in demand: mean impressions over the 12 weeks after the decision point against the mean over the 12 weeks before it; growth above 10% means after > 1.1 × before
- signal
- the “takeoff” and “growing” stages: mean impressions over the last 8 weeks are at least 20% above the previous 8; features use only the weeks before the point
- P(·)
- share of phrases at the decision point
- Sample
- 65 decision points out of 76: in eleven the signal picked no phrase. The 3,820,805 decisions are not independent. One phrase appears in all 76 points, and phrases move together, so what counts is not “how many decisions” but “in how many months it worked”.
- Spread
- Median across points 1.15, worst point 0.68 (July 2024), best 1.78 (April 2023); 43 of 65 points are above 1. In two months out of three the signal helps; in one it does not, or even hurts. No confidence interval across points was computed; the distribution is given instead.
- Checks
- Shuffling outcomes within a point gives a lift of exactly 1.00. Shifting the outcome point by 12 weeks with the same features gives 1.00 and 0.97. Both checks were declared before the run, and both passed.
- Does not generalize
- The lift was measured on Wordstat phrases and answers “will demand for the phrase grow”. Competition (the concentration index and the leader's share) is not part of the backtest, so the number says nothing about whether a niche is worth writing for. The signal is not an artifact of phrase length: the lift is 1.12–1.19 at every length from one word to seven.
- Not measured
- Usefulness of the niche verdict: it needs a recommendation log four weeks deep, and that horizon cannot build up during the pause.
The core was measured on 3,820,805 decisions: the demand signal predicts growth with a lift of 1.38, both look-ahead leakage checks passed, and naive momentum on the same data gives 0.93. 82,560 phrases went through the system. Collection was stopped on Aug 17, 2026 as planned, and the quota went to another project.
Kaufman Lab is an AI architecture lab. We design processes and methodologies. Code goes stale with every new generation of models, but methodology does not, so the projects focus on methodology. Any current AI model can build a system from this design.
What went wrong
The pitfall log holds 122 entries in one format: symptom → the diagnosis that helped → fix → rule. Below are eight, each of which changed the platform's design rather than one line of code.
Peak memory after removing 57% of rowsthe gauge showed the limit, not the usage
Database memory limittwo limits in different files, now one
What else can be added to the code
Raw data in bronze is immutable, and silver and the data mart are rebuilt from it for free. So a new source gets its own folder next to Wordstat and Dzen, and a new feature is one more recalculation. Below is what will strengthen the platform once the pause ends, including paid services.
What people usually ask
Below are the questions people ask about the calculation itself: quota usage, why collection was stopped, and how we checked that the calculation does not peek into the future.
What is free niche research?
Free niche research matches search demand for a topic against the number of pieces already published on it: many people ask, few write. It is for authors, editors and content agencies choosing what to write next. Kaufman Lab builds a platform that combines weekly query history with a topic taxonomy and gives each niche a verdict: 9,722 niches in the latest build. It does not predict traffic and does not write articles.
How much quota does round-the-clock collection use?
Usage is measured in API calls, not money: 167,488 Wordstat calls over the project's whole life as of Aug 17, 2026. Weekly history itself took 118,209, or 1.4 calls per measured phrase, because the API is retroactive and returns eight years of history in one call. The other 49,279 are daily detail (26,295), geo (19,678) and errors (3,306); they added no measured phrases. Storage is 2.6 GB on one server. A ready-made alternative costs money: Semrush Pro is $139.95 a month at the public list price as of Aug 25, 2026, and it gives a view of competitors, not demand compared with supply.
Why was collection stopped if the platform was already built and working?
Another project needed the cloud grant quota for 30 days, and the owner decided to hand it over in full rather than split it. The shutdown went in order: first the five timers, then the process, with SIGTERM rather than a forced kill. The collector turns that signal into a normal exit, closes the snapshot and writes the end-of-stream marker. Everything collected that day was loaded into the database, and the final data mart rebuild was the best in the project's history: 3,011 niches with demand against 2,780 three days earlier.
Why not just take search volume and write about the most popular topics?
For two reasons. First, query volume no longer predicts traffic: a noticeable share of queries is answered right on the results page, before any click to a site. Second, and this was measured: the “what grows will keep growing” strategy gives a lift of 0.93 and 0.95 over 3.8M decisions, which is worse than a random choice. Last week's popularity is not the same thing as next quarter's demand.
How did you check that the calculation does not peek into the future?
With two experiments declared before the run. In the first, outcomes were shuffled within each decision point: the relationship must vanish, and the lift fell to exactly 1.00. In the second, the outcome point was shifted by 12 weeks with the same features: the result must change, and it did, to 1.00 and 0.97. Features at each point are computed strictly from the weeks before it; no week after the point enters them.
Why does the same metric give different numbers in different reports?
Because it is computed over different populations, and that is a property of the design, not an error. The registry is every phrase in the vocabulary, including those never measured; the data mart holds only phrases with a measured series; the queue is the difference between them. On Aug 17, 2026 the registry held 152,218 phrases and 82,560 were measured. The mart is half the size, so “has a carrier” and “live niche” are two different numbers. In this project a metric without a named population is invalid: within two days a label was twice read as something it was not, and each time it cost a day.
What happens to the collected data while the project is paused?
Nothing, and that is the point of the stop. Snapshots are immutable by design and the database is rebuilt from them for free, so the pause does not damage a single row. The irreplaceable data sits in two places: 2.6 GB on the server and 977 MB of raw snapshots in a copy on a local machine, with checksums of the unpacked streams verified. The collector checkpoint, the registry of loaded snapshots and the log of measured phrases survive the pause. When it resumes, collection continues where it stopped, with no manual fixes.
How much does niche analysis cost?
The running cost is $0: Wordstat serves weekly statistics on a free quota, and the calculation uses no paid models at all. On the market the same work costs about $3,800 a month at Upwork rates for an SEO expert. For that $0 we measured 82,560 phrases, and the latest data mart build on Aug 17, 2026 holds 9,722 niches.
How do you find a free niche?
A free niche is a gap: there is demand for a topic, but little has been written about it. The platform combines weekly Wordstat query history with the number of pieces already published on Dzen and gives a niche a verdict from three values: demand, the concentration index and the leader's share. A niche with unmeasured demand is marked “quiet”. In the latest data mart build on Aug 17, 2026, demand is measured for 3,011 niches out of 9,722.
How do you check demand for a topic?
Demand is checked against weekly impression history: mean impressions over the last 8 weeks are compared with the previous 8, and growth of at least 20% counts as a signal. On history this was tested on 3,820,805 decisions: among flagged phrases, demand went on to grow by more than 10% 1.38 times as often as the sample average. It is an edge, not a certainty: of 65 decision points, 43 show a lift above 1.
- Hirschman A. O. National Power and the Structure of Foreign Trade. University of California Press, 1945. Concentration index.
- Herfindahl O. C. Concentration in the Steel Industry. Columbia University, 1950. Concentration index.
- Cho J., Garcia-Molina H. Effective Page Refresh Policies for Web Crawlers. ACM Transactions on Database Systems, 28(4), 2003. Theorems 5.1–5.2. doi:10.1145/958942.958945
- Nosek B. A., Ebersole C. R., DeHaven A. C., Mellor D. T. The preregistration revolution. PNAS, 115(11), 2018. doi:10.1073/pnas.1708274114
- Fisher R. A. The Design of Experiments. Oliver & Boyd, 1935. Permutation test.
- Pardo R. Design, Testing and Optimization of Trading Systems. Wiley, 1992. Walk-forward validation.
- Simpson E. H. The Interpretation of Interaction in Contingency Tables. Journal of the Royal Statistical Society B, 13(2), 1951. Simpson's paradox.
- Tversky A., Kahneman D. Belief in the law of small numbers. Psychological Bulletin, 76(2), 1971. doi:10.1037/h0031322
- Reyna V. F., Brainerd C. J. Numeracy, ratio bias, and denominator neglect in judgments of risk and probability. Learning and Individual Differences, 18(1), 2008. doi:10.1016/j.lindif.2007.03.011
- Popper K. Logik der Forschung. Springer, 1934. Falsifiability.
- Nygard M. Documenting Architecture Decisions, 2011. cognitect.com
- Databricks. What is a medallion architecture? databricks.com