Building a web search engine from scratch in two months

A developer built a web search engine from scratch in two months to test whether neural text embeddings can answer complex, intent-based queries better than keyword matching. A cluster of 200 GPUs generated 3 billion SBERT embeddings, and at peak hundreds of crawlers ingested 50K pages per second, culminating in an index of 280 million pages. End-to-end query latency landed around 500 ms, with RocksDB and HNSW sharded across 200 cores, 4 TB of RAM, and 82 TB of SSDs. The stated motivations were a perceived decline in search quality, including more SEO spam and less relevant content.