Also I suspect that, unlike when Google et al started crawling the web, most sites just aren't linked to, or link to, other sites. We used to get around the whole web with links, which is how the web was designed, and now… no links. All the big sites rely on search because they wouldn't be caught dead actually *linking* to other sites - theirs is *obviously* the only site anyone would want to ever go to, after all. Why should they give their traffic to some other, undeserving site? 🙄

huxley(fur)

2024-07-24 14:36:17

last boost:

An undertold story is the monopoly on search engine *scrapers*. Thanks to DDOS hardcoding, it's essentially impossible to create a new webcrawler. All there is is google, bing, and one or two (forget exactly) chinese bots. It is literally not possible to create a new search engine because it's not possible to scrape the web.