It’s hard to put this into words succinctly. But when I was a kid before the Internet, the library was the main source of human knowledge, and it was very organized through the Dewey Decimal System which imposed a kind of tree-like hierarchy across a wide spectrum of topics. And while I imagined that one day this might all wind up served by computers, I thought this organization would survive the transition. But it didn’t.
I suppose in the early days of the Internet, they tried? Yahoo was kind of an Internet directory in the beginning, and the Whole Internet catalog was another such effort. But these eventually fell apart and we had to resort to search engines to find anything. Frankly, it’s a bit like the early days of personal computing when we used flat file systems instead of hierarchical ones with proper directory trees. When you were limited to what could fit on a floppy, this wasn’t such a big burden. But the Internet as it stands might as well be a giant flat file system.
Today, even the search engines are failing us, and we are turning to AI. In the pre-Internet era, we had sort of an equivalent to this also. They were called librarians. But a librarian’s job was made easier by the fact that all the books were carefully organized by topic. For AI, it’s as though the library were just a massive pile of books tossed around haphazardly and the librarian had to make sense of it enough to pull whatever you’re looking for out of the chaos. Is it at all surprising, then, that it takes a huge amount of energy and computing resources to make any of this work?
Ah, so you’re saying it’s more of an enshitification thing that killed the search engines? I can believe that to an extent.
But as the amount of content on the Internet has increased, it’s also become more difficult to narrow the queries to get what you’re really looking for. The main thing for me with AI is it allows you to refine the search without discarding the context of what you have already tried.
If the content were more organized to begin with, though, I suspect you could drill a bit deeper into the knowledge tree and then apply your conventional (non-AI) search over a more relevant branch?
Let’s not forget Search Engine Optimization (SEO) and content farms which were actively corrupting (adding entropy to) search engine results even before LLMs, yay capitalism. They’ve dialed it to twelve with LLMs.
Heh, I had actually forgotten about that. It seems almost quaint compared to where we’re at now.
Google has a financial incentive for you to spend more time on their site. If you are gone in one click, you see less ads. Same problem with dating platforms. The user’s interest is in conflict with the platform’s.
There are leaks/interviews how their search and ads teams were at a conflict internally, ofc money won.
Does the ad revenue account for people who hardly realize the ads are even there?
If you hardly realize they’re even there, you’re more likely to click on them since most advertisers pay to be shown on relevant searches, not random ones.
E.g if you run a pet food store, you’ll let Google show your store whenever pet related things are searched for. And that’s where having a blog comes in handy, you can have it show articles relevant to the search rather than just the store front page.
It’s not just increase in content. Search engines don’t even do searches based on keywords or logic operators anymore. They run some NLP on the query input and do who knows what to give you results. If we could still make searcher that contain specifically certain terms or exclude others, it would be still possible to navigate into the mess
Search algorithms are meant to work regardless of the amount of content or sorting. Page rank used to be a game changer back then to sort websites by relevance.
AI has no search algorithms. It’s about 1000% less efficient, because its algorithms are built for token prediction, not sifting through websites.
So yeah, my money is on enshitification.
It doesn’t need to; It just relies on the normal web search to get better data than what it could generate from its training data alone. This wasn’t a thing with the early popular LLM interfaces, which is why they were ridiculously bad. They still make plenty of mistakes, but at least now you get a link to what it looked up (so you can facepalm when you see it’s way too generic and your question was very specific, so the generated answer is still wrong)
I often get dead links or those that state something completely different from what the LLM says. If we’d just cut out AI as the middleman, oh well. 🥲