Are Search Engines "Deleting" the Web?
Most search and metasearch engines have started to drastically limit the number of results that you have access to. This limiting happens even with short generic queries which should result in a large number of webpages. It’s no wonder then that videos like Disturbing Proof They're Quietly “Deleting” the Internet, are popping up.
The linked video suggests that search engines are giving the illusion that the web is bigger than it is. As a real search engine, one which goes out and crawls the web in order to build an index, we can tell you that the web has not suddenly shrunk as seen by our crawlers.
Unlike other navigators, which give you an estimated number of results at the top of the page, with Mojeek the number we give is exact. What also makes Mojeek different is that we always show you the maximum number of results we can, normally 1,000. Nowadays others show you a variable number, one which is anywhere between 10 and around 300.
But why do we show 1,000, and not every result we find?
We can't tell you why there are so few results for other providers, but we can explain why we limit results to 1,000 (or 100 pages if using the default 10 results per page) for Mojeek. On top of this we can show you a method that you can use if you would like to see more.
Why 1,000?
When it comes to offering access to an index of results, a balancing act has to be struck so that we can focus resources effectively. The process of populating page after page is very resource intensive. On top of this, by design the results will get less and less relevant as you go further into the results pages. We had to pick somewhere to stop, and 100 pages totalling 1,000 individual results was a compromise between depth and cost. On top of this, the speed of Mojeek is considerable, and this speed is possible due to this limit. Without it we would not be able to provide results for a query in around 200ms.
This being said, running tests across the other search and metasearch options it would appear that Mojeek is alone in providing consistent access to page 100 (1,000 results) where there are enough for a query. On top of this, there is a way to see more…
Diving Deeper
Mojeek clusters its results so any host will only appear once in the results. For example, a search for mojeek will only show a single result from the host www.mojeek.com regardless of how many pages you click through. This will not only provide 1,000 results more consistently than other search engines, but it also means Mojeek will show a lot more diversity than any other search engine. This way you won’t end up with Wikipedia, the BBC, or any other site appearing over and over. When there is more than one matching result from a host, you will see a link to 'see more results from host', clicking this will show up to an additional 999 results from that host, plus the original result. Let's illustrate this:
- undertake a search, which will give access to 1,000 results providing they are there;
- on result 1 if you click see more you will be taken to a page with a further possible 1,000 results;
- navigate back to the results page and do the same on result 2; then
- you will also be taken to a page with a further possible 1,000 results.
This method works for all results with a see more link beneath them.
So as you can see, Mojeek can be used to find a lot more than 1,000 results for most queries, with a possible top end of 1,000,000.
Mojeek is not hiding or deleting the web, and stands by its goal of providing alternative and diverse search results. We want to aid your discovery and navigation. Some of us believe in the web...
