Growing the Index

mojeek

27 September 2013

3 min

It's all hands on deck at Mojeek. One priority we have right now (on a long, long list) is in growing the index. And for good reason, the size of the index is relative - to a degree - to the quality of results that we deliver to you. This is why Mojeek aims to increase its index size by around 10X over the next 6 - 9 months.

The first stage of that growth started a few weeks back when we installed the first batch of new servers, which have now been put to work. In just a couple of weeks time we'll be returning to the datacentre for the next phase, which includes the installation of more servers.

Our Strategy

While we're still growing we have to be selective about which pages to crawl - the simplest example of that would be crawling wide as opposed to deep. This means that while you may find your small local bakery's website through Mojeek, you won't yet find a profile page four pages deep into Facebook.

As we increase the breadth and capacity of our results, we can afford to be less discerning between domains and the depth of indexing; our index will increasingly go deeper.

One issue we do have currently is the chicken and egg problem. This is a similar problem to what social startups face, except they require users to attract other users and we require more pages. Therefore, without satisfactory results people won't be prepared to switch to Mojeek, and the only way to continue investing into our technology is to continue growing the company on the strength of an increasing user-base.

Fortunately, with the recent investment we're in a position to put in the grunt work necessary, so the quandary of the chicken and the egg is averted. For now.

What's Next?

While the trench work is being done it's certainly not an excuse to rest on our laurels. So I'll be hitting the pavement in London and Brighton, heading to tech meetups, conventions, presentations and anything else that takes my fancy.

I hope to meet some interesting minds in the tech space, perhaps I'll even bump into you, so don't forget to say hi if you spot me as I'm always up for a chat. I'd especially like to meet you if Mojeek piques your interest. (And especially, especially if you're a full stack web developer!)