The technology behind the site

Preamble

In it's conception stages we envisioned a simple to use, highly scalable platform to use as a basis for serving job vacancies to the United Kingdom. We quickly identified the bottlenecks in creating such a platform and went away searching for the best way to overcome these. After toying with a few other noSQL data stores like Facebook's cassandra, Apache's CouchDB and Hypertable we stumbled across MongoDB. Still in it's infancy we quickly realised the power and potential of MongoDB's schemaless data-stores and the speed & simplicity of developing with it.

Why MongoDB?

Alot of websites do not utilise the full power of MongoDB across the breadth of the application - just using it for parts of reporting software or to carry out high load tasks where traditional RDBMS's choke but that wasn't good enough for us. From the start we wanted to create a platform that used in it's entirety MongoDB for all aspects of storing, searching and manipulating data. We found the support of MongoDB was exceptional and bugs were fixed in a very fast manner. We feel that MongoDB is a perfect fit for our project and intend to levy the power of it for future applications and enchancments to this website to attain perfect synergie between application and datastore.

Enter Geospatial Searching

From the research we have done we concluded that most current job sites use string based searching to match keywords entered by a user to strings in a database. This is systemically flawed in that (certainly in the United Kingdom) there are many places that have the same name but are in different geographical locations. We decided to move away from this current way of searching by using geospatial searching within a radius for which MongoDB was extremely fast providing search results in the blink of an eye. Over the years the developers of the site have done alot with Geospatial data in traditional RDBMS's but we allways found the process to be resource hungry. Using MongoDB's geospatial searching (while not accurate to the nearest metre) the search was extremely fast and suited our needs, nb: recent improvements in the MongoDB core have allowed for greater accuracy while maintaining the speed and power.

To compliment the geospatial searching of job vacancies on the site we had to create 2 datastores of geographic geocoded data which comprised of a list of places in the United Kingdom with a latitude/longitude pairing and also a store of unique UK postcodes. The postcode datastore comrises some 1.84 million documents and the areas store comprises some 41,000 unique placenames in the United Kingdom. The data is used in a variety of ways when interacting with the site but for the most part it powers a predictive search to allow the user to start entering a postcode or area and have a few suggestions displayed to them from which they choose and submit a search. Obviously this had to be fast and responsive and Mongo came up trumps with the average time to perform the sarch and return the results being around 30ms- which is pretty impressive.

The Stats

With the datastore system out of the way we looked at potential other bottlenecks on the site and the first we came across was the web server. We decided to move away from a LAMP stack when the site was conceptualised to a lighttpd setup - what is good enough for youtube is good enough for us!. Harnessing the flexability of lighttpd plus the pure power and speed of MongoDB enabled us to get pageload times to average 0.7 seconds. This allows us to serve a greater audiance because content is pushed to the browser in a very fast manner freeing up the webserver resources to allow more content to be served to the end user. Below are 2 examples of how MongoDB helped us achieve a superfast responsive website/application with a lead development time of 2 months.

Site performance in webmaster tools

Page speed score

Developers notes

All in all MongoDB was the right choice for us, we took a huge leap of faith in using such a fledgling project and it paid off in leaps and bounds. MongoDB allows us rapid prototyping of concepts and easy data manipulation, all in all we can't commend it enough. While the diehard SQL database administrators do and allways will pick faults with something outside their realm of knowledge we implore them to try it's simplicity without being amazed. Traditional RDBMS's certainly have their place in the world of computing but they are not very scalable in the real world without shelling out masses on hardware and infrastructure costs. With MongoDB + lighttpd + varnish our application is infinately scalable with MongoDB seemlessly sharding nodes across our infrastructure we can run on relitavley inexpensive hardware and moderate disk space and still scale in a horizontal manner as wide as we need. With the upcoming release of Mongo 1.8 and the future plans for a lucene style full text search we believe the power of MongoDB will grow at a rapid pace and will soon be adopted in alot of cloud-style applications. For now we are happy and confident in it's power and reliability to never go back to an exploitable RDBMS system without the worry that some day we wil have to redevelop the codebase to allow for extra scalability.

Kudos to the MongoDB team, you have provided us with a world class data store which allows us to innovate beyond current technologies and think outside the box with very little constraints.