Implementing MongoDB at shutterfly

Kenny Gorman, Data Architect at shutterfly shows us how some NoSQL data stores are ready for primetime. In this case it's MongoDB. This presentation discusses how shutterfly started to make the transition from a traditional Oracle RDBMS to MongoDB. The results, 500% improvement in cost, 900% improvement in performance and shards on demand. Pretty impressive.

Couple of interesting stats about shutterfly:

• 20TB of RDBMS storage
• 10000 ex/sec
• 6 Billion photos
• Adding 400TB a month

Key lessons learned:

• Keep it simple
• Data Modeling
• Walk before you run
• Use Jira for MongoDB issues
• There is life after Larry



Schema Design with MongoDB

If you ask most developers to design a schema for a traditional relational database they would have no problem doing so. However, introduce a document oriented database like MongoDB and they might struggle. Fortunately, Dwight Merriman, CEO of 10gen, has come to our rescue. In his presentation he walks through a couple of different use cases and describes techniques to handle those use cases.

Zero to Mongo in 60 Hours

Ryan Angilly, a Senior Developer from MyPunchbowl.com and self proclaimed "pretty awesome dude" brings us our next presentation about how MyPunchbowl.com integrated MongoDB into their software stack and into production in 60 hours.

Why did they choose MongoDB? Ryan lists six strengths of MongoDB:

1. Easy to get running
2. Open Source
3. Support in multiple (computer) languages. Prototype in Ruby, move to Java if necessary
4. Very active development
5. Full featured
6. Great ecosystem

Ryan does a good job of describing the various stages of development, testing and deployment. Finally, Ryan discusses where they are at 200 days later and what tripped them up during the process.

Java Development with MongoDB

James Williams, a Software Engineer at BT/Ribbit, demonstrates how to use Java with MongoDB. In addition, James introduces us to Morphia an open source Apache 2 licensed library that:

• Brings Hibernate/JPA paradigms to MongoDB
• Allows annotating of POJOs to make converting them between MongoDB and Java very easy
• Supports DAO abstractions
• Offers type-safe query support
• Compatible with GWT, Guice, Spring and DI frameworks

Flexible Event Logging - Analyzing Funnels, Retention and Viral Spread with MongoDB

his presentation is by Paul Gebheim from Justin.tv. In the presentation Paul discusses how MongoDB helped answer create:

A general framework for create, deploying and analyzing A/B tests in terms of Funnels, Virality and Retention

In addition, the solution should be flexible, queryable, scalable and easy to work with. So how do you do it? With Python, Map/Reduce and MongoDB. Paul walks through the use cases and shows how justin.tv uses this recipe to create the framework.

Hadoop, Pig and HBase at Twitter

Again the folks at Twitter provide us the material for this next post. Specifically, Dimitriy Ryaboy a member of the Analytics team at Twitter, discusses Twitter's usage of Hadoop, Pig and HBase. Now technically both Hadoop and Pig are not really pure NoSQL, really they are ancilary components that interact with a NoSQL data store HBase.

However, HBase is a big part of the Hadoop ecosystem and Pig provides a simplified query mechanism for HBase, so in my opinion they are worth the discussion.

There are obviously several important points made throughout the discussion but one slide I found particularily interesting in which Dimitriy explains how they use Cassandra and HBase and what they use them for.

Rough Analogy: Cassandra is OLTP and HBase is OLAP

SQL anti-patterns and NoSQL Alternatives

This slide presentation is by Gleicon Moraes. Gleicon discusses various anti-patterns that show up frequently when trying to use the traditional relational database and fix those anti-patterns when you use a NoSQL database.

Scaling Twitter with Cassandra

Let's paint a little picture about how it was at Twitter prior to Cassandra.

• Horizontal and vertically partitioned MySQL
• Memcached (rows, indexes and fragments
• Application Managed

This obviously had drawbacks such as:

• Many single points of failure
• Hardware Intensive
• Manpower Intensive
• Tight Coupling

The solution, Cassandra, the NoSQL data store originally developed by Facebook and given to the Apache Software Foundation. The presentation below is from Ryan King, a member of the Storage Team at Twitter. The presentation dives into details of how Cassandra solved the problems listed above. Enjoy!

NoSQL at Twitter

This is a slide presentation by Kevin Weil, Analytics Lead at Twitter about NoSQL at Twitter. Kevin details the  how Twitter arrived at using NoSQL. For a majority of their applications, both front and back end, they were using MySQL using the standard scalability techniques. However, were still running into issues batch jobs not completing within a specified time and write limits just to name a few.

Twitter uses several different pieces of the NoSQL ecosystem, employing the tool that is best suited for the job. Hadoop and HBase for batch oriented tasks such as analytics. Using Cassandra for storing tweets.

In addition, Twitter has been an active participant not only in patching the various tools it uses it has also contributed back to the open source community with its release of FlockDB, a distributed graph database.

Definitely a great read to see how an application, with high scaling and performance requirements, uses NoSQL to solve it’s problems.

8,000,000 operations per second

That is the level of performance that Eliot Horowitz has reached with MongoDB. The following are the slides from his presentation on sharding and a link to the video of the presentation as well.

Geodata + CouchDB = GeoCouch

Public geospatial data repositories seem to be an increasingly popular application lately. Especially when you see SimpleGeo receive $8 Million in VC money to continue to develop their geospatial service. I’ll be writing about SimpleGeo and their usage of Cassandra in later posts.

Anyway, I digress this post is about an open source geospatial service built on top of CouchDB. The presentation, by Volker Mische creator/developer of GeoCouch, gives the standard introduction to CouchDB and the benefits it has to offer its users schema free, high concurrency and replication just to name a few. It’s a good and quick read.

Read More:  Geodata and CouchDB