How Digg is Built

I personally always enjoy these types of posts. Dave Beckett of Digg has written a post describing what technologies are used to build Digg. Now Digg, as of late, has not had an easy go of things since v4 was released. You'll recall that Kevin Rose through Cassandra under the bus during that whole fiasco.

Well despite that Cassandra is listed as one of the datastore technologies used by the site. In addition, Redis makes an appearance as well. Here is what they use each data store for:

• Cassandra: The primary store for "Object-like" access patterns for such things as Items (stories), Users, Diggs and the indexes that surround them. Since the Cassandra 0.6 version we use does not support secondary indexes, these are computed by application logic and stored here. This allows the services to look up, for example, a user by their username or email address rather than the user ID. We use it via the Python Lazyboy wrapper.
• Redis: The primary store for the personalized news data because it needs to be different for every user and quick to access and update. We use Redis to provide the Digg Streaming API and also for the real time view and click counts since it provides super low latency as a memory-based data storage system.

Read the entire post at: How Digg is Built p>