Big Data in Real Time at Twitter

Another presentation from the folks over at Twitter, specifically from Nick Kallen. This presentation focuses on how Twitter deals with big data in real time. The presentation addresses how Twitter handles Tweets, Timelines, Social Graphs and Search Indices.

General principles that come from their various solutions to their problems are summarized with the following points in the presentation:

• All engineering solutions are transient
• Nothing’s perfect but some solutions are good enough for a while
• Scalability solutions are not magic. They involve partitioning, indexing and replication
• All data for real-time queries MUST be in memory. Disk is for writes only.
• Some problems can be solved with pre-computation, but a lot can’t
• Exploit locality where possible.