Scaling vs performance

I am a performance guy. I love algorithms and data structures. When I was 16 years old I already programmed a software that could play chess against you using a high performance programming technique called Bitboards.
But thinking about metalcon I realized that web scale is not so much about performance of single services or parts of the software but rather about the scalability of the entire architecture. After many discussions with colleagues Heinrich Hartmann and me came up with a software architecture from which we believe it will scale for web sites that are supposed to handle several million monthly active users (probably not billions though). After discussing the architecture with my team of developers Patrik wrote a nice blog article about the service oriented data denormalized architecture for scalable web applications (which of course was not invented by Heinrich an me. Patrik found out that it was already described in a WWW publication from 2008).
Anyway this discussion showed me that Scalability is more important than performance! Though I have to say that the stand alone services should also be very performant. if a service can only handle 20 requests per seconds – even if it easily scales horizontally – you will just need too many machines.

Performance vs. Maintainable code

Especially after the code review but also having the currently running metalcon version in mind I came to the conclusion that there is an incredibly high value in maintainable code. The hackers community seems to agree on the fact that maintainability comes over performance (only one of many examples).
At that point I want to recall my initial post on the redesign of metalcon. I had in mind that performace is the same as scaling (which is a wrong assumption) and asked about Ruby on rails vs GWT. I am totally convinced that GWT is much more performant than ruby. But I have seen GWT code and it seems almost impractical to maintain. On the other side from all that I know Ruby on Rails is very easy to maintain but it is less performant. The good thing is it easily scales horizontally so it seems almost like a no brainer to use Ruby on Rails rather than GWT for the front end design and middle layer of metalcon.

Maintainable code vs scalability

Now comes the most interesting fact that I realized. A software architecture scales best if it has a lot of independent services. If services need to interact they should be asynchronous and non blocking. Creating a clear software architecture with clear communication protocols between its parts will do 2 things for you:

1. It will help you to maintain the code. This will cut down development cost and time. Especially it will be easy to add , remove or exchange functionality from the entire software architecture. The last point is crucial since
2. Being easily able to exchange parts of the software or single services will help you to scale. Every time you identify the bottleneck you can fix it by exchanging this part of the software to a better performing system.

Summary

I find this to be very interesting and counter intuitive. One would think that performance is a core element for scalability but I have the strong feeling that writing maintainable code is much more important. So my ranked list of priorities for backend web programming (!) looks like that:

1. Scalability first: No Maintainable code helps you if the system doesn’t scale and can’t be served to millions of users
2. Maintainable code: As stated above this should go almost hand in hand with scalability
3. performance: Of course we can’t have a data base design where queries need seconds or minutes to run. Everthing should happen within a few milliseconds. But if the code can become more maintainable at the cost of another few milliseconds I guess thats a good investment.

• a user base of 10’000 users.
• about 500 registered bands
• highly semantic and interlinked data base (bands, geographical coordinates, friendships, events)
• 624 MB of text and structured data about the mentioned topics.
• fairly good visibility in search engines.
• > 30k lines of code (mostly PHP)
• a bad scaling architecture (own OR-mapper, own AJAX libraries, big monolithic data base design, bad usage of PHP,…)
• no unit tests (so code maintenance is almost impossible)
• no music and audio files
• no processes for content moderation
• no processes to fight spam and block users
• a really bad usability (I could write tons of posts at which points the usability lacks)
• no clear distinction of features for users to understand
• …

When we built metalcon no one on the team had experience with high scaling web applications and we were about happy to get it running any way. After returning from china and starting my PhD program in 2011 I was about to shut down metalcon. Though we became close friends the core team was already up on new projects and we have been lacking manpower. On the other side everyone kept on telling me that metalcon would be a great place to do research. So in 2011 Jonas and me decided to give it another shot and do an open redevelopment. We set up a wiki to document our features and the software and we created a developer blog which we used to exchange ideas. Also we created some open source project to which we hardly contributed code due to the lacking manpower…
Well at that time we already knew of too many problems so that fixing was not the way to go. At least we did learn a lot. Thinking about high scaling architectures at that time I new that a news feed (which the old version of metalcon already had) was very core for the user experience. Reading many stack exchange discussions I knew that you wouldn’t build such a stream on MySQL. Also playing around with graph databases like neo4j I came to my first research paper building graphity a software which is designed to distribute highly personalized news streams to users. Since our development was not proceeding we never deployed Graphity within metalcon. Also building an autocomplete service for the site should not be a problem anymore.

Roadmap for the redesign

• Over the next weeks I hope to read as many interesting articles about technologies and high scalability as I can possibly find and I will be more than happy to get your feedback and suggestions here. I will start reading many articles of http://highscalability.com/ This blog is pure gold for serious web developers. 
• During a nice discussion about scalability with Heinrich we already came up with a potential architecture of metalcon. I will soon introduce this architecture but want to check first about the best practices in the high scalability blog.
• In parallel I will also collect the features needed for the new metalcon version and hopefully be able to pair them with usefull technologies. I already started a wikipage about features and planned technologies to support them.
• I will also need to decide the programming language and paradigms for the development. Right now I am playing around with ruby on rails vs GWT. We made some greate experiences with the power of GWT but one major drawback is for sure that the website is more an application than some lightweight website.

So again feel free to give input, share your ideas and experiences with me and with the community. I will be ver greatfull for every recommendation of articles, videos, books and so on.

• how to integreate a SuggestBox instead of a textfield into the GWT Starter project
• how to set up the neccessary stuff (extending a SuggestOracle) to fire a remote procedure call that requests suggestions if the user has typed something.
• how to override the necessary methods from the SuggestOracle Interface

So here we go with the second part of the screencast which you can of course directly download from wikipedia:

Feel free to ask questions, give comments and improve the screencast!

• source code
• slides from the talk
• data sets
• proposal
• cypher benchmark

was great talking here and again we are open source, open data and so on. So if you have suggestions or want to contribute feel free. Just do it or contact us. We are really looking forward to meet some other hackers that just want to go geek and change the world
the video of the talk can be found here:

part 1: introduction of which parts exist and where to find them

• Set up a gwt project
• Erease all files that are not required
• Create a basic Design

part 2: AutoComplete via RPC

• Neccesary client side Stuff
• Integration of SuggestBox and Suggest Oracle
• Setting up the Remote procedure call

part 3: A basic AutoComplete Server

• show how to fill data with it and where to include it in the autocomplete
• disclaimer! of not a good solution yet
• Always the same suggestions

part 4: AutoComplete Pulling suggestions from a data base

• inlcuding a data base
• locking the data base for every auto complete http request
• show how this is a poor design
• demonstrate low response times speed

part 5: Introducing the context Listener

• introducing a context listener.
• demonstrate lacks in speed with every network request

part 6: Introducing a fast Index (Suggest Tree)

• inlcude the suggest tree
• demonstrate increased speed

part 7: Introducing client side caching and formatting

• introducing caching
• demonstrate no network traffic for cached completions

not covered topics: (but for some points happy for hints)

• on user login: create personalized suggest tree save in some context data structure
• merging from personalized AND gobal index (google will only display 2 or 3 personalized results)
• index compression
• schedualing / caching / precalculation of index
• not prefix retrieval (merging?)
• css of retrieval box
• parallel architectures for searching

The Graph that I used is extracted from the complete revision dump of the bavarian wikipedia and has the following properties:

• 8’760 entities
• ~95’000 Content items
• ==> together more than 100’000 nodes
• almost 20’000 different relationship types (more to come in bigger graphs!)
• about 100’000 edges connecting the 8’760 entities
• mainting the index was possible with (only :[ ) 177 writes / second on a slow notebook computer

I interpret the wikipedia graph in the following way as a social network: 

• every article corresponds to an entity
• every link corresponds to a directed friendship (follower)
• every revision corresponds eather to a status update (content item) or a change in the friendship graph

I have no idea yet how many reads I can do per second. Even though I am a little disappointed about the low speed for writing on the graph I am sure that I will achieve my theoretical goals for reads per second. I also hope to increase writing spead if I introduce better transaction management. Anyway I will blog about the results of the reads / second later.