Making Mozilla is great!

So this is actually why Mozilla thinks making Mozilla is great and what their motivation is. They think firefox enables them to change the way the web is developed. If they don’t like something they won’t implement it. So you clearly see one asset of Firefox is being in the gate keeper position. And being there is valuable in its own. That is probably one of the reasons why Google put so much effort on chrome and android. Both products go very deep and make google a gatekeeper giving them tons of power to steer developments in the “right” direction. 

Look at the feature!

Bobby pointed out that if you want to contribute to Mozilla are Gecko you should not even consider to follow a top down approach. He said that it is impossible to understand 7 mio. lines of (fast changing) code. He rather suggests to look at the specific feature you want to develope and  look at the documentation. You should play around and get it running. In my own experience that is on of the things that makes the difference betweed some coder and an amazing hacker. If you are really able to hack your things in an entire system without overview of the entire thing then your “in business”

Tests are awesome!

Ok that was the most interesting take away. Bobby said that tests are important not only so you can see that your code works properly and that you can check that your code doesn’t interfere with others. But tests are also great so if people change their parts they can check your feature will still work with their. He said your tools won’t continue working for longer than a couple weeks unless you provide tests so others see their plugins don’t destroy your work… Metalcon was only 40’000 lines of code and pretty manageable but we already saw weired things getting messed up once we checked in some code at some other point. To quote Bobby: 

If you check in some new lines you can smoke test them but if you check them in and 200 test lamps go green it feels good. More importantly. A huge community will make new stuff so if you didn’t write a test within a month your new feature might not work…

What happened?

After some debugging I did a fulltext search on the database for %/var/www/wordpress/ and i found one entry in wp_options table having: option_name =”upload_path” option_value=”/var/www/wordpress/wp-content/uploads” so obviously on installation time wordpress is saving the path of the upload folder in its option table to the data base. Still not sure weather this is a bug or a feature but I don’t have time to further investigate. Anyone some ideas?

Wikipedia is huge!

Even though I went for the German Wikipadia which doesn’t even have half the size of the English one I ran into serious trouble due to the huge amount of data. As we know data mining usually is not complex because the algorithms are so difficult but rather because the amount of data is so huge. I would consider Wikipedia to be a relatively small data set but as stated above sufficient big to cause problems.
After downloading the correct data base dump which was about 2 GB in size I had to unzip it. Amazingly no zip program was able to unzip the 7.9 GB huge XML file that contained all current Wikipedia articles. I realized that changing to my Linux operating system might have been a better Idea. So I put the file on my external hard drive and reboot my system. Well Linux didn’t work ether. After exactly 4 GB the unzipping process would stop. Even though I am aware of the fact that 2^32 = 4 GB I was confused an asked Yann for advice and he immediately asked weather I would use Windows or Linux. I told him that I just switched to Linux but then it also came to my mind. My external hard drive has Fat32 as a file system which cannot handle files bigger than 4 GB.
After copying the zipped database dump to my Linux file system the unzipping problem was successfully solved. I installed a media wiki on my local system in order to have all the necessary data base tables. Mediawiki also comes with an import script. This script is php based and incredibly slow. About two articles per second will be parsed and imported to the data base. 1 million articles would therefor need about 138 hours or more than five and a half days. For a small data set and experiment this is unacceptable. Fortunately Wikipedia also provides a java file called mwdumper which can process about 70 articles per second. The first time I was running this java program it crashed after 150’000 articles. I am still not to sure what reason caused the crash but I decided to tweak the mysql settings in /etc/mysql/my.cnf a little bit. So after assigning more memory to mysql I started the script for a second time realizing that it couldn’t continue to import the dump. After truncating all tables I restarted the whole import process again. Right now it is still ongoing but it already has included 1’384’000 articles and my system still seems stable.

Summery: How to download Wikipedia in a nutshell?

1. Install some Linux on your computer. I recommend Ubuntu
2. Use the package manager to install mysql, apache and PHP
3. Install and set up mediawiki (this can also be done via package manger)
4. Read http://en.wikipedia.org/wiki/Wikipedia:Database_download in opposite to the German version file size issues are discussed within the article
5. Find the Wikipedia dump that you want to download on the above page
6. Use the Java Programm MWDumper and read the instructions.
7. Install Java if not already done (can be done with package manager)
8. Donate some money to the Wikimedia foundation or at least contribute to Wikipedia by correcting some articles.

So obviously I haven’t even started with the real interesting research of German Wikipedia. But I thought that it might already be interesting to share the experience I already had with you. A nice but not surprising effect is by the way that the local version of Wikipedia is amazingly fast. After I have crawled and indexed Wikipedia and transferred the data to some format that I can better use I might write another article and maybe I will even publish a dump of my data structures.