If you're a redditor and you run a website, you're probably hoping that one day your site will make it to the front page of reddit.com. Well, last night one of our post from 2009 did just that. WiiLoveMario.com made it to the front page of reddit (going up to the #21 position) and all was great. After upgrading our server to a new dedicated this year, we thought it could handle anything, but we were so wrong! MySQL died when it hit ~300 concurrent connections and had to be restarted a few times; Apache was also restarted about a dozen times as well. Here is how last night's evening was spent:
8pm: We submitted the site to /r/todayilearned/. This was when we noticed a little increase in traffic. We didn't think anything of it.
10pm: The post started to gain some momentum and the upvotes were around ~100. This is when WiiLoveMario.com started to slow down.
12pm: At around this time, the site was crawling and MySQL was getting overloaded with only 300 concurrent connections:
This was when we had to constantly restart mysqld and httpd. After restarting both services a few times, we realized that it was just going to crash again and something else had to be done. By this time the site reached the front page of the /r/todayilearned subreddit.
Here are some useful commands to run when you're running into problems restarting httpd:
Problem: this was the error returned when trying to restart httpd:
Starting httpd: (98)Address already in use: make_sock: could not bind to address [::]:80 (98)Address already in use: make_sock: could not bind to address 0.0.0.0:80 no listening sockets available, shutting down Unable to open logs
Solution: We ran the following command to see what was using port 80:
netstat -lnp | grep 80
The netstat command returned the following:
tcp 0 0 :::80 :::* LISTEN 3595/httpd
That tells you that httpd was taking the port. The command we used to kill it so that apache would restart is:
killall -9 httpd
After killing the httpd, we were able to successfully start httpd (apache).
Killing the httpd and mysqld every 10-15 mins wasn't keeping the server up. What we did was kind of a hack job, but it took all the load coming from reddit.com and gave it all to Google. We added a .htaccess redirect from the original WiiLoveMario post to the Google cache page. Here's the .htaccess entry:
Redirect /2007/03/20/super-mario-bros-theme-song-has-lyrics http://webcache.googleusercontent.com/search?q=cache:http://www.wiilovemario.com/2007/03/20/super-mario-bros-theme-song-has-lyrics
After adding that entry, the server started to cool down some and the traffic went up to around 500 concurrent users:
This solution worked great! The server was stable again and the redditors were able to view the post.
10am: As of this morning, the server is still running smoothly and the upvotes were coming in.
Getting to the homepage of reddit.com was pretty awesome. The only headache is you need to have a server that can handle the traffic. This was a good exercise that will definitely prepare us if the reddit army decides to pay us a visit again.