Blogged by buffi

Like I mentioned in a previous post, I got my Master’s degree earlier this year. Since then I’ve been taking some courses at the University while looking for a nice place to work. In other words, I’ve mostly been slacking around

About two months ago I decided to browse through a Swedish job database and found an opening as a programmer at Interactive Institute: Sonic Studio which seemed quite cool. I sent in my CV, went on some interviews and finally signed the hiring papers about 1-2 weeks ago. My first day on the job is about a week from now, and I’m really looking forward to it.

Since Sonic Studio is positioned in PiteÃ¥ which is about 50km from LuleÃ¥ (where I live) by car, I’m currently browsing for a decent used car to purchase. This sucks quite a lot since my knowledge in cars is pretty much limited to knowing where the motors and wheels are positioned, but hopefully I’ll be able to work this out since the other option is taking the bus which leaves me with about thirty less minutes sleep in the mornings.

Otherwise I’ve done a bunch of small stuff during the latest months. Among other things, I started coding on a game-engine for shoot-em-up development since I’ve been wanting to build one of those for some time, and I felt the need to brush up on my C++ after doing a bunch of Java and Python development. This is pretty much just a pet-project that I’ve been playing around with occasionally when I get bored, an early proof of concept video is available here in case someone would be interested in that.

I’ve also been reading a bunch of nerd-literature. Since I’ve played around a bit with C++ again, I finally decided to read through Thinking in C++ which helped me brush up on my syntax, among other things. In my opinion, it is a pretty great book for learning and improving ones skills in C++ but I haven’t really read any other books about the language to compare it to.

When I finished up that book, I started with The Pragmatic Programmer which has been on my “to read list” for some time. The book is essentially a list of tips for developing software, where some of them might seem obvious but others actually forces you to rethink your development process. One of them which I know that I’ve made myself guilty of occasionally is the tip they call “Leave no broken windows”. The real world example given in the book is a car that is left in the streets. The car can be left alone for quite some time but once the first window is destroyed, the rest of the car is usually plundered and vandalized within a short span of time. The same hold true for software, and I have personally ignored small bugs to find them come back with friends later on to torture me. Once you start compromising your code quality, it usually snowballs and gets worse quite fast.

It touches a lot on the non-programming aspects of development as well, such as making sure that you gather the correct requirements for you project and use proper tools such as revision control. Of course there are a lot of other great tips in the book as well, and I’d recommend it to pretty much anyone who are involved with development of software.

Finally I’ve read a new book on lighttpd. As I’ve written previously in the blog, I’ve migrated most of my web-stuff from Apache to lighttpd and I couldn’t be more pleased. Setting up lighttpd is a lot easier than setting up Apache, so you wont really need a book for it but I’d definitely recommend it to anyone who either wants a good guide for setting it all up or just tuning an existing installation. It covers a wide arrange of topics including setup, encryption, DOS-protection and migration from other web servers. All in all, it isn’t an essential book for anyone who wants to run an lighttpd-installation but I’d recommend it since it beats the online resources in readability.

buffi Programming & scripting, Python, Uninteresting, Web development c++, job, lighttpd, programming

A few weeks ago a pretty nasty bug appeared on my image hosting service pici.se about a month ago. For some reason, some people started noticing that their thumbnails were changed to pictures which didn’t belong to them. This was a rather serious issue, and I had a look at the code to figure out what was going on. I realized that I had recently changed my deployment from Django under apache to Django under lighttpd and this was surely related to that… but how?

It turned out that it was a combination of bugs that together fucked stuff up for me pretty bad. Since I started allowing pictures over 1MB a while ago, but wanted to limit hotlinking to them, I put them into a separate directory from the other pictures being served. That meant that the large pictures were given filenames such as “large/asSDavXZ.jpg” instead of “asSDavXZ.jpg” where the filename is randomly generated. I had however forgotten to update my code to check for the presence of these images when fetching a new random file name, that is my code looked something like this (pseudo-code):

filename = get_random_stuff()
while file_exists(filename):
  filename = get_random_stuff()

when it should have been changed to this:

filename = get_random_stuff()
while file_exists(filename) or file_exists("large/" + filename):
  filename = get_random_stuff()

When I saw this, the first thing that came to mind was that this shouldn’t be such a big deal. Generating two filenames which were the same should pretty much never happen since there are so many possible combinations of characters.

Well… it did. And quite often. Running some scripts on the server showed that there were quite a lot of these collisions which seemed REALLY weird, and then I found this bug in Django. It seems like fastCGI deployment using method=prefork gives the same random seeds to each process. In combination with my fuckup, this made these collisions happen quite a lot and people got their thumbnails overwritten since all thumbnails were stored in the same folder without the “large/” prefix for each image.

That is, for a pictures thumbnail to be overwritten the following had to happen:

1. Someone uploads a picture below 1MB and is handed to fastCGI process 1 and given a “random” string for it’s filename.
2. Someone else uploads a picture larger than 1MB and is handed to fastCGI process 2, and given the same randomized string as a filename due to the random seeds being non-random due to a bug in Django.
3. My code has a nasty bug in it and doesn’t detect this collision.
4. Thumbnails are generated for image2 at the same target as the thumbs for image1.

Fixing the bug in my code was rather trivial but I also patched my Django installation to avoid any other weird issues due to non-random seeds. The current patch which is available for the bug should however not be utilized since it uses time.ctime() as a random seed for each request, and ctime() will only change once a second which means that subsequent requests given to the same fastCGI process during the same second will be given the same seed. Instead time.time would be better, so I patched my installation with pretty much the same thing but using the following instead.

random.seed("%d%f" % (getpid(), time())) 

This seems to generate random values for each request as far as my testing goes.

buffi Programming & scripting, Python, Web development bugs, django, fastcgi, lighttpd, Python

I run a medium traffic imagehosting site which serves about 8-10000 pageviews per day using Django and I have been using the recommended deployment method (Apache + mod_python) for just under two years and it has mostly worked well. However, about two months ago I was starting to notice increasing delays from the server and Apache would occasionally fail in spectacular ways which brought the CPU load to 100% for long periods of time.

I have always used lighttpd to serve static content, and since I don’t really enjoy the Apache configuration syntax I decided to give the lighttpd + FastCGI deployment method a try. I expected this migration to be complicated, but it took me less than an hour to figure it out. I have some minor documentation of my changes below in case you are interested in the basics of how to handle a Apache -> lighttpd Django migration.

Apache configuration

My old configuration, using Apache looked like this (some stuff omitted).

<VirtualHost *>
        ServerName pici.se
        DocumentRoot /var/www
        ErrorLog /var/log/apache2/pici_error.log
        ServerAlias pici
        <Location "/">
            PythonPath "['/home/buffi/site'] + sys.path"
            SetHandler python-program
            PythonHandler django.core.handlers.modpython
            SetEnv DJANGO_SETTINGS_MODULE pici.settings
            PythonDebug On
        </Location>
        <Location "/css/"> SetHandler None </Location>
        <Location "/js/"> SetHandler None </Location>
        <Location "/im/"> SetHandler None </Location>
        <Location "/picisendfiles/"> SetHandler None </Location>
        <Location "/pictures/"> SetHandler None </Location>
        <Location "/thumbs/"> SetHandler None </Location>

        # Static content served with lighttpd.
        RewriteEngine on
        RewriteRule  ^/pictures(.*) http://static.pici.se:8080/pictures$1
        RewriteRule  ^/thumbs(.*) http://static.pici.se:8080/thumbs$1
</VirtualHost>

lighttpd configuration

The corresponding lighttpd configuration became:

$HTTP["host"] =~ "pici\.se" {
    server.document-root = "/home/buffi/site/pici/"
    fastcgi.server = (
        "/pici.fcgi" => (
            "main" => (
                "socket" => "/home/buffi/site/pici/pici.sock",
                "check-local" => "disable",
            )
        ),
    )

    alias.url = (
        "/css/" => "/home/buffi/site/pici/picipage/css/",
        "/js/" => "/home/buffi/site/pici/picipage/js/",
        "/im/" => "/home/buffi/site/pici/picipage/im/",
        "/thumbs/" => "/var/www/static/thumbs/",
        "/pictures/" => "/var/www/static/pictures/",
        "/picisendfiles/" => "/var/www/picisendfiles/",
       )

    url.rewrite-once = (
        "^(/css.*)$" => "$1",
        "^(/im.*)$" => "$1",
        "^(/js.*)$" => "$1",
        "^(/picisendfiles.*)$" => "$1",
        "^(/thumbs.*)$" => "$1",
        "^(/pictures.*)$" => "$1",
        "^(/.*)$" => "/pici.fcgi$1",
        )
}

You might notice the path of a FastCGI socket.

“socket” => “/home/buffi/site/pici/pici.sock”,

To create this socket, simply use the FastCGI script available through manage.py. I create my socket using the following command.

./manage.py runfcgi method=prefork socket=/home/buffi/site/pici/pici.sock pidfile=pici.pid

All Django requests will then be forwarded from lighttpd to the FastCGI daemon.

Conclusion

By migrating from Apache to lighttpd I noticed a nice performance boost and got a more enjoyable syntax for my configuration. I haven’t really bothered to measure the decrease in CPU load, but it is easily noticeable and my server doesn’t become as sluggish as before during heavy load. I’ve used lighttpd + FastCGI for about a month or so now, and everything seems stable. I’d recommend any Django developer using Apache to give it a try.

buffi Programming & scripting, Python, Web development django, lighttpd, pici, Python, Web development