Whoa, Echochrome has level sharing between users!

March 31st, 2008

This post isn’t about programming for once. I also tend to spend a lot of time in front of my PS3 and just found out something that is quite cool and seems to have been overlooked.

Echochrome has been out in the Japanese and Hong Kong store for a little while now, and apparently it supports level sharing! And not just that, but it uses your PSN friends list as well (you can also send levels to people who aren’t your friends). There is also a feature for “uploading to a public server” but I couldn’t get that to work. Anyways, I really like this step towards community driven content, and I hope more games will use it. Here are some pictures:

The “Factory” for custom levels:

A test level I made just to try it out:

The send-dialog box:

Ingame friends list:

Sending…:

Transmission completed!:

Unfortunately Echochrome is scheduled for May outside of Asia, which is weird as hell considering that the version from the Hong Kong store is completely in English. There doesn’t seem to be any limitations for purchasing it for users outside of Hong Kong (I’m located in Europe) so there really doesn’t exist any reason to wait. Get a HK-account and download it now :)

One of the main reasons that so many PHP-coders suck at programming?

February 2nd, 2008

I was about to just write a very small thing in PHP, and wondered what the nicest way to get a random string i PHP would be (lacking a nice sample method like in python) and googled “random string php”.

I found this.

Just… what is that? It just might be the ugliest piece of code I’ve ever seen. Not only does it have a nice little switch statement for each letter in the alphabet, for some reason the coder also seeds the random seed for each iteration in the loop.

This might be the worst I’ve seen, but I’ve seen a lot of PHP example code that is pretty close, and I think that this might be one of the main reasons that a lot of PHP-developers produce unreadable code (have a look at tbsource among other large projects for some horrifying code).

It’s nice that people like to share their code to teach others, but you are more likely to hurt development if you don’t have a clue what you are doing.

I’m not saying that all PHP-developers are horrible programmers (because they aren’t), and I use quite a few PHP-applications myself (such as wordpress) but the ratio of bad code compared to other languages seems to be way above average.

How to get random rows from mysql using django without hurting your server

January 20th, 2008

Apparently using ORDER BY RAND() in mysql is a really bad idea (for tables containing a fair ammount of rows).

I did not know this, and tried getting random rows in django by using order_by(”?”), which uses ORDER BY RAND() to get a set of 32 random rows from one of my tables containing about 200 000 rows. This turned out to be a really bad idea, and it pretty much violated my server, having mysql consuming all of my CPU and most of my memory. Going through the slow queries log showed this:

# Query_time: 70  Lock_time: 0  Rows_sent: 0  Rows_examined: 0
SELECT `picipage_picture`.`id`,`picipage_picture`.`name`,`picipage_picture`.
`header`,`picipage_picture`.`description`,`picipage_picture`.`uploadednick_id`,
`picipage_picture`.`uploadedip`,`picipage_picture`.`views`,`picipage_picture`.
`timestamp`,`picipage_picture`.`gallery_id`,`picipage_picture`.`private`,
`picipage_picture`.`privid`,`picipage_picture`.`camera` FROM `picipage_picture`
WHERE (`picipage_picture`.`private` = 0) ORDER BY RAND() LIMIT 32;

A query time of 70 is pretty much insane (the limit for a query to be concidered slow is by default 2).

The blog post mentioned at the top mentions how to avoid this, and I’m gonna go ahead and post the djangoed version of the solution.

Getting a single row is simple.

random_pic = Picture.objects.order_by("?")[0] # Slow!

becomes

from random import randint
num_pics = Picture.objects.count()
random_pic = Picture.objects.all()[randint(0, num_pics-1)] # Fast!

Getting a set of random objects is harder.

random_pics = Picture.objects.order_by("?")[:32] # SUPER slow!

becomes

from random import sample
PICS_TO_GET = 32
num_pics = Picture.objects.count()

# Get a bunch of extra numbers, to avoid missing ID's
# Assumes enough rows
rand_nums = sample(xrange(1,num_pics), PICS_TO_GET*10)

# Match ID's of pictures to the sampled list
random_pics = random_pics.filter(id__in = rand_nums)[:PICS_TO_GET] # Fast!

This is messy, but really fast and works beautifully on tables with a bunch of rows (but there are probably nicer ways to do it).
Thanks goes out to mattmcc in #django@freenode for pointing me towards the blog entry.

Easy concurrency in Python using decorators

January 9th, 2008

I ran across this reddit link about daemonizing processes in unix with Python and I thought that a decorators for forking processes might be a bit nicer. Someone has probably already done this but whatever.

I should also right away mention that my code is heavily inspired by these two URL’s
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/66012
http://www.myelin.co.nz/post/2003/3/13/

Basically, threading in python is a bit gimped by the global interpreter lock. This isn’t necessarily a bad thing, but if you want to be able to use “true” concurrency in python you will have to use multiple processes instead of threads or as the python FAQ puts it:
“This doesn’t mean that you can’t make good use of Python on multi-CPU machines! You just have to be creative with dividing the work up between multiple processes rather than multiple threads.”

Using pythons generator syntax it is quite easy to create a generator that transforms a normal function into a separate process, and returns a pipe from which output can optionally be receieved. The syntax of use will be something like this (I will explain the implementation of @forked below).

@forked
function my_function_goes_here(arguments):
    stuff()

This simple addition of the forked generator will when calling your function instead return a data pipe and launch the function in a separate process. Here is a more concrete example of usage:

# This process is just an example of usage
#
# The function write_times is forked into a
# separate process using the @forked generator
@forked
def write_times(arg, num, delay=0):
    """Write_times(arg, num, delay=0)
    print the argument arg num times with an optional
    delay inbetween"""
    import time

    for i in xrange(num):
        time.sleep(1)
        print arg,

Then you can simply call the function, receive the pipe, keep on doing stuff in your normal process and when you want the data from the other process, read it from the pipe like this.

# Create the forked process
r_pipe = write_times("foo", 3, 1)

print "This will be outputted directly after the fork"

print "Waiting for input from fork..."
in_data = r_pipe.read()

print "Data received:", in_data
r_pipe.close() # Clean up pipe

Running this will produce the output

$ python test.py
This will be outputted directly after the fork
Waiting for input from fork...
Data received: foo foo foo

Finally. Here is the implementation of the @forked generator. Nothing too crazy goes on in here. It pretty much just performs a standard double fork, and the parents returns the read end of the pipe while the child goes on about doing the passed function through a wrapper and outputting all its data through the pipe.

def forked(f):
    """Generator for creating a forked process
    from a function"""
    import os, sys

    # Make a pipe
    r, w = os.pipe()

    # Perform double fork
    if os.fork(): # Parent
        os.close(w) # close write end of pipe
        r = os.fdopen(r)

        # Return a function that returns the read pipe
        return lambda *x, **kw: r 

    # Otherwise, we are the child 

    # Perform second fork
    os.setsid()
    os.umask(077)
    os.chdir('/')
    if os.fork():
        os._exit(0) 

    os.close(r) # Close read part of pipe

    w = os.fdopen(w, 'w') # Get write part for writing

    # Bind stdout to pipe
    sys.stdout.flush()
    sys.stdout = w

    def wrapper(*args, **kwargs):
        """Wrapper function to be returned from generator.
        Executes the function bound to the generator and then
        exits the process"""
        f(*args, **kwargs)
        w.close() # Clean up pipe
        os._exit(0)

    return wrapper

That wasn’t very hard now was it? :)

The full source can be downloaded here, including the test-case. It’s only about 40-50 lines of code or so excluding comments.
Please feel free to leave comments and suggestions for improvements. I’m not really an Unix expert or anything.

Cross-platform suppressing of output in python

January 5th, 2008

A common way to suppress output under Unix/Linux in python is by doing

import sys
sys.stdout = open("/dev/null","w")
print "Hello world!" # Will not be printed to stdout

The reason for this could for an example be to let a forked process run silently, or something similar which could not be achieved by simply redirecting the entire application to /dev/null.

The issue with this is that this won’t work on non-unix systems such as windows, which normally isn’t an issue, but if you want platform independency then it isn’t very hard to achieve. Simply create a “file-like object” which implements write, and let it do nothing. This will work on all systems.

import sys

class Silencer(object):
  def write(self, data):
    pass

sys.stdout = Silencer()
print "Hello world!" # Will not be printed to stdout

If you know that you are using other methods of stdout such as writelines, implement them as well.

I made an FTP-server (easyFTPD v0.1)

December 16th, 2007

… and you can get it here!

Setting up virtual users in most unix/linux-FTP-servers is actually a lot harder than it should be so I decided to create an easy to use FTP-server, targeted at virtual users.

I started off with an implementation using twisted but I soon found pyftpdlib which fitted this project perfectly. Since pretty much everything you need is already implemented, the application became reduced to programming a wrapper around pyftpdlib and setting it up for easy configuration and deployment.

The first version can be found here, and I really like the way it turned out. Creating virtual users are simply done by adding them to a users-file with the syntax

username:password:permissions:share_folder

where password might be a salted hash, or plaintext. More info on user configuration here.

Getting it running is as easy as first installing it by doing the following as superuser

python setup.py install

It can then be started with or without options

easyftpd              # Default settings
easyftpd -p 2100 -s   # Port 2100 and silent (no logging)
easyftpd -p 12345 -d  # Port 12345 and running as daemon

Information about more options and the configuration file can be found at the project page.

Hopefully I’m not the only one who has needed an easy to use FTP-server. proFTPD is simply too scary for me ;)

Abusing python: Game of life using ncurses in one line of code

November 4th, 2007

I like to make one-liners. Not because they are useful but because it’s fun, and it teaches you stuff about the workings of a language and how to abuse it. When I say one-liner, I mean a single statement that does not use eval (otherwise one-liners are trivial) or ; to separate multiple statements.

I have never written any implementation of Conway’s Game of Life before, and tried to see if I could get it done using a single line of code.

I started out by coding a function that when given a sequence of (x,y) coordinate tuples returned a new dictionary with the next set of coordinates set to True. The choice of a dict for the return value was actually not perfect, as a set would have done just fine, but I used a dictionary earlier in the algorithm for checking which of the coordinates that should be alive, and I didn’t really see any need to rewrite it since a dict here pretty much is a set with a (in this case) redundant value.

The first working algorithm looked like this:

def get_next(old):
  new, new2 = {}, {}
  offset = ((-1, -1),(-1, 0),(-1, 1),(0, -1),
    (0, 1),(1, -1),(1, 0),(1, 1))
  for (x, y) in old:
    for (ox, oy) in offset:
      new[(x+ox, y+oy)] = \
        new.get((x+ox, y+oy), 0) + 1
  for (x, y) in new:
    v = new[(x, y)]
    if v == 3 or v == 2 and (x, y) in old:
      new2[(x, y)] = True
  return new2

A bit of abuse got this down to a single line that looked like this (as you might see, the backslashes are used to make this more readable, it really is just a single line of code.

def get_next(old): return globals().__setitem__("new",{}) \
  or globals().__setitem__("new2",{}) or \
  [[new.__setitem__((x+ox,y+oy),
  new.get((x+ox,y+oy), 0) + 1) \
  for (ox, oy) in ((-1, -1),(-1, 0),(-1, 1),(0, -1),(0, 1),
  (1, -1),(1, 0),(1, 1))] for (x,y) in old if old[(x,y)]] \
  and [new2.__setitem__((x,y),True) for (x,y) in new if \
  (new[(x,y)] == 3 or new[(x,y)] == 2 and (x,y) in old)] and new2

This works fine for getting the next “map” of the game of life, but how fun is that? I wanted ncurses support so that I could actually see this. Eventually I ended up with this:

print globals().__setitem__("f",globals().__setitem__) or \
  f("d",{(1,2):True,(2,3):True,(3,1):True,(3,2):True,(3,3):True,}) \
  or f("curses",__import__("curses")) or f("stdscr", curses.initscr()) \
  or stdscr.nodelay(1) or curses.noecho() or curses.cbreak() \
  or (curses.curs_set(0) or 1) and f("bar",[1]) or [1]+[(bar.append(1) \
  or stdscr.clear() or [1]+[stdscr.addstr(0,0,"Press q to quit") \
  or stdscr.addch(y%20+1,x%20,"x") for (x,y) in d] and stdscr.refresh() \
  or __import__("time").sleep(0.2) or f("d",(lambda old: f("new",{}) \
  or f("new2",{}) or [[new.__setitem__((x+ox,y+oy),
  new.get((x+ox,y+oy), 0) + 1) \
  for (ox, oy) in ((-1, -1),(-1, 0),(-1, 1),(0, -1),(0, 1),
  (1, -1),(1, 0),(1, 1))] \
  for (x,y) in old if old[(x,y)]] and [new2.__setitem__((x,y),True) \
  for (x,y) in new if (new[(x,y)] == 3 or new[(x,y)] == 2 and (x,y) \
  in old)] and new2)(d))) for foo in bar if (stdscr.getch() !=
  ord('q'))] and curses.nocbreak() or stdscr.keypad(0) or \
  curses.echo() or curses.endwin() or "Bye!"

Now THAT is a one-liner! :)

Running this creates a ncurses window and a “glider” that will move on forever (wraps to x=0,y=0 at position x=20,y=20).
Here are some pics of it running:

If you want to run this yourself to have a look you can download the one-liner here. It should work on any unix/linux/mac-system supporting ncurses. I don’t think that ncurses works in more obscure operating systems such as windows so if you are a windows user then you will just have to take my word on that it works ;)
This version also lacks the line separation backslashes used for the presentation here which perhaps makes it clearer that it is just a single line of (rather horrible) code.

Like I said, I am well aware of the fact that this code isn’t pretty, it isn’t meant to be. It does however contain a nice bunch of clever tricks that some people might not be aware of.

Some nice things to know about operators in python

October 11th, 2007

There are a few operator related things that aren’t (as far as I know) common knowledge in python.

The first is the not in and is not operators. Basically, not is normally a unary operator and in and is are binary. Hower these two operations are valid

foo not in bar # same as "not foo in bar"
foo is not bar # same as "not foo is bar"

This is not obvious since not in this case doesn’t do unary negation but rather forms a new binary operator together with is or in.

Another cool thing is chaining comparisons. In C this would evaluate to False (or rather 0 since C lacks booleans).

int foobar = 3>2>1; // in C evaluates to (3>2)>1 = 1 > 1 = 0

In python however this will return True! This is because of python evaluating this chaining the same way as it is used in mathematics and so on.

foobar = 3>2>1 # in python evaluates to True

Basically this is translated by python to

foobar = 3>2 and 2>1

Both of these things might be common knowledge but I didn’t know about it and someone else might not as well.

Operator overloading madness in python (equation builder)

September 27th, 2007

I work extra helping out as a lab assistant in the newbie programming course at the university. One of the students had a question that really didn’t matter much for the assignment, but it inspired me to write this small class, to play around a bit with abusing operator overloading to generate equations.

Basically the class is used to set up a variable, and that variable can then be used freely to create functions with one variable like this

from equationbuilder import FuncVar
>>> x=FuncVar()
>>> f = 3*x**2 - 5*x + 8
>>> f(2)
10
>>> f(3)
20

The class for this is somewhat fun in my opinion as it constructs the final callable by using nested lazy lambda functions.

from operator import add, sub, mul, div

class FuncVar(object):
    def __new__(cls, *vals):
        cls.__rmul__ = cls.__mul__
        cls.__rdiv__ = cls.__div__
        cls.__radd__ = cls.__add__
        cls.__rsub__ = cls.__sub__
        return super(FuncVar, cls).__new__(cls)

    def __init__(self, callfunc=lambda x:x):
        self.callfunc = callfunc

    def __call__(self, arg):
        return self.callfunc(arg)

    def __add__(self, other):
        return self._rfunction(other, add)
    def __sub__(self, other):
        return self._rfunction(other, sub)
    def __div__(self, other):
        return self._rfunction(other, div)
    def __mul__(self, other):
        return self._rfunction(other, mul)
    def __pow__(self, other):
        return self._rfunction(other, pow)
    def __neg__(self):
        return -1*self

    def _rfunction(self, other, op):
        if isinstance(other, FuncVar):
            return FuncVar(lambda x: op(self(x), other(x)))
        return FuncVar(lambda x: op(self(x), other))

Download script

No real point to this blog post other than to show up a few programming tricks that some people might now know about. The __new__ constructor in python for an example works wonders to set __rmul__ and so on to the same method as __mul__.
__rmul__ and the other __rXXX_ methods is called when the class is being multiplied (or whatever) by an object “from the left” such as

3 * my_instance

and the left object throws an NotImplemented exception.

The object also defaults to having a __call__ method of

lambda x:x

which means that it will simply return the value passed. As other objects apply their operations, this method will be nested inside other call-methods and when the object is finally called it will all fit together like a nice puzzle :)

querypy - Write HTML without writing HTML (by using python)

September 20th, 2007

I’ve never liked writing HTML, so I thought that I could create a tool that does it for me… so that’s what I did.
Link here

querypy let’s you write HTML by using python with a syntax heavily inspired by jquery’s “chaining”. It uses overloaded operators to get a simple syntax that is small but still powerful. Both HTML4 and XHTML is supported, and can be chosen by using an utility function. querypy structures your code into an object oriented tree structure which makes it easy to reuse and change parts of you HTML.

It should be noted that querypy is not meant to be used as CGI to create dynamic content, but rather as a design tool for creating templates. Afterwards you should use a nice framework like perhaps django to implement the actual content.

Here is an example of usage:

from querypy import *

html, head, body = HTML(), HEAD(), BODY()
html + head
html + body

head + ( TITLE() + "A Hello World page" )

body + ( H1() + "Hello world" )

print doctype_html4_strict()
print html

Which outputs

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd">
<HTML>
  <HEAD>
    <TITLE>
      A Hello World page
    </TITLE>
  </HEAD>
  <BODY>
    <H1>
      Hello world
    </H1>
  </BODY>
</HTML>

If you prefer XHTML, simply add use_xhtml(True) before printing the root node.
Here is the same example using XHTML and more chaining.

from querypy import *

use_xhtml(True)
print doctype_xhtml_strict()
print HTML() + ( HEAD() + ( TITLE() + "A Hello World page" ) ) + \
  ( BODY() + ( H1() + "Hello world" ) )

and it’s output

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
 <head>
  <title>
   A Hello World page
  </title>
 </head>
 <body>
  <h1>
   Hello world
  </h1>
 </body>
</html>

The project can be found at http://querypy.com. Feel free to drop me feedback here or by email.