querypy - Write HTML without writing HTML (by using python)

September 20th, 2007
5 comments

I’ve never liked writing HTML, so I thought that I could create a tool that does it for me… so that’s what I did.
Link here

querypy let’s you write HTML by using python with a syntax heavily inspired by jquery’s “chaining”. It uses overloaded operators to get a simple syntax that is small but still powerful. Both HTML4 and XHTML is supported, and can be chosen by using an utility function. querypy structures your code into an object oriented tree structure which makes it easy to reuse and change parts of you HTML.

It should be noted that querypy is not meant to be used as CGI to create dynamic content, but rather as a design tool for creating templates. Afterwards you should use a nice framework like perhaps django to implement the actual content.

Here is an example of usage:

from querypy import *

html, head, body = HTML(), HEAD(), BODY()
html + head
html + body

head + ( TITLE() + "A Hello World page" )

body + ( H1() + "Hello world" )

print doctype_html4_strict()
print html

Which outputs 

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd">
<HTML>
  <HEAD>
    <TITLE>
      A Hello World page
    </TITLE>
  </HEAD>
  <BODY>
    <H1>
      Hello world
    </H1>
  </BODY>
</HTML>

If you prefer XHTML, simply add use_xhtml(True) before printing the root node.
Here is the same example using XHTML and more chaining.

from querypy import *

use_xhtml(True)
print doctype_xhtml_strict()
print HTML() + ( HEAD() + ( TITLE() + "A Hello World page" ) ) + \
  ( BODY() + ( H1() + "Hello world" ) )

and it’s output

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
 <head>
  <title>
   A Hello World page
  </title>
 </head>
 <body>
  <h1>
   Hello world
  </h1>
 </body>
</html>

The project can be found at http://querypy.com. Feel free to drop me feedback here or by email.

buffi Programming & scripting, Python

python statements are (or rather can be) messy

September 15th, 2007
3 comments

I recently realized something about python that I wasn’t aware about. Python statements doesn’t force you to use whitespace before or after them.

When it comes to the = statement, this is rather obvious I guess… I sometimes write stuff like

foo="bar"

In other cases I had no idea it worked like this… for an example this is perfectly valid

print"hello"

In a language that uses indentation to block statements for nice readability, it feels kind of weird to allow this since it can make your code really really messy. How about these examples that are all perfectly valid:

"foo"if"bar"else"foobar"
for(x,y)in[(1,2),(3,4)]:print(x,y)
[(y)for(x,y)in[("foo",2),("bar",4)]if"foo"in(x)]

Simply forcing whitespace before and after statements in python would in my opinion be wise… even though you really REALLY shouldn’t write code like this anyways.

buffi Programming & scripting, Python

Small jQuery-plugin for enabling/disabling animations

September 9th, 2007
No comments

I’ve been playing around a bit with jQuery for about a month as I’m developing an AJAX-driven admin interface for pici which is coming along nicely.

For this I’m also using some stuff from Effects in jQuery and some might concider animations and so on a bit bloat (although they are so pretty!) so I wanted some way of toggling them. That is, the user should be able to disable them and in that case simply have the data removed/added instead of nicely scrolled up/down.

I wrote a very small plugin for this called jquery.customeffects.js which is probably not a very appropriate name, but whatever.

A functional but not very pretty demo can be found here.

The plugin is only about 20 lines of code, but does it’s job and I thought I might as well put it up here, and not ONLY post about python all the time.
Download here

Using it is explained in the demo. Basically use 

$.setFancyAnimations(true)

to enable fancy animations and 

$.setFancyAnimations(false)

to disable them.

Then use 

hideCustom(optional_params)
showCustom(optional_params)
toggleCustom(optional_params)

to handle the displaying/hiding of data just like the normal show and hide functions in jQuery.

buffi Programming & scripting

geektorrent goes live! First django powered bittorrent tracker/indexer?

September 3rd, 2007
11 comments

geektorrent is a tracker made for geeks. It is supposed to be a place to gather legal content such as programming screencasts/conferences, gaming videos, physics papers and basically anything vaguley “geeky”. It is however NOT a tracker for the latest movie and software releases… there are plenty of those. Instead I hope to target a more specific group of people, and bring lots of interesting content together at a single site.

The basic rule is “don’t upload it if you think that the creator wouldn’t want it there”. Old tv-documentaries might be copyrighted, but if there are no profit to be made any more, I see no reason for not allowing it. If any copyright owner spot anything on the site that they feel have been uploaded against their will, I will of course remove the content. I want to be a nice guy and the best way for preservation of good content is global distribution.

To be honest I’m not 100% sure that this is a niche that needs filling, and that people will start to contribute to the site by uploading nice content, but there are only one way to find out :)

Visit me!
geektorrent

And now for some of the technology behind the scenes…

I’ve been planning on creating a torrent tracker for quite some time and have explored my options of technologies to use. The only one thing I was 100% sure of was that I was not going to use TBSource. I downloaded their source code to browse through it for ideas and it was just horrible. Since this is a extremely common php-application I was actually a bit surprised on how horribly designed it is. I could probably write an entire entry about this, but I’ll leave that for later… let’s just say that a torrent tracker/indexer is not a very hard thing to write, and I’m surprised that there aren’t any better open source options.

I did however have a look at using XBT Tracker as a backend, which did look quite nice, but turned out to be a bitch to interface with django, due to the lack of binary storage fields and it’s use of binary hashes. It was also a bit non responsive, and lacked decent documentation, but all in all it seems to be a decent option… just not with django. I actually made a complete working implementation using XBTT but it was so messy that I ended up scrapping that idea completely.

I basically ended up doing the entire thing in django, including the backend tracker. This will give some overhead (apache loves overhead!) for the announce, but will be easy to scale, and my implementation would actually not be very hard at all to change to include a separate server for the backend if needed later. I do tend to avoid premature optimization though, and I am 100% sure that this will scale just fine for now.

As far as I know there are no major (or minor?) bittorrent trackers/indexers powered by django so far, so this might be the first one (please correct me if I’m wrong) :)
Google only turns up an incomplete project anyways.

I do intend to open source my code base as a general django bittorrent tracker once it is up to my rather high standards, and proper documentation has been written, so hopefully people will have a viable option to TBSource rather soon ;)

buffi Programming & scripting, Python

Django (SVN) and non-unicode HTTP GET-data

August 28th, 2007
2 comments

Django merged with the unicode branch a while ago. If you are still using 0.96 then this doesn’t affect you, but if you are running the SVN-versionen then it does.

A HTTP request can have GET arguments such as http://mydomain.com/mysite/?foo=bar&banana=purple which has the GET data foo = “bar” and banana = “purple”. Not all characters are allowed to be in these request-URL’s which forces an alternative syntax which is the character % followed by the hexadecimal representation of the character. For an example the character with ascii value 255 can be represented as %FF.

Bittorrent is a good example of a protocol that sends data (for an example it’s info_hash which is a 20 byte sha1 hash) as HTTP GET-data which uses this encoding. Django does however not handle this well at all since the merge with the unicode branch.

Here is an example:

Let’s say that you have a page at http://mydomain.com/ which you want to send the character with ordinal 238 (EE in hex) as a GET argument.
http://mydomain.com/?info_hash=%EE

You might then have a views that does something like this

def handle_stuff(request):
  get_data = request.GET.copy()
  info_hash = get_data["info_hash"]
  assert(False) # for debugging

Assert false will bring up djangos VERY nice debug screen if you have debug turned on in your settings.py and show that 

get_data = <MultiValueDict?: {u'info_hash': [u'\ufffd']}>
info_hash = u'\ufffd'

u’\ufffd’ is the unicode symbol for “OH FUCK THIS DIDN’T WORK” (or something like that).
Basically… forcing unicode in django broke recieving non unicode GET-data through the GET MultiValueDict.

Ok… so why am I posting it here and not as a bug report in django?
Well, I did also post a bug report about this today but the reason for posting here is that it exists an ok workaround for now, and I hope that this post might assist people in finding it.

request.META['QUERY_STRING'] hold the raw query string that is requested for a page request, and you can use the parse_qs (or parse_qsl) method in the cgi module to get the non-violated GET-data. It even properly escapes the %XX notation into “regular” characters.
Thank you Crast in #django @ freenode for reminding me about the existence of META['QUERY_STRING'].

get_data = cgi.parse_qs(request.META['QUERY_STRING'])
info_hash = get_data["info_hash"][0]
info_hash_as_hex = get_data["info_hash"][0].encode("hex")

Ugly? Yeah…
Functional? Yup!

buffi Programming & scripting, Python

Simplified IP-ban middleware

August 23rd, 2007
1 comment

This is based on the last post where I showed a simple middleware for IP banning in django.
In the comments, SmileyChris mentioned that simply rendering the template in the middleware would eliminate the need for a url configuration and view for the IP ban middleware. This was actually a really great idea, since it also removes the need to check the path for the ban page, and everything becomes a lot cleaner. Thanks :)

I rewrote the middleware to use this method instead, and it now looks like this

from pici.picipage.models import Ban
from django.shortcuts import render_to_response

class IPBanMiddleware(object):
  """
  Simple middleware for taking care of bans from specific IP's
  Redirects the banned user to a ban-page with an explanation
  """
  def process_request(self, request):
    ip = request.META['REMOTE_ADDR'] # user's IP

    # see if user is banned
    try:
      # if this doesnt throw an exception, user is banned
      ban = Ban.objects.get(ip=ip)
      reason = ban.reason

      # return the "ban page"
      return render_to_response("subpages/banned/banned.html",
        {"reason": reason})
    except Ban.DoesNotExist: # not banned! goodie
      pass

No additional view or url-patterns needed, simply add the Ban model that I wrote in the last post into your model, and create the template and you are good to go!

My Ban model looked like this (same as last post), but can of course be improved to fit your needs.

class Ban(models.Model):
    ip = models.IPAddressField()
    reason = models.TextField()

    def __str__(self):
        return self.ip

    class Admin: pass

Thanks for the idea, and I hope that other people will start “bashing my code” as well ;)
Constructive criticism is an awesome thing, and I know that there are a whole lot of more experienced django developers than me out there.

buffi Programming & scripting, Python

Simple middleware for database driven IP ban using django.

August 21st, 2007
6 comments

The internet is stupid, and your users will misbehave. Therefore every homepage should have some way of banning users from posting content, or perhaps in some cases from viewing your page altogether.
I wanted a simple way to ban users from accessing one of my django made homepages, and then having them redirected to a page with an explanation why. The IP’s and explanations should be entered through the django admin interface. For this django middleware is a decent choice.

First of all, lets create a Ban model in your models file. Something like this should do (you might want timestamps and other stuff as well, I don’t really need it).

class Ban(models.Model):
    ip = models.IPAddressField()
    reason = models.TextField()

    def __str__(self):
        return self.ip

    class Admin: pass

Do a python manage.py syncdb to insert the model into your database.

Now it’s time to create the middleware. The middleware can be placed anywhere reachable from your python path.
I named my middleware file ipban.py and placed in in a subfolder called “middleware”. Also create an empty __init__.py file in this directory if you do so as well to indicate that it holds python files. The content of my ipban.py is

from django import http
from pici.picipage.models import Ban

class IPBanMiddleware(object):
  """
  Simple middleware for taking care of bans from specific IP's
  Redirects the banned user to a ban-page with an explanation
  """
  def process_request(self, request):
    ip = request.META['REMOTE_ADDR'] # user's IP
    path = request.path # requested path

    # see if user is banned
    try:
      # if this doesnt throw an exception, user is banned
      Ban.objects.get(ip=ip) 

      # only redirect when not already at the ban page
      if not path.startswith("/banned/"):
        return http.HttpResponseRedirect("/banned/%s/" % ip)
    except Ban.DoesNotExist: # not banned! goodie
      pass

pici.picipage.models should of course be changed to point to your own models module containing the Ban model.
This middleware checks if the users IP is in the list of banned users at every request. If the user is banned, then he is redirected to the ban-page “/banned/ipgoeshere/”.

Now open your settings.py file and add this class to your MIDDLEWARE_CLASSES.
In my case I added 

'pici.middleware.ipban.IPBanMiddleware',

Add a urls.py entry with something like this (I’m somewhat lazy at regular expressions). Obviously replace miscviews.banned with whatever you are going to call your view-handling method and it’s module.

(r"^banned/([^/]+)/$", miscviews.banned),

Add the handler for the banning in your view. Mine looks like this. Replace “subpages/banned/banned.html” with the path to your template.

def banned(request, banip):
  try:
    ban = Ban.objects.get(ip=banip)
    reason = ban.reason
  except Ban.DoesNotExist:
    reason = "None given"
  return render_to_response("subpages/banned/banned.html",
    {"reason": reason}, context_instance=RequestContext(request))

Finally add the template and you are done!
Adding the template should be straight forward if you know anything about django, but it can look something like this

{% extends "index.html" %}

{% block main_content %}
  <div class="normal_content_centered">
    <h1>BANNED!</h1>
    <p>The reason for your banning is:</p>
    <p><b>{{ reason }}</b></p>
    <img src="url_to_crying_baby_goes_here.jpg" />
  </div>
{% endblock %}

The image of a crying baby is essential.

This can of course quite easily be optimized by using sessions to avoid having to do the lookup for each request, however a very simple database hit per request is not a big deal in my case so I’ll save that for later. Premature optimization is evil and all that.

Have fun banning users from your homepage :)

edit: Improved version up

buffi Programming & scripting, Python

No good way using django to interface with external applications using binary data.

August 12th, 2007
No comments

I posted a ticket about it being seemingly impossible to use djangos db api (even with custom SQL) to handle binary data.
http://code.djangoproject.com/ticket/5135
and got the reply.

Storing binary data was unsafe before (what if your binary data contained a zero byte?), so it was kind of lucky — and unsupported — that it worked at all. It just works even less well know.

The real fix here is something like #2417 (adding a propery binary field type). The current workaround is to use base64 encoding (or base96 or some other binary->ascii encoding) on the data before storing it. There’s nothing we can do at the text field level, since we are assuming Unicode strings for text and databases obviously use an encoding when they store stuff, hence we have to convert between the encoding and Python Unicode objects.

Although I do agree with him about a new BinaryField being the best way to solve this, I also realize that this means that django does not have any nice way to handle binary data at all right now, and that it probably won’t have it any time soon :/
I will have to resort to ugly “external” MySQLdb hacks for now.

Using base64 or similar is of course a good choice if you write your tables from scratch, but interfacing with a legacy application makes it impossible without digging through a whole lot of source code, and modifying that application (which also makes it a non-generic solution that pretty much requires a ugly fork of that app to work).

One would really think that interfacing with external application should be big enough to at least be able to handle “unsupported” field types using custom SQL. I mean… there’s even a chapter in djangobook about using legacy databases / external apps.

I hope that a BinaryField will eventually be incorporated in django, but I’m not the right person to do it since I only have experience using mysql, and it should of couse be generic :)
Due to django using unicode all the way down to custom SQL, I bet there will be quite a few issues with creating one though.

Edit: Also, this is important.
This only deals with the SVN-release.
0.96 was releases before the merge with the unicode-branch, and custom SQL to blob fields work there.

buffi Programming & scripting, Python

Django SVN-release can’t handle binary data insertion?

August 11th, 2007
No comments

Django does not have a BinaryField, or BlobField or whatever you want to call it, which is a bit sad. I made a fix for this a few posts back that worked fine in 0.96. However when trying the SVN-release it seems like everything that goes to the database is first turned into unicode, which of course includes the binary data.
Even doing custom SQL won’t work

An example from my code.
The info_hash field is a blob

def create_xbt_file(info_hash, timestamp):
  query = "INSERT INTO xbt_files (info_hash, mtime, ctime) VALUES (%s, %s, %s)"
  from django.db import connection
  cursor = connection.cursor()
  cursor.execute(query, [info_hash, timestamp, timestamp])

This throws a nasty UnicodeDecodeError whenever a byte with a position that unicode doesn’t like is in info_hash.

If I bypass django completely using MySQLdb, then it works fine

def create_xbt_file(info_hash, timestamp):
  import MySQLdb
  db = MySQLdb.connect("localhost", DATABASE_USER, DATABASE_PASSWORD, DATABASE_NAME)
  cursor = db.cursor()
  query = "INSERT INTO xbt_files (info_hash, mtime, ctime) VALUES (%s, %s, %s)"
  cursor.execute(query, [info_hash, timestamp, timestamp])
  db.close()

This obviously is a ugly hack, but it seems to be the only way to fix it right now.
If I’m wrong then please correct my since I would obviously not like to have this in my code.

buffi Programming & scripting, Python

Quick and ugly fix for Data truncated error using FloatField in django 0.96

August 5th, 2007
2 comments

I have a FloatFIeld in one of my models that looks lite this

score = models.FloatField(default=0.0, max_digits=2, decimal_places=1)

This would raise this error for me when inserting certain floats (since there are too many decimal places I guess…)

Data truncated for column 'score' at row 1

A very easy fix for this is to make a string representation of the float and then insert that into the FloatField. This works just fine due to “duck typing”. The string quacks like a float ;) 

fixed_float = str(my_float)[:3] # only works for floats < 10 obviously

buffi Programming & scripting, Python