Intro to Twisted
I presented a short Intro to Twisted at the SoCal Piggies meetup in Santa Monica last night. It went pretty well and I think I did a good job explaining the basics. I know it took me a long time to even become even remotely familiar with the way Twisted operates, so I felt like this was a good choice for a five-minute presentation. Short, sweet, and to the point!
Here is is for your viewing pleasure:
I borrowed the image on slide five from the Twisted Intro, which is a very excellent introductory tutorial written by Dave Peticolas. If you are curious about Twisted, what it can do, or are even experienced and want a great refresher, it’s definitely worth your time to give it a look!
12.19.09On Using eval() in Python
Originally posted on StackOverflow.
I have used eval() in the past (and still do from time-to-time) for massaging data during quick and dirty operations. It is part of the toolkit that can be used for getting a job done, but should NEVER be used for anything you plan to use in production such as any command-line tools or scripts, because of all the reasons mentioned in the other answers.
You cannot trust your users–ever–to do the right thing. In most cases they will, but you have to expect them to do all of the things you never thought of and find all of the bugs you never expected. This is precisely where eval() goes from being a tool to a liability.
A perfect example of this would be using Django, when constructing a QuerySet. The parameters passed to a query accepts keyword arguments, that look something like this:
results = Foo.objects.filter(whatever__contains='pizza')
If you’re programmatically assigning arguments, you might think to do something like this:
results = eval("Foo.objects.filter(%s__%s=%s)" % (field, matcher, value))
But there is always a better way that doesn’t use eval(), which is passing a dictionary by reference:
results = Foo.objects.filter( **{'%s__%s' % (field, matcher): value} )
By doing it this way, it’s not only faster performance-wise, but also safer and more Pythonic.
Moral of the story?
Use of eval() is ok for small tasks, tests, and truly temporary things, but bad for permanent usage because there is almost certainly always a better way to do it!
| Posted in Python, Tech, Tutorials | No Comments »
Dynamically Determining the Variables for a Django Template
This is something that came up on StackOverflow, and I took the time to provide a very detailed answer. I haven’t posted in a while, and since I spent time on this answer, here it is verbatim. Enjoy!
I’d like to be able to instantiate a template from a file (presumably using the django.template.loader.get_template(filename) ), and then determine the set of variables that should be defined in whatever context it is passed.
(snip)
I tried synack’s answer and found that (with the replacement of filterexpression.token by filterexpression.var, to get the actual name of the variable without tags and so on) it returned the variables that are defined locally in the template, but did not work for variables that are defined in the parent that it extends.
So for example, suppose I have templates in two files:
toyparent.html:
{%block base_results%}
Django is {{adjective}}
{%endblock base_results%}
toychild.html:
{% extends "toyparent.html" %}
{%block base_results%}
{{block.super}}
I {{verb}} it.
{%endblock base_results %}
And I load the child template:
>>> toy=django.template.loader.get_template('toychild.html')
This renders properly:
>>> toy.render(django.template.Context(dict(adjective='cool',verb='heart')))
u'\n \nDjango is cool\n\n I heart it.\n\n'
But I can’t get the two variables from it:
>>> v=toy.nodelist.get_nodes_by_type(VariableNode)
>>> for k in v: print k.filter_expression.var
...
block.super
verb
You are able to visually inspect a template and observe the presence of any “Variable Node” objects in that template’s nodelist:
>>> from django.template import Template, Context
>>> t = Template("Django is {{ adjective }} and I {{ verb }} it.")
>>> t.nodelist
[<Text Node: 'Django is '>, <Variable Node: adjective>, <Text Node: ' and I '>, <Variable Node: verb>, <Text Node: ' it.'>]
These are of the type VariableNode, which is a class that can be directly imported for use in comparisons. Any Node instance has a get_nodes_by_type() method that can be called against a nodelist, which return all nodes of that type for the template. Example:
>>> from django.template import VariableNode
>>> varnodes = t.nodelist.get_nodes_by_type(VariableNode)
>>> varnodes
[<Variable Node: adjective>, <Variable Node: verb>]
So now you have a list of the variables for the template. This will need to be taken a step further to extract the actual name of each variable without peforming stupid string-slicing tricks on their repr names.
The variable name itself is stored in filter_expression.token for each VariableNode:
>>> varnodes[0].filter_expression.token
u'adjective'
And so a simple list comprehension gets us all of the variable names for the template:
>>> template_vars = [x.filter_expression.token for x in varnodes]
>>> template_vars
[u'adjective', u'verb']
So, not the simplest solution, but if there is a better way I don’t know about it.
Bonus: A function!!
from django.template import VariableNode
def get_template_vars(t):
varnodes = t.nodelist.get_nodes_by_type(VariableNode)
return [x.filter_expression.token for x in varnodes]
Ok, it’s not so complex after all!
Follow-up Edit: Getting variables from parent templates
(This follow-up is using the information from the updated question).
This is where it does actually get complex because the nodelist of the toy template is a single ExtendsNode (in this case).
>>> toy.nodelist
[<ExtendsNode: extends "mysite/toyparent.html">]
I would imagine that in larger templates there could be multiple ExtendsNode objects. Anyhow, if you inspect the ExtendsNode, and extract the parent template from it, you are able to treat the parent the same as my original example:
>>> enode = toy.nodelist[0]
>>> enode.parent_name
u'mysite/toyparent.html'
>>> parent = enode.get_parent(enode.parent_name)
>>> parent
<django.template.Template object at 0x101c43790>
>>> parent.nodelist.get_nodes_by_type(VariableNode)
[<Variable Node: adjective>]
And there is your adjective variable extracted from the parent template. To perform a test against an ExtendsNode you can import the class from django.template.loader_tags:
>>> from django.template.loader_tags import ExtendsNode
>>> ext = toy.nodelist.get_nodes_by_type(ExtendsNode)
>>> ext
[<ExtendsNode: extends "mysite/toyparent.html">]
So, you could do some tests against templates for the presence of an ExtendsNode and walk backwards to the parent template and individually get those variable names. However, this is starting to seem like a can of worms.
For example, if you were to do this:
>>> toy.nodelist.get_nodes_by_type((ExtendsNode, VariableNode))
[<ExtendsNode: extends "mysite/toyparent.html">, <Variable Node: block.super>, <Variable Node: verb>]
Now you’ve got the ExtendsNode and VariableNode objects and it just starts to get confusing. What do we do then? Do we attempt to ignore any block variables returned from such tests? I don’t know!!
In any case, this is the information you wanted, but I don’t think that this is a practical solution. I insist that there is still probably a better way. It might be worth looking into what you are trying to solve and see if there is another approach you can take.
| Posted in General | 1 Comment »
Logical Meltdown
I have been totally unable to accomplish any coding this week. I’ve been rewriting this application at work and while I’ve made some decent progress, I haven’t gotten a damn thing done this week and it’s frustrating as hell!
Of course now that it’s Friday and I’m feeling the urge to start writing more code having to do with dongs. What the fuck is wrong with me? Don’t answer that.
I figured that by explaining some of what I’m struggling with, some magic will happen. I need to blog more and work on my “explaining complex shit in a meaningful way” skills, so if this in any way interests you please read on.
WARNING: TECHNICAL SHIT BELOW!
The application I am rewriting is a MySQL database-driven app that has two interfaces:
- A command-line interface (CLI) utility written in Perl for direct manipulation (addition/removal of records) and querying (searching) of the database. Only engineers have access to this tool.
- A web application written in PHP for searching the database for non-engineers. Mostly support and operations staff use this. This also provides a hacked up “API” that returns a list of comma-separated values (CSV) of matching records.
There are a shit-ton of logical pitfalls in that code that have really thrown me off. Being that we have a Perl CLI tool and a PHP web app, the much of the code is duplicated for each language. The tool was originally written in Perl with the web app piece tacked on as an afterthought. At the time I was really into PHP’s native support for MySQL but was pretty much a novice so ended up making some downright lazy and bad choices.
With PHP and it’s bloated cousin Perl, there are so many ways to do the same thing that there really isn’t one right way to do things. Typically that means there are that many more wrong ways to do them. For example, horrible crap like this where we have PHP making a system() call to execute a Perl script from within index.php:
<table>
<tbody>
<tr valign="top">
<td>
<?
system('perl body.pl column="' . $POST['column'] . '" value="' . $POST['value'] . '" netmask="' . $_POST['method'] . '"');
?>
</td>
</tr>
</tbody>
</table>
Couple that with my prior lack of regimented programming style because I originally learned to code from the back of a Cracker Jack box (aka the internet), and what I am working with is pretty much a nightmare.
“Why is that?”, you ask? Well let me tell you!
First things first, the primary users of the app (aka the stakeholders) have been requesting some new features for a long time and with everything else going on it hasn’t really been a high priority for me. Especially because it was written in Perl, which I have grown to hate passionately since I started using Python.
Next, the systems from which the CLI tool runs are being replaced, which means the app has to be moved to the new systems. We run a Red Hat offshoot distribution of Linux, so we use RPMs to manage what gets installed on machines. What this means is that any Perl module dependencies need to be installed via RPM. With RPM and Perl there are always dependencies of your dependencies, and you can bet that some of those won’t be available in the repository. This opens a massive can of worms of having to download the module source and convert it to RPM. For those of you who are familiar with Perl’s CPAN and RPM, you’re probably thinking “Just use cpan2prm!”. Right, if only it were that simple…
You see, the new hosts are running a 64-bit kernel, which means that the subset of available packages is even smaller because packages have to be specifically built (or rebuilt) for 64-bit installation. Even if it’s the same files inside of the package, it still has to be re-packaged for 64-bit.
So, I was left with the choice of either uprooting the tool from its current location, wrangling package dependencies, rebuilding packages, modifying the Perl code to shoehorn it into the new systems all the while overlooking the implementation of the requested features; or rewriting it using Python & Django.
Naturally, I chose to rewrite it because the thought of having to trudge through antiquated Perl hackery gave me diarrhea and I really don’t want to ever touch Perl or PHP again if I can help it. By rewriting the app in Python, I can implement the new features and bring the code into the present at the same time without having to go through bouts contemplating suicide. Annnddd… That leads us back to the nightmare that is Perl and PHP.
As I mentioned above, with Perl & PHP there are so many ways to do things that most of them are convoluted, confusing, hard-to-read, and just plain fucking wrong. With Python–and Django for that matter–there is usually only one right way to do things, and that way is pretty much always elegant. Plus, one of the most beautiful things about Django is that it wholly and completely abstracts the database interaction from the source code. That means not having to write complex SQL statements manually, and the less SQL I have to write, the better.
In meeting with the stakeholders on rewriting the application, we decided that all of the energy should be invested in replacing the web app with a real API with a RESTful interface. If you don’t know the basic premise of REST (which is a methodology, not a protocol), it’s that URLs should be easy-to-read, self-describing and beautiful. Beautiful URLs? WTF? How can a URL be beautiful? I’m glad you asked!
The old PHP version of the API is very heavy on the query string, requiring GET variables to be passed in the URL itself which is not only cumbersome, but can get confusing and hard to read very quickly:
http://server.whatever.com/api.php?show=all&user=jathan
Compare that to a RESTful version of the same:
http://server.whatever.com/api/show/all/jathan/
This is where Django excels because it encourages pretty URLs by making it easier to use pretty URLs than not to.
So, cutting this post off at the knees, I’ve been pounding on this for the past few weeks, learning a lot about how NOT to do things from my old code and how TO do things using Django as my guide. I made a lot of progress, got the API working, and a basic search interface that returns stuff all using the same code.
But this week, I’ve been stuck. I haven’t made any progress, have been getting really confused, and just generally bashing my skull into the keyboard. I think I’m just fried from trudging through all of Perl’s curly braces, semicolons, ampersands, dollar signs… OMG THE SYNTAX! KILL ME NOW!!
Besides, it’s Fried-day anyways, so I hope that just by spelling this all out here that I’ll be able to dig back into the code after the weekend with a renewed perspective.
p.s. Fuck Perl.
p.p.s. Fuck PHP too, but wrap that rascal. She gets around!
| Posted in Nerd Up, Python, Tech | No Comments »
More Dongs on a Friday. Surprise!!
Due to popular demand, mkdong has been modularized!! I present to you dong.py version 0.0.1.
Enjoy:
#!/usr/bin/env python
''' MKdong turned into a module. I think I was smoking crack and/or high on Red Bull this day.
Dong goes into underpants Dong goes into boxers Dong goes into vagina Sperm comes out of dong Sperm goes into vagina Sperm goes into egg Dong goes into mouth Pee comes out of dong Pee goes into toilet Why am I iterating this crap? '''
import os, sys
blue = '\\e[0;34m' # blue
class Dong:
def __init__(self, maxlen=40, color=blue):
self.maxlen = maxlen
self.color = color
self.dong = None
def mkdong(self):
self.dong = '(_)/(_)'
for i in range(self.maxlen): self.dong += '/'
self.dong += 'D'
os.system('echo -e "%s%s"' % (self.color, self.dong))
class Sperm(Dong): def init(self, count=500): self.count = count self.spermcount() Dong.init(self)
def __repr__(self):
return '' % self.count
def tighty_whities(self):
print 'Tighty Whities lowered your sperm count!'
self.count -= 50
def boxers(self):
print 'Boxers raised your sperm count!'
self.count += 50
def spermcount(self):
print 'Sperm count is %d' % self.count
def bike_seat(self):
print 'asdjfoisadjs'
pass
def radiation(self):
print ':-x'
pass
def castration(self):
print ':('
pass
def smoke(self):
print 'awwww yeahhhhh.'
pass
class Egg(Sperm): pass
if name == 'main':
try:
donglen = int(sys.argv[1])
except:
print "usage: mkdong "
sys.exit()
if donglen > maxlen:
print 'warning: a %s" dong is too big! cannot be longer than %s"!' % (donglen, maxlen)
sys.exit()
else:
d = dong(donglen)
d.mkdong()</pre>
There’s really no excuse. I should be ashamed of myself but I’m not.
| Posted in Python | No Comments »
Virginia Blows
That is all. Been here a week and while I’ve had fun, it feels like it’s been a month.
I miss my friends and my family that are in this area, but that’s still not enough for me to want to stay here for any meaningful length of time. But hey, it’s been like 13 months since last time, so it’s not all bad. I have a gracious host, fun coworkers, and also the internet.
But despite that, when I’m here it feels like I’m not living my own life. Maybe it has something to do with the fact that I couldn’t wait to leave when I moved away over two years ago, and now every time I come back that feeling quickly returns.
Blah… Whatever, bitches.
Actually, I think it’s your fault. Yes, you!
Here enjoy this: Someone ported mkdong to Perl. Not sure how I feel about that. Bittersweet?
Taking over the internet, one dong at a time.
| Posted in General | 1 Comment »
Blue Dongs for a Friday Afternoon
Today I wrote an awesome program called mkdong that will make a dong of your desired length and print it to your terminal, like this:
% ./mkdong usage: mkdong <length> % ./mkdong 5 ()/()/////D % ./mkdong 25 ()/()/////////////////////////DThat last one is impressive, isn’t it? Hmm… Yeah, it’s Friday. What do you want from me? I still got work done! Cool thing is if the dong is too big, well then it throws an error:
% ./mkdong 60 warning: a 60" dong is too big! cannot be longer than 40"!“What is the point of this?”, you might ask yourself. That’s a good question. I’ve been so busy with other shit lately that I’ve barely had time to code. I suppose I was itching to write something, anything… Dongs!!
It all started harmlessly enough with a silly AIM conversation with my coding buddy at work. We were talking about a bug, and well, read on and you’ll see. It regressed quickly.
So I took the stupidity and ran with it and mkdong was born!
The initial dongs were a little primitive and sickly looking. So I took his suggestion and improved their visual style. Here is how it turned out:
#!/usr/bin/env python
import sys
maxlen = 40
try: donglen = int(sys.argv[1]) except: print "usage: mkdong <length>" sys.exit()
if donglen > maxlen: print 'warning: a %s" dong is too big! cannot be longer than %s"!' % (donglen, maxlen) sys.exit() else: dong = '()/()' for i in range(1, donglen): dong += "\" dong += 'D'
print dong We laughed. We joked. We Tweeted. And then it regressed even further:
A feature request! I had to make it print in blue! But to do that I had to replace all of the “\” that make up the dong itself, with “/” so as to not have the ANSI escape codes eat up the extra backslashes. (Backslashes are interpreted characters, duh.) I also had to replace the print statement with a system call to echo -e so that the colorization would be interpreted. This is high tech shit, man!!
And then I released it to the public. So there you have it. Here is the final release of mkdong 2.0 for your pleasure:
#!/usr/bin/env python
import os, sys
maxlen = 40 color = '\\e[0;34m' # blue
try: donglen = int(sys.argv[1]) except: print "usage: mkdong <length>" sys.exit()
if donglen > maxlen: print 'warning: a %s" dong is too big! cannot be longer than %s"!' % (donglen, maxlen) sys.exit() else: dong = '()/()' for i in range(donglen): dong += '/' dong += 'D'
os.system('echo -e "%s%s"' % (color, dong)) Use it well. And remember they aren’t bugs, they’re dongs! Squish? Gross.
| Posted in Life, Python | 3 Comments »
Python List Comprehension for Dummies
So I code a lot. I code in Python a lot. You might say I love Python. I might say you’re right.
One of the most powerful things about Python is its ability to iterate over ANYTHING as if it were a list. Lists, tuples, dictionaries, and even strings can all be iterated quickly and elegantly. Python also introduces a concept known as list comprehension which allows you to do rather complex filtering of list contents within a single statement.
To illustrate how awesome and powerful list comprehension is, let’s start with a basic example that is NOT using it:
>>> mylist = [1,2,3,4,5] >>> for item in mylist: ... if item % 2 == 0: print item, 'is an even number.' ... 2 is an even number. 4 is an even number.So, let’s assume that we want to identify all even numbers inside of
mylist, and put them into a new list called evens the old-fashoned way:
>>> mylist = [1,2,3,4,5] >>> evens = [] >>> for item in mylist: ... if item % 2 == 0: evens.append(item) ... >>> evens [2, 4]Why the old-fashioned way sucks First things first, the empty list called
evens had to be declared ahead of time. This is because when we looped thru the list called mylist using the for statement, when the if test is performed on each item we have to reference evens by name to append() the even numbers to it.
Why list comprehension rocks With list comprehension, the logic that isolates the even numbers and the declaration of the list that will capture this output are compressed into a single statement:
>>> mylist = [1,2,3,4,5] >>> evens = [i for i in mylist if i % 2 == 0 ] >>> evens [2, 4]The logic is encapsulated in [square brackets] indicating that the output will be a list. The list comprehension itself is the logic between the brackets that determines what will be in the list that it spits out.
So list comprehensions at their most basic level allow for compression of code and streamlining of logical statements. Advanced usage of list comprehension can get pretty silly, but then so can nested loop statements. It supports nesting as many statements as you can throw at it so longs as they are syntactically correct.
If you find yourself coding shit like this:
>>> losers = ['Joe','Jim','Jon','Jen']
>>> for u in losers:
... if u.startswith('J'):
... if u.endswith('n'):
... if u != 'Jon':
... print u
...
Jen
Then maybe list comprehension is for you:
>>> [u for u in losers if u.startswith('J') and u.endswith('n') and u != 'Jon']
['Jen']
No offense to anyone named Joe, Jim, or Jon.
| Posted in Python, Tutorials | No Comments »
| Posted in Python, Tutorials | No Comments »