Intro to Twisted
I presented a short Intro to Twisted at the SoCal Piggies meetup in Santa Monica last night. It went pretty well and I think I did a good job explaining the basics. I know it took me a long time to even become even remotely familiar with the way Twisted operates, so I felt like this was a good choice for a five-minute presentation. Short, sweet, and to the point!
Here is is for your viewing pleasure:
I borrowed the image on slide five from the Twisted Intro, which is a very excellent introductory tutorial written by Dave Peticolas. If you are curious about Twisted, what it can do, or are even experienced and want a great refresher, it’s definitely worth your time to give it a look!
05.26.10The White Room
(Originally journaled by me on 2003-11-15)
I just awoke from a dream in which I found myself in a room with two girls and two older women, all obviously related. The grandma (in her 50′s), mother, and two daughters. I was sitting across from them at a table in a white room with no door, high ceiling, and a ledge off to the right wall with a tall window. Words were written/painted on the window and there was some gimp-like creature up on the ledge.
I had the feeling of being kidnapped because I had no recollection of going there or how I got there. I was playing it cool however, inquiring casually thru conversation about where we were.
It seemed as if nowhere. When asking about the girls they could not remember the age gap between them and looked accusingly at their mother who just smirked.
They all had black hair. The sisters were hot, with kind large features (butt, boobs, teeth, nose, etc.). The grandma never spoke; nor the mom.
When I inquired how I knew them, the girl on the left smiled knowingly and said I used to work with her. This was after she used the word ‘grep’ in conversation. She asked me if I remembered the screen name “heart with an a” and I said it sounded familiar.
I was mildly attracted to her. I felt relieved to meet someone new. Thru more conversation she wrote some C code on a napkin and I inquired if she was going to school or was self-taught.
Things started getting weird. I think the other sister was jealous. Someone threw something like clubs or bowling pins at the gimp-like creature in the window ledge… It got pissed.
It shrieked and cursed. I felt a connection between it and the girl on the right. I think it was the mom that threw the clubs at the gimp.
I think the grandma eluded to the mom to do it.
For some reason I did the same and the gimp jumped up and down and hissed and jumped off the ledge.
The next thing I realized is that the gimp is the girl on the right. She is a prisoner somehow because of the mom. She is the bad sister.
She writes things on the windows and receives gifts from some godlike being. They are S.O.S. messages.
I find myself on the ledge trying to atone for throwing the clubs at the gimp. There are gold coins and I notice pillars of white in each corner of the room. The coins are gifts from the god to the gimp.
I start to feel immense sorrow and pain. I can’t move or think. I am overwhelmed. I am being attacked. I collapse in pain screaming and crying in depression and sadness. I feel her pain.
I wake up in a white room. I see no one, I hear no one, but I know I am supposed to escape. I know I am a prisoner of the bad girl. The good girl is trying to help me. She says the bad girl has all the keys but she took them, and is helping me unlock doors (not physically, the keys just appear and the doors open). I am running down a hallway with the same lighting and blue carpet. It is very surreal. I am terrified realizing I am trapped and trying to escape without really knowing why or how.
I fail.
The puzzle resets. It’s a game. I’m still trapped. I wake up FOR REAL.
I am lying in bed and I can’t move. My bedroom is lit like the hallways in my dream. I can see my hall from my bed. In the lucid state, unable to clearly see, I see the puzzle. It is moving and self-contained but I can’t figure it out.
I start waking up. I get up. I feel groggy. I walk out to see my friend Megan on the couch. REALITY. I’m still scared.
I go to the spare room and type “heart with an a” into Google.
Nothing relevant comes up.
I’m awake. But the dream was real. What the fuck was that!!
I know my room was similar to the room in the dream. The kitten was in the window ledge. Was she the gimp? I had thrown a pillow at her earlier before the dream because she was making a lot of noise.
04.23.10Friday Cutions
Words ending with -cution are surprisingly similar!
allocution
noun
a formal speech giving advice or a warning.
elocution
noun
the skill of clear and expressive speech, esp. of distinct pronunciation and articulation.
electrocution
verb [ trans. ]
a formal “speech” giving “advice”, esp. of “distinct” lightning bolts.
I don’t think that last one means what I think it means.
| Posted in General | No Comments »
All you need to know is…
Doodling is awesome!!
What am I?
A bald, large-breasted bust mounted on a banana stand? I’m so confused! This is what meetings do to my mind.
A doodle of a kitty cat
Done with red, green, & black fine point Sharpies
So far this year, I haven’t been blogging much at all, so today I decided I’m going to start sharing my taste for “fine” art with the internet. At the very least I’ll be posting all of the random crap I doodle to fill the gaps between witty outbursts or technical diatribes.
Frozen: A Scary & Innovative Horror Flick
Go see this movie.
Watch the trailer for Frozen, and then check out the official site.
I had the pleasure of meeting the writer & director, Adam Green, this past weekend. My friend Robert and he are old buddies from college who just reconnected. Adam invited us to go to a screening in Santa Monica on Saturday after which there was a meet & greet with the cast & crew. After that we went out for drinks and I had a really good time. Ahhh, schmoozing, the second-oldest profession.
So, not only do I like the guy and want to see his movie succeed, but I genuinely thought the movie was a good time and I want to get more people out there to see it!
It’s an independently produced horror movie, which already stacks the odds against it. It is also an original production, not a remake, which further hardens the sell. It seems most horror fans out there cry for new original stories over remakes, but when it comes time to show their support, movies like Frozen get overlooked. This is sad.
I am a huge horror fan, and I am one of those people who is disappointed with the glut of remakes out there. So I am taking some of my time to get the word out there about Frozen because we need more movies like it!
It was an intense flick that had me gripping the edge of my seat for most of the movie. It grabbed me and didn’t let go, had me cringing and going “OHHHHHH” at some of the gory scenes. At a few moments I was so uncomfortable that I had to disengage by looking away or pretending I was in a happy place. To me, that is a sign of a good movie. It affected me. It wasn’t hum-drum, boring, or predictable. It still had a few of those campy, horror movie moments and I think that is where its charm is. If a horror movie can be charming, that is.
The acting was really good, and I think that Adam’s directing style had a lot to do with this. The entire movie was shot “practically” which means “on location”. There was no green-screen, no studio, no tricks. It was all filmed outdoors, on a ski slope, with the actors IN the gondola.
During the meet & greet they said that the actors had to be up on the gondola for six hours at a time. The filming was done on a ski resort in Utah and the lift was so long that to get off they had to ride all the way to the top of the mountain and all the way down, which took over an hour. So they had to limit the amount of interruptions in order to maximize filming time.
All the scenes in which it is snowing outside are real. When you can no longer see the actors’ breath, that’s because they are running out of body heat. That seems brutal to have endured that during filming for the actors, but I don’t feel too bad for them. THIS IS HORROR!
I felt like some of the scenarios were a little over-the-top, but I am ok with it. Isn’t that the point of a good horror film? Personally, I don’t want too much realism. I want fear, death, blood, and other delicious things that would probably never happen in real life. I definitely wouldn’t enjoy seeing people die in real life, I promise.
If you’re in for a good thrill, check your local movie listings and check out Frozen. I mean, come on, being that the majority of the Northeastern U.S. is under feet of snow & ice right now, how appropriate is this film?
VERY.
| Posted in Movies | No Comments »
On Using eval() in Python
Originally posted on StackOverflow.
I have used eval() in the past (and still do from time-to-time) for massaging data during quick and dirty operations. It is part of the toolkit that can be used for getting a job done, but should NEVER be used for anything you plan to use in production such as any command-line tools or scripts, because of all the reasons mentioned in the other answers.
You cannot trust your users–ever–to do the right thing. In most cases they will, but you have to expect them to do all of the things you never thought of and find all of the bugs you never expected. This is precisely where eval() goes from being a tool to a liability.
A perfect example of this would be using Django, when constructing a QuerySet. The parameters passed to a query accepts keyword arguments, that look something like this:
results = Foo.objects.filter(whatever__contains='pizza')
If you’re programmatically assigning arguments, you might think to do something like this:
results = eval("Foo.objects.filter(%s__%s=%s)" % (field, matcher, value))
But there is always a better way that doesn’t use eval(), which is passing a dictionary by reference:
results = Foo.objects.filter( **{'%s__%s' % (field, matcher): value} )
By doing it this way, it’s not only faster performance-wise, but also safer and more Pythonic.
Moral of the story?
Use of eval() is ok for small tasks, tests, and truly temporary things, but bad for permanent usage because there is almost certainly always a better way to do it!
| Posted in Python, Tech, Tutorials | No Comments »
Dynamically Determining the Variables for a Django Template
This is something that came up on StackOverflow, and I took the time to provide a very detailed answer. I haven’t posted in a while, and since I spent time on this answer, here it is verbatim. Enjoy!
I’d like to be able to instantiate a template from a file (presumably using the django.template.loader.get_template(filename) ), and then determine the set of variables that should be defined in whatever context it is passed.
(snip)
I tried synack’s answer and found that (with the replacement of filterexpression.token by filterexpression.var, to get the actual name of the variable without tags and so on) it returned the variables that are defined locally in the template, but did not work for variables that are defined in the parent that it extends.
So for example, suppose I have templates in two files:
toyparent.html:
{%block base_results%}
Django is {{adjective}}
{%endblock base_results%}
toychild.html:
{% extends "toyparent.html" %}
{%block base_results%}
{{block.super}}
I {{verb}} it.
{%endblock base_results %}
And I load the child template:
>>> toy=django.template.loader.get_template('toychild.html')
This renders properly:
>>> toy.render(django.template.Context(dict(adjective='cool',verb='heart')))
u'\n \nDjango is cool\n\n I heart it.\n\n'
But I can’t get the two variables from it:
>>> v=toy.nodelist.get_nodes_by_type(VariableNode)
>>> for k in v: print k.filter_expression.var
...
block.super
verb
You are able to visually inspect a template and observe the presence of any “Variable Node” objects in that template’s nodelist:
>>> from django.template import Template, Context
>>> t = Template("Django is {{ adjective }} and I {{ verb }} it.")
>>> t.nodelist
[<Text Node: 'Django is '>, <Variable Node: adjective>, <Text Node: ' and I '>, <Variable Node: verb>, <Text Node: ' it.'>]
These are of the type VariableNode, which is a class that can be directly imported for use in comparisons. Any Node instance has a get_nodes_by_type() method that can be called against a nodelist, which return all nodes of that type for the template. Example:
>>> from django.template import VariableNode
>>> varnodes = t.nodelist.get_nodes_by_type(VariableNode)
>>> varnodes
[<Variable Node: adjective>, <Variable Node: verb>]
So now you have a list of the variables for the template. This will need to be taken a step further to extract the actual name of each variable without peforming stupid string-slicing tricks on their repr names.
The variable name itself is stored in filter_expression.token for each VariableNode:
>>> varnodes[0].filter_expression.token
u'adjective'
And so a simple list comprehension gets us all of the variable names for the template:
>>> template_vars = [x.filter_expression.token for x in varnodes]
>>> template_vars
[u'adjective', u'verb']
So, not the simplest solution, but if there is a better way I don’t know about it.
Bonus: A function!!
from django.template import VariableNode
def get_template_vars(t):
varnodes = t.nodelist.get_nodes_by_type(VariableNode)
return [x.filter_expression.token for x in varnodes]
Ok, it’s not so complex after all!
Follow-up Edit: Getting variables from parent templates
(This follow-up is using the information from the updated question).
This is where it does actually get complex because the nodelist of the toy template is a single ExtendsNode (in this case).
>>> toy.nodelist
[<ExtendsNode: extends "mysite/toyparent.html">]
I would imagine that in larger templates there could be multiple ExtendsNode objects. Anyhow, if you inspect the ExtendsNode, and extract the parent template from it, you are able to treat the parent the same as my original example:
>>> enode = toy.nodelist[0]
>>> enode.parent_name
u'mysite/toyparent.html'
>>> parent = enode.get_parent(enode.parent_name)
>>> parent
<django.template.Template object at 0x101c43790>
>>> parent.nodelist.get_nodes_by_type(VariableNode)
[<Variable Node: adjective>]
And there is your adjective variable extracted from the parent template. To perform a test against an ExtendsNode you can import the class from django.template.loader_tags:
>>> from django.template.loader_tags import ExtendsNode
>>> ext = toy.nodelist.get_nodes_by_type(ExtendsNode)
>>> ext
[<ExtendsNode: extends "mysite/toyparent.html">]
So, you could do some tests against templates for the presence of an ExtendsNode and walk backwards to the parent template and individually get those variable names. However, this is starting to seem like a can of worms.
For example, if you were to do this:
>>> toy.nodelist.get_nodes_by_type((ExtendsNode, VariableNode))
[<ExtendsNode: extends "mysite/toyparent.html">, <Variable Node: block.super>, <Variable Node: verb>]
Now you’ve got the ExtendsNode and VariableNode objects and it just starts to get confusing. What do we do then? Do we attempt to ignore any block variables returned from such tests? I don’t know!!
In any case, this is the information you wanted, but I don’t think that this is a practical solution. I insist that there is still probably a better way. It might be worth looking into what you are trying to solve and see if there is another approach you can take.
| Posted in General | 1 Comment »
Logical Meltdown
I have been totally unable to accomplish any coding this week. I’ve been rewriting this application at work and while I’ve made some decent progress, I haven’t gotten a damn thing done this week and it’s frustrating as hell!
Of course now that it’s Friday and I’m feeling the urge to start writing more code having to do with dongs. What the fuck is wrong with me? Don’t answer that.
I figured that by explaining some of what I’m struggling with, some magic will happen. I need to blog more and work on my “explaining complex shit in a meaningful way” skills, so if this in any way interests you please read on.
WARNING: TECHNICAL SHIT BELOW!
The application I am rewriting is a MySQL database-driven app that has two interfaces:
- A command-line interface (CLI) utility written in Perl for direct manipulation (addition/removal of records) and querying (searching) of the database. Only engineers have access to this tool.
- A web application written in PHP for searching the database for non-engineers. Mostly support and operations staff use this. This also provides a hacked up “API” that returns a list of comma-separated values (CSV) of matching records.
There are a shit-ton of logical pitfalls in that code that have really thrown me off. Being that we have a Perl CLI tool and a PHP web app, the much of the code is duplicated for each language. The tool was originally written in Perl with the web app piece tacked on as an afterthought. At the time I was really into PHP’s native support for MySQL but was pretty much a novice so ended up making some downright lazy and bad choices.
With PHP and it’s bloated cousin Perl, there are so many ways to do the same thing that there really isn’t one right way to do things. Typically that means there are that many more wrong ways to do them. For example, horrible crap like this where we have PHP making a system() call to execute a Perl script from within index.php:
<table>
<tbody>
<tr valign="top">
<td>
<?
system('perl body.pl column="' . $POST['column'] . '" value="' . $POST['value'] . '" netmask="' . $_POST['method'] . '"');
?>
</td>
</tr>
</tbody>
</table>
Couple that with my prior lack of regimented programming style because I originally learned to code from the back of a Cracker Jack box (aka the internet), and what I am working with is pretty much a nightmare.
“Why is that?”, you ask? Well let me tell you!
First things first, the primary users of the app (aka the stakeholders) have been requesting some new features for a long time and with everything else going on it hasn’t really been a high priority for me. Especially because it was written in Perl, which I have grown to hate passionately since I started using Python.
Next, the systems from which the CLI tool runs are being replaced, which means the app has to be moved to the new systems. We run a Red Hat offshoot distribution of Linux, so we use RPMs to manage what gets installed on machines. What this means is that any Perl module dependencies need to be installed via RPM. With RPM and Perl there are always dependencies of your dependencies, and you can bet that some of those won’t be available in the repository. This opens a massive can of worms of having to download the module source and convert it to RPM. For those of you who are familiar with Perl’s CPAN and RPM, you’re probably thinking “Just use cpan2prm!”. Right, if only it were that simple…
You see, the new hosts are running a 64-bit kernel, which means that the subset of available packages is even smaller because packages have to be specifically built (or rebuilt) for 64-bit installation. Even if it’s the same files inside of the package, it still has to be re-packaged for 64-bit.
So, I was left with the choice of either uprooting the tool from its current location, wrangling package dependencies, rebuilding packages, modifying the Perl code to shoehorn it into the new systems all the while overlooking the implementation of the requested features; or rewriting it using Python & Django.
Naturally, I chose to rewrite it because the thought of having to trudge through antiquated Perl hackery gave me diarrhea and I really don’t want to ever touch Perl or PHP again if I can help it. By rewriting the app in Python, I can implement the new features and bring the code into the present at the same time without having to go through bouts contemplating suicide. Annnddd… That leads us back to the nightmare that is Perl and PHP.
As I mentioned above, with Perl & PHP there are so many ways to do things that most of them are convoluted, confusing, hard-to-read, and just plain fucking wrong. With Python–and Django for that matter–there is usually only one right way to do things, and that way is pretty much always elegant. Plus, one of the most beautiful things about Django is that it wholly and completely abstracts the database interaction from the source code. That means not having to write complex SQL statements manually, and the less SQL I have to write, the better.
In meeting with the stakeholders on rewriting the application, we decided that all of the energy should be invested in replacing the web app with a real API with a RESTful interface. If you don’t know the basic premise of REST (which is a methodology, not a protocol), it’s that URLs should be easy-to-read, self-describing and beautiful. Beautiful URLs? WTF? How can a URL be beautiful? I’m glad you asked!
The old PHP version of the API is very heavy on the query string, requiring GET variables to be passed in the URL itself which is not only cumbersome, but can get confusing and hard to read very quickly:
http://server.whatever.com/api.php?show=all&user=jathan
Compare that to a RESTful version of the same:
http://server.whatever.com/api/show/all/jathan/
This is where Django excels because it encourages pretty URLs by making it easier to use pretty URLs than not to.
So, cutting this post off at the knees, I’ve been pounding on this for the past few weeks, learning a lot about how NOT to do things from my old code and how TO do things using Django as my guide. I made a lot of progress, got the API working, and a basic search interface that returns stuff all using the same code.
But this week, I’ve been stuck. I haven’t made any progress, have been getting really confused, and just generally bashing my skull into the keyboard. I think I’m just fried from trudging through all of Perl’s curly braces, semicolons, ampersands, dollar signs… OMG THE SYNTAX! KILL ME NOW!!
Besides, it’s Fried-day anyways, so I hope that just by spelling this all out here that I’ll be able to dig back into the code after the weekend with a renewed perspective.
p.s. Fuck Perl.
p.p.s. Fuck PHP too, but wrap that rascal. She gets around!
| Posted in Python, Tutorials | No Comments »