Tag Archives: code

Auto-indent + Paste = Murder

I use Vim pretty much exclusively as my IDE for all programming tasks be it Python, HTML, JavaScript… whatever. One of the Nifty Features™ of vim is auto-indentation which saves a lot of extra tab keystrokes in the longrun. The side effect of this Nifty Feature™ is that when you go to paste code from another source into a file you’re editing, it goes berserk and indents everything you just pasted. If the code you’re pasting is already indented, you end up with nested double indentation, and with languages like Python where whitespace is significant this can be a huge pain in the ass.

So I finally had enough and decided to share this solution along. This is using Vim’s pastetoggle, which allows you to map a keystroke to toggle auto-indentation when you want to paste stuff. Put this into your .vimrc:

To use it, you hit F3 to toggle paste mode, paste what you need to paste, then hit F3 again to turn it back off. And done!

Loki: A Python Infrastructure Protocol Suite

It’s been a while since I last posted, so I’m going back in time to write briefly about an awesome talk I attended at Black Hat 2010 in Las Vegas (last year, duh).

The toolkit is (poorly) named Loki. There are so many packages out there named Loki already, but I get where they were going with it.

It was written by three brilliant German fellows: Daniel Mende, Rene Graf and Enno Rey of ERNW. The primary focus is manipulating and exploiting infrastructure (layer 2/3) protocols, but of course these are all based on the entire protocol stack, so they have laid a nice framework that is way easier to use than other packet manipulation modules out there like Scapy, albeit more specialized. As of right now it has support for BGP, OSPF, ARP, ISIS, RIP, HSRP (v1 and v2), VRRP (v2 and v3), BFD, LDP, and MPLS. And it can spoof all of them!

Loki itself is pure Python, although it does have some system level dependencies. It centers mostly around loki.py, which is a GUI application that is its bread and butter. This app is effectively a router emulator that allows you to establish peerings and execute brute force attacks. For example all you need is a single BGP packet from the wire and you can then perform your brute-force agains the authentication key offline.

In some cases for protocols with no authentication such as VRRP–which is commonly used for providing redundant default gateways for Internet egress–you can push out another peer as a default gateway and take over that role with mind-blowing ease.

If you poke around you’ll see that they’ve encapsulated each protocol into its own module making almost of all of it reusable outside of the GUI. Additionally if you go to the main page of their site and scroll down to Tools, you’ll see a few other packages they’ve released.

For a more technical read, check out a recent blog post by Mike Poor.

Intro to Twisted

I presented a short Intro to Twisted at the SoCal Piggies meetup in Santa Monica last night. It went pretty well and I think I did a good job explaining the basics. I know it took me a long time to even become even remotely familiar with the way Twisted operates, so I felt like this was a good choice for a five-minute presentation. Short, sweet, and to the point!

Here is is for your viewing pleasure:

I borrowed the image on slide five from the Twisted Intro, which is a very excellent introductory tutorial written by Dave Peticolas. If you are curious about Twisted, what it can do, or are even experienced and want a great refresher, it’s definitely worth your time to give it a look!

On Using eval() in Python

Originally posted on StackOverflow.

I have used eval() in the past (and still do from time-to-time) for massaging data during quick and dirty operations. It is part of the toolkit that can be used for getting a job done, but should NEVER be used for anything you plan to use in production such as any command-line tools or scripts, because of all the reasons mentioned in the other answers.

You cannot trust your users–ever–to do the right thing. In most cases they will, but you have to expect them to do all of the things you never thought of and find all of the bugs you never expected. This is precisely where eval() goes from being a tool to a liability.

A perfect example of this would be using Django, when constructing a QuerySet. The parameters passed to a query accepts keyword arguments, that look something like this:

If you’re programmatically assigning arguments, you might think to do something like this:

But there is always a better way that doesn’t use eval(), which is passing a dictionary by reference:

By doing it this way, it’s not only faster performance-wise, but also safer and more Pythonic.

Moral of the story?

Use of eval() is ok for small tasks, tests, and truly temporary things, but bad for permanent usage because there is almost certainly always a better way to do it!

Dynamically Determining the Variables for a Django Template

This is something that came up on StackOverflow, and I took the time to provide a very detailed answer. I haven’t posted in a while, and since I spent time on this answer, here it is verbatim. Enjoy!

Original Question

I’d like to be able to instantiate a template from a file (presumably using the django.template.loader.get_template(filename) ), and then determine the set of variables that should be defined in whatever context it is passed.

(snip)

I tried synack’s answer and found that (with the replacement of filterexpression.token by filterexpression.var, to get the actual name of the variable without tags and so on) it returned the variables that are defined locally in the template, but did not work for variables that are defined in the parent that it extends.

So for example, suppose I have templates in two files:

toyparent.html:

{%block base_results%}
Django is {{adjective}}
{%endblock base_results%}

toychild.html:

{% extends "toyparent.html" %}

{%block base_results%}
{{block.super}}
I {{verb}} it.
{%endblock base_results %}

And I load the child template:

>>> toy=django.template.loader.get_template('toychild.html')

This renders properly:

>>> toy.render(django.template.Context(dict(adjective='cool',verb='heart')))
u'\n    \nDjango is cool\n\n    I heart it.\n\n'

But I can’t get the two variables from it:

>>> v=toy.nodelist.get_nodes_by_type(VariableNode)
>>> for k in v: print k.filter_expression.var
...
block.super
verb

My Answer

You are able to visually inspect a template and observe the presence of any “Variable Node” objects in that template’s nodelist:

>>> from django.template import Template, Context
>>> t = Template("Django is {{ adjective }} and I {{ verb }} it.")
>>> t.nodelist
[<Text Node: 'Django is '>, <Variable Node: adjective>, <Text Node: ' and I '>, <Variable Node: verb>, <Text Node: ' it.'>]

These are of the type VariableNode, which is a class that can be directly imported for use in comparisons. Any Node instance has a getnodesby_type() method that can be called against a nodelist, which return all nodes of that type for the template. Example:

>>> from django.template import VariableNode
>>> varnodes = t.nodelist.get_nodes_by_type(VariableNode)
>>> varnodes
[<Variable Node: adjective>, <Variable Node: verb>]

So now you have a list of the variables for the template. This will need to be taken a step further to extract the actual name of each variable without peforming stupid string-slicing tricks on their repr names.

The variable name itself is stored in filter_expression.token for each VariableNode:

>>> varnodes[0].filter_expression.token
u'adjective'

And so a simple list comprehension gets us all of the variable names for the template:

>>> template_vars = [x.filter_expression.token for x in varnodes]
>>> template_vars
[u'adjective', u'verb']

So, not the simplest solution, but if there is a better way I don’t know about it.

Bonus: A function!!

from django.template import VariableNode
def get_template_vars(t):
   varnodes = t.nodelist.get_nodes_by_type(VariableNode)
   return [x.filter_expression.token for x in varnodes]

Ok, it’s not so complex after all!

Follow-up Edit: Getting variables from parent templates

(This follow-up is using the information from the updated question).

This is where it does actually get complex because the nodelist of the toy template is a single ExtendsNode (in this case).

>>> toy.nodelist
[<ExtendsNode: extends "mysite/toyparent.html">]

I would imagine that in larger templates there could be multiple ExtendsNode objects. Anyhow, if you inspect the ExtendsNode, and extract the parent template from it, you are able to treat the parent the same as my original example:

>>> enode = toy.nodelist[0]
>>> enode.parent_name
u'mysite/toyparent.html'
>>> parent = enode.get_parent(enode.parent_name)
>>> parent
<django.template.Template object at 0x101c43790>
>>> parent.nodelist.get_nodes_by_type(VariableNode)
[<Variable Node: adjective>]

And there is your adjective variable extracted from the parent template. To perform a test against an ExtendsNode you can import the class from django.template.loader_tags:

>>> from django.template.loader_tags import ExtendsNode
>>> ext = toy.nodelist.get_nodes_by_type(ExtendsNode)
>>> ext
[<ExtendsNode: extends "mysite/toyparent.html">]

So, you could do some tests against templates for the presence of an ExtendsNode and walk backwards to the parent template and individually get those variable names. However, this is starting to seem like a can of worms.

For example, if you were to do this:

>>> toy.nodelist.get_nodes_by_type((ExtendsNode, VariableNode))
[<ExtendsNode: extends "mysite/toyparent.html">, <Variable Node: block.super>, <Variable Node: verb>]

Now you’ve got the ExtendsNode and VariableNode objects and it just starts to get confusing. What do we do then? Do we attempt to ignore any block variables returned from such tests? I don’t know!!

In any case, this is the information you wanted, but I don’t think that this is a practical solution. I insist that there is still probably a better way. It might be worth looking into what you are trying to solve and see if there is another approach you can take.

Logical Meltdown

I have been totally unable to accomplish any coding this week.  I’ve been rewriting this application at work and while I’ve made some decent progress, I haven’t gotten a damn thing done this week and it’s frustrating as hell!

Of course now that it’s Friday and I’m feeling the urge to start writing more code having to do with dongs.  What the fuck is wrong with me?  Don’t answer that.

I figured that by explaining some of what I’m struggling with, some magic will happen.  I need to blog more and work on my “explaining complex shit in a meaningful way” skills, so if this in any way interests you please read on.

WARNING: TECHNICAL SHIT BELOW!

The application I am rewriting is a MySQL database-driven app that has two interfaces:

  1. A command-line interface (CLI) utility written in Perl for direct manipulation (addition/removal of records) and querying (searching) of the database. Only engineers have access to this tool.
  2. A web application written in PHP for searching the database for non-engineers.  Mostly support and operations staff use this. This also provides a hacked up “API” that returns a list of comma-separated values (CSV) of matching records.
It’s something I wrote years ago using PHP & MySQL that is way more complex than I remember it and double-dog dare you more complex than it needed to be. It’s strange going back in time with my code, especially PHP (shame), and realizing how competent I was even then, but how unnecessarily convoluted I made some things.

There are a shit-ton of logical pitfalls in that code that have really thrown me off.  Being that we have a Perl CLI tool and a PHP web app, the much of the code is duplicated for each language. The tool was originally written in Perl with the web app piece tacked on as an afterthought.  At the time I was really into PHP’s native support for MySQL but was pretty much a novice so ended up making some downright lazy and bad choices.

With PHP and it’s bloated cousin Perl, there are so many ways to do the same thing that there really isn’t one right way to do things.  Typically that means there are that many more wrong ways to do them.  For example, horrible crap like this where we have PHP making a system() call to execute a Perl script from within index.php:

Couple that with my prior lack of regimented programming style because I originally learned to code from the back of a Cracker Jack box (aka the internet), and what I am working with is pretty much a nightmare.

“Why is that?”, you ask?  Well let me tell you!

First things first, the primary users of the app (aka the stakeholders) have been requesting some new features for a long time and with everything else going on it hasn’t really been a high priority for me.  Especially because it was written in Perl, which I have grown to hate passionately since I started using Python.

Next, the systems from which the CLI tool runs are being replaced, which means the app has to be moved to the new systems. We run a Red Hat offshoot distribution of Linux, so we use RPMs to manage what gets installed on machines.  What this means is that any Perl module dependencies need to be installed via RPM. With RPM and Perl there are always dependencies of your dependencies, and you can bet that some of those won’t be available in the repository. This opens a massive can of worms of having to download the module source and convert it to RPM.  For those of you who are familiar with Perl’s CPAN and RPM, you’re probably thinking “Just use cpan2prm!”. Right, if only it were that simple…

You see, the new hosts are running a 64-bit kernel, which means that the subset of available packages is even smaller because packages have to be specifically built (or rebuilt) for 64-bit installation. Even if it’s the same files inside of the package, it still has to be re-packaged for 64-bit.

So, I was left with the choice of either uprooting the tool from its current location, wrangling package dependencies, rebuilding packages, modifying the Perl code to shoehorn it into the new systems all the while overlooking the implementation of the requested features; or rewriting it using Python & Django.

Naturally, I chose to rewrite it because the thought of having to trudge through antiquated Perl hackery gave me diarrhea and I really don’t want to ever touch Perl or PHP again if I can help it. By rewriting the app in Python, I can implement the new features and bring the code into the present at the same time without having to go through bouts contemplating suicide. Annnddd… That leads us back to the nightmare that is Perl and PHP.

As I mentioned above, with Perl & PHP there are so many ways to do things that most of them are convoluted, confusing, hard-to-read, and just plain fucking wrong. With Python–and Django for that matter–there is usually only one right way to do things, and that way is pretty much always elegant.  Plus, one of the most beautiful things about Django is that it wholly and completely abstracts the database interaction from the source code.  That means not having to write complex SQL statements manually, and the less SQL I have to write, the better.

In meeting with the stakeholders on rewriting the application, we decided that all of the energy should be invested in replacing the web app with a real API with a RESTful interface. If you don’t know the basic premise of REST (which is a methodology, not a protocol), it’s that URLs should be easy-to-read, self-describing and beautiful. Beautiful URLs? WTF? How can a URL be beautiful?  I’m glad you asked!

The old PHP version of the API is very heavy on the query string, requiring GET variables to be passed in the URL itself which is not only cumbersome, but can get confusing and hard to read very quickly:

http://server.whatever.com/api.php?show=all&user=jathan

Compare that to a RESTful version of the same:

http://server.whatever.com/api/show/all/jathan/

This is where Django excels because it encourages pretty URLs by making it easier to use pretty URLs than not to.

So, cutting this post off at the knees, I’ve been pounding on this for the past few weeks, learning a lot about how NOT to do things from my old code and how TO do things using Django as my guide.  I made a lot of progress, got the API working, and a basic search interface that returns stuff all using the same code.

But this week, I’ve been stuck.  I haven’t made any progress, have been getting really confused, and just generally bashing my skull into the keyboard.  I think I’m just fried from trudging through all of Perl’s curly braces, semicolons, ampersands, dollar signs… OMG THE SYNTAX!  KILL ME NOW!!

Besides, it’s Fried-day anyways, so I hope that just by spelling this all out here that I’ll be able to dig back into the code after the weekend with a renewed perspective.

p.s. Fuck Perl.

p.p.s. Fuck PHP too, but wrap that rascal. She gets around!

More Dongs on a Friday. Surprise!!

Due to popular demand, mkdong has been modularized!!  I present to you dong.py version 0.0.1.

Enjoy:

There’s really no excuse. I should be ashamed of myself but I’m not.

Blue Dongs for a Friday Afternoon

Today I wrote an awesome program called mkdong that will make a dong of your desired length and print it to your terminal, like this:

That last one is impressive, isn’t it? Hmm… Yeah, it’s Friday. What do you want from me? I still got work done! Cool thing is if the dong is too big, well then it throws an error:

“What is the point of this?”, you might ask yourself. That’s a good question. I’ve been so busy with other shit lately that I’ve barely had time to code. I suppose I was itching to write something, anything… Dongs!!

It all started harmlessly enough with a silly AIM conversation with my coding buddy at work. We were talking about a bug, and well, read on and you’ll see. It regressed quickly.

jughead: well there is the problem, now I just gotta figure out wtf jathanism: what did you do! jughead: nothing, it’s something in the rt stuff jathanism: hmm, maybe with a certain ticket jathanism: maybe one that is blank or something jathanism: or missing xml data jathanism: or might be a bug jughead: ok from now on jughead: bugs are not bugs jughead: they are dongs jathanism: ok jughead: “I found a dong in this code” jathanism: or might be a dong jughead: #1 deterrent of all linux exploits jughead: change root account to “poop”, chown -R poop.poop / jughead: no uid 0 jughead: no user root jathanism: what? jathanism: you rename uid 0 to poop ? jughead: poop will get it’s own uid jughead: and gid jathanism: oh jathanism: would that work? jathanism: there aren’t any hard-coded things to uid 0? jughead: no idea, lets try it jathanism: yeah let’s try it on marduk! jughead: hahahahfdsaf jughead: dsfa jughead: hmm…. /dev/ would have to be rebuilt jughead: MKNOD jughead: hate that shit jughead: MAKEDEV jughead: FUCK YOU LINUX jathanism: mkdong jathanism: aww yeah jathanism: i just made mkdong jathanism: % ./mkdong 5 8====0 jathanism: % ./mkdong 15 8==============0 jathanism: % ./mkdong 25 8========================0 jathanism: % ./mkdong 41 warning: a 41″ dong is too big! cannot be longer than 40″! jughead: you should modify that jughead: ()/()\\\\\\\\\\\\\\D jathanism: hahahlk jughead: looks much better jathanism: ok! jathanism: % ./mkdong 40 ()/()\\\\\\\\\\\\\\\\\\\\D jathanism: aww yeah

So I took the stupidity and ran with it and mkdong was born!

The initial dongs were a little primitive and sickly looking. So I took his suggestion and improved their visual style. Here is how it turned out:

We laughed. We joked. We Tweeted. And then it regressed even further:

jughead: dude everyone loves mkdong jathanism: aww yeah jathanism: it needs easter eggs jughead: DUDE jughead: MAKE IT PRINT IN BLUE jathanism: ok! jughead: how exactly does one “suck a fuck” jathanism: the ascii coloring fucks up the length jughead: you can just put the blue at the beginning jughead: and at the end jughead: doesn’t have to be each char jathanism: it’s not jughead: is there a dong in the code? jathanism: /tmp/mkdong 5 jathanism: will just use forward slashes instad jathanism: released jughead: man mkdong is the best ever

A feature request! I had to make it print in blue! But to do that I had to replace all of the “\” that make up the dong itself, with “/” so as to not have the ANSI escape codes eat up the extra backslashes. (Backslashes are interpreted characters, duh.) I also had to replace the print statement with a system call to echo -e so that the colorization would be interpreted. This is high tech shit, man!!

And then I released it to the public. So there you have it. Here is the final release of mkdong 2.0 for your pleasure:

Use it well. And remember they aren’t bugs, they’re dongs! Squish? Gross.

Python List Comprehension for Dummies

So I code a lot. I code in Python a lot. You might say I love Python. I might say you’re right.

One of the most powerful things about Python is its ability to iterate over ANYTHING as if it were a list. Lists, tuples, dictionaries, and even strings can all be iterated quickly and elegantly. Python also introduces a concept known as list comprehension which allows you to do rather complex filtering of list contents within a single statement.

To illustrate how awesome and powerful list comprehension is, let’s start with a basic example that is NOT using it:

So, let’s assume that we want to identify all even numbers inside of my_list, and put them into a new list called evens the old-fashoned way:

Why the old-fashioned way sucks First things first, the empty list called evens had to be declared ahead of time. This is because when we looped thru the list called my_list using the for statement, when the if test is performed on each item we have to reference evens by name to append() the even numbers to it.

Why list comprehension rocks With list comprehension, the logic that isolates the even numbers and the declaration of the list that will capture this output are compressed into a single statement:

The logic is encapsulated in [square brackets] indicating that the output will be a list. The list comprehension itself is the logic between the brackets that determines what will be in the list that it spits out.

So list comprehensions at their most basic level allow for compression of code and streamlining of logical statements. Advanced usage of list comprehension can get pretty silly, but then so can nested loop statements.  It supports nesting as many statements as you can throw at it so longs as they are syntactically correct.

If you find yourself coding shit like this:

Then maybe list comprehension is for you:

No offense to anyone named Joe, Jim, or Jon.