HappyPython

Generating SVG Output (from Graphviz) in your Django App

2009-11-28T13:11:00.001-08:00

So accomplishing new coding tasks can be a challenge with interruptions and I've had a lot of interruptions this week but I finally got there. And I'm thankful!

I have an app that is storing data, meaning Django models for the uninitiated. What is in there doesn't matter, but it is something that is conducive to plotting with graphviz. So the starting point is a string that is in the .dot format. I have some code that makes queries to the database and I end up with a string.

So there is a utility function that creates this string...


def make_svg_str:
#blah blah blah snip
dot_string += "}"
p = subprocess.Popen('/usr/bin/dot -Tsvg', shell=True,\
stdin=subprocess.PIPE, stdout=subprocess.PIPE)
(stdout,stderr) = p.communicate(dot_string)
return stdout

So I almost got this right the first time except that I forgot the stdout in Popen() which caused the output to go to stdout (and not be assigned to the string) so I saw the .xml in the dev web server logs.

The graphviz string (dot_string) is being piped to the dot executable and then the function is returning the XML SVG as a string, and is obviously assigned to the stdout variable in the tuple.

Now the tricky part within my views.py.

My first mistake was using the Django CSV docs instead of the PDF docs because the latter is what we need. I also didn't remember that HttpResponse is a file-like object so we can can just write to it once we have the SVG text.


def svg(request):
f = Foo.objects.all()
response = HttpResponse(mimetype='image/svg+xml')
response['Content-Disposition'] = 'filename=somefilename.svg'
response.write(make_svg_str(f))
return response

So this will display your image within your browser (which is what I wanted) instead of downloading file if you the use the "attachment" in the Content-Disposition key.

The name of the game is taking shortcuts that get the job done. I'm using the admin interface to provide a good-enough UI to enter the data and now I'm using Graphviz to visualize that data without having to spend a lot of time writing UIs or nasty JavaScript.

Google Protocol Buffers

2008-12-13T07:53:00.000-08:00

Google Protocol Buffers was mentioned on the Python LinkedIn Community.

Protocol buffers are a flexible, efficient, automated mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages. You can even update your data structure without breaking deployed programs that are compiled against the "old" format.

Here is the Link to the Python API.

Talking 3.0

2008-12-06T16:49:00.000-08:00

I'm embarrassed to say I've hardly touched 2.6 let alone 3.0 but Let's talk about Python 3.0 has some interesting background on the limitations of Python 2.x as well as thoughts on process improvement

This is a startlingly good analogy for the way lots of corporations do things; once a particular process is entrenched (and especially after a couple rounds of employee turnover), there’s nobody left who remembers why the company does things this way. There’s nobody who stops to think about whether this is still a good way to do things, or whether it was even a good idea way back at the beginning. The process continues through nothing more than inertia, and anyone who suggests a change is likely to end up viciously attacked by monkeys.

Of course CentOS5 still only comes with Python 2.4.3, which is what I'm stuck for much of my development unless is compile from scratch.

Back to my Labor Day App

2008-11-21T17:19:00.000-08:00

One of the best rules of programming is to leave your code alone if you are banging your head against a problem. When upgrading to 1.0 Well I finally fixed the issue with my admin interface. It helps if you put admin.py in the right directory!

Within the app instead of the project.

DOH!

Func: Python Systems Management

2008-09-27T06:56:00.000-07:00

Func: Fedora Unified Network Controller looks pretty interesting.

A lot of programs, systems, and tools need some way to communicate. Func provides a two-way authenticated system for generically doing these sort of things. You can build your own applications on top of it, and easily expand func by adding in additional modules, whether you want these to work through the func command line or by means of some other application. If you just want to use the func command line, that's great. If you want to build apps on the func tools, that's great too. If this sounds generically vague, it's only because it really is that expandable.

With some examples


# func target.example.org call hardware info
# func "*.example.org" call yumcmd update
# func "*" call moduleyouwrote functionname 1234 5678 firetruck acmelabs hike!

Converting Subnet to CIDR in Python

2008-09-21T14:10:00.000-07:00

A nice recipe, here is the meat of an even faster version in the comments.


def calcDottedNetmask(mask):
    bits = 0xffffffff ^ (1 << 32 - mask) - 1
    return inet_ntoa(pack('>I', bits))

Which also led me to the Python netaddr project. Very cool!

Rundown on Django 1.0 Upgrade Problems (and Solutions)

2008-09-21T13:50:00.000-07:00

I've been too swamped with my teaching lately to make much progress on a little app I initially wrote in 0.96.2 and have been trying to port to 1.0.

Besides some bonehead typos that wasted a lot of time, here are the summary of issues I've run across:

Creation of the new admin.py
Modifications of urls.py
Conversion between max_length and maxlength in your modles

I still don't have my admin interfaces working even though I've followed the exact steps, so I'm sure it is something else.

The main problems related to the admin interface, but there are a number of resources I've run across including Porting your apps from Django 0.96 to 1.0 and the complete list of backward incompatible changes and a conversion script to create the new admin interface conversion.

And how easy is Django CSV?

2008-09-01T05:14:00.000-07:00

I'm a few hours (and of course I already have a functional app) into converting a spreadsheet "database" we currently store on Sharepoint with a Django app and I was pleased to see how easy it was to do CSV dumps of data. And it worked on the first time. Amazing.

And CherryPy Looks to be the Winner

2008-05-11T06:35:00.000-07:00

So I've been looking for a pure Python lightweight HTTP server for serving up a small django app.

Here are the steps I used (assuming Django is already installed however you installed it) for Ubuntu Hardy LTS:

1) Install CherryPy 3.x (I installed the python-cherrypy3 package)

2) Get DjangoCerise and follow the docs. Documentation and these wrapper scripts were the make or break difference.

3) Increase number of threads in th e SERVER_THREADS file. My crude app worked fine but the django admin interface was a bit sluggish.

While it properly daemonizes (possibly because I screwed up the scripts) shutdown wasn't working for me and (quite obviously, in hindsight) unless you run the startup scripts as root (which I will be) once this app qoes into "production" it won't properly set the user to nobody.

From Aspen to Karrigell and web.py

2008-05-10T19:14:00.000-07:00

Been struggling to get Aspen working with Django. My app works fine, still having issues with "static" apps, which are necessary for the media directory and I ran across two framworks I hadn't seen before.

The first was Karrigell

Karrigell is a flexible Python web framework, with a clear and intuitive syntax. It is independant from any database, ORM or templating engine, and lets the programmer choose between a variety of coding styles

The package includes a powerful built-in web server, so there's no need to download, install and configure a separate one, and a pure-Python database engine, PyDbLite, which is used for the demos

and another was web.py which is so barebones it is not worth quoting the website.

I doubt I'll use either of these, in particular I'm not sure the point of using a web framework unless there is an ORM?

No Model Inheritence in Django?

2008-04-25T06:28:00.000-07:00

Based on this wiki page on Model Inheritance and this hack on extending models (yuck) not looking too good. I would guess this "just works" in Rails, since I've just redesigned my database now would be the time to switch but I really want Django's admin interface.

Not Too Happy with Django Migration

2008-04-17T19:10:00.000-07:00

So after playing around with dmigration and dbmigration I'm probably going back to mucking with SQL (which I'm even less happy) or maybe hacking something together that meets my needs and actually works.

fun with ieeemac

2008-03-02T10:53:00.000-08:00

This week I count have actually used ieeemac when I was writing some scripts to automate packet generation using Click Router.

But of course I was using Ruby so I couldn't take advantage of this.

>>> import ieeemac
>>> import commands
>>> o = commands.getoutput("ifconfig -a")
>>> o
'lo0: flags=8049 mtu 16384\n\tinet 127.0.0.1 netmask 0xff000000 \n\tinet6 ::1 prefixlen 128 \n\tinet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 \ngif0: flags=8010 mtu 1280\nstf0: flags=0<> mtu 1280\nen0: flags=8863 mtu 1500\n\tether 00:11:24:45:4b:1a \n\tmedia: autoselect (none) status: inactive\n\tsupported media: none autoselect 10baseT/UTP 10baseT/UTP 10baseT/UTP 100baseTX 100baseTX 100baseTX \nen1: flags=8863 mtu 1500\n\tinet6 fe80::211:24ff:fea4:9fc7%en1 prefixlen 64 scopeid 0x5 \n\tinet 192.168.1.103 netmask 0xffffff00 broadcast 192.168.1.255\n\tether 00:11:24:a4:9f:c7 \n\tmedia: autoselect status: active\n\tsupported media: autoselect\nfw0: flags=8863 mtu 2030\n\tlladdr 00:11:24:ff:fe:45:4b:1a \n\tmedia: autoselect status: inactive\n\tsupported media: autoselect \nwlt1: flags=41 mtu 1500\ntap0: flags=8842 mtu 1500\n\tether 74:61:70:00:00:00 \n\tclosed\ntun0: flags=8850 mtu 1500\n\tclosed'
>>> ieeemac.find_macs(o)
[, , , ]
>>> ml = ieeemac.find_macs(o)
>>> ml
[, , , ]
>>> ml[0]

>>> dir(ml[0])
['__doc__', '__eq__', '__getattr__', '__init__', '__module__', '__str__', '_formats', 'format', 'formats', 'groups', 'groups_need_fixing', 'to_format']
>>> ml[0].to_format("cisco")
'0011.2445.4b1a'

Easy Win32 Python and Registry

2008-02-24T12:57:00.000-08:00

So I have some experience with APIs for accessing the registry (when I wrote some SCADA Nessus Plugins) but it can't get any easier than this recipe.

NetworkX: This I might actually find time to play with

2008-01-13T19:57:00.000-08:00

Just ran across NetworkX tonight:

NetworkX (NX) is a Python package for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks. Features: Includes standard graph-theoretic and statistical physics functions, Easy exchange of network algorithms between applications, disciplines, and platforms, Includes many classic graphs and synthetic networks, Nodes and edges can be "anything" (e.g. time-series, text, images, XML records)

A Better Python Cmd Library

2008-01-04T19:05:00.000-08:00

The Python cmd module has always been been better than the Ruby imitation (after all like most things about Ruby, it is cheap copy of the original, distinctive feel?) but cmd2 turns this into a real ass-kicking with the following features:


 * Searchable command history (commands: "hi", "li", "run")
 * Load commands from file, save to file, edit commands in file
 * Multi-line commands
 * Case-insensitive commands
 * Special-character shortcut commands (beyond cmd's "@" and "!")
 * Settable environment parameters
 * Parsing commands with flags

Django on Jython?

2008-01-04T18:59:00.000-08:00

I previously whined about the non-progress of Jython (especially relative to JRuby) but this is great news.

The most important thing to know about Django on Jython is that we are almost there, and with clean code. End-to-end functionality is demonstrated by the admin tool running in full CRUD, along with a substantial number of unit tests and syncdb.

Maybe someday I'll have time to read on Python Bytecode

2008-01-02T08:51:00.000-08:00

Maybe someday I'll get a chance to fully digestExploring Python Bytecode.

For the past month or so, I’ve been trying to understand what appears to be a black art mostly because of lacking documentation - Python bytecode generation and peephole optimization. Some notes from the study for the benefit of IRC-mate ‘jstatm’ and anyone else living on similar planes of insanity.

Although bytecode applies to objects other than functions such as tracebacks, dictionaries and strings, I am only interested in optimization and flow analysis of class methods, functions and lambdas. Lets get to the action straight away with an example. Here is a small python program to disassemble and display the bytecode of a function in human readable form.

Decorators, Python Black Magic and the Residue of Past Programming

2008-01-01T07:57:00.000-08:00

While trying to figure out decorators I ran across a really cool presentation called Python Black Magic (or how I learned to stop writing Java in Python). I understand this, since much of the Python I've written over the years, was even worse -- I was writing Perl in Python. For the longest time Python OO conventions seemed strange compared to Java/C# and I refused to use them sticking to horrific Perl-like hash of a hash of a hash data structures. On the one hand if you are sort of programming-ADD-like-me, I definitely see the advantage of developing a set of crisp clean, language agnostic design patters. Who cares if your code isn't completely Pythonic or if you aren't exploiting all the Ruby functional programming fu, your code will be accessible to the largest possible audience.

Web Scripting with Twill (I wish I had this when I used to do WebApp Assessments)

2007-12-28T21:16:00.001-08:00

A few weeks ago I thought I had the need to script (for the life of my I can't remember why I wanted to do this, I should have blogged on it) Firefox. Well I didn't any Python or Ruby tools for taking control of the browser (I remember seeing how to do this with Python and IE a loooong time ago) but tonight I ran across Twill which looks pretty cool and I assume uses the cmd module

mfranz@gutsy61:~$ twill-sh

-= Welcome to twill! =-

current page: *empty page*

>> go http://www.threatmind.net/secwiki
==> at http://www.threatmind.net/secwiki
current page: http://www.threatmind.net/secwiki
>> showforms

Form #1
## ## __Name__________________ __Type___ __ID________ __Value__________________
1 action hidden (None) fullsearch
2 context hidden (None) 180

3 value text searchinput
4 1 titlesearch submit titlesearch Titles
5 2 fullsearch submit fullsearch Text

Form #2
## ## __Name__________________ __Type___ __ID________ __Value__________________
1 action select (None) ['raw'] of ['raw', 'print', 'refresh ...
2 1 None submit (None) Do

Form #3
## ## __Name__________________ __Type___ __ID________ __Value__________________
1 action select (None) ['raw'] of ['raw', 'print', 'refresh ...
2 1 None submit (None) Do

current page: http://www.threatmind.net/secwiki
>> help

Undocumented commands:
======================
add_auth fa info save_html title
add_extra_header find load_cookies setglobal url
agent follow notfind setlocal
back formaction redirect_error show
clear_cookies formclear redirect_output show_cookies
clear_extra_headers formfile reload show_extra_headers
code formvalue reset_browser showforms
config fv reset_error showhistory
debug get_browser reset_output showlinks
echo getinput run sleep
exit getpassword runfile submit
extend_with go save_cookies tidy_ok

Obviously I'm not Even an Intermediate Level Python Programmer

2007-12-27T14:35:00.000-08:00

While a thread on regex performance revealed how much I've forgotten (or never knew) even though I started coding in Python (1.5.x) back in 1999. My confusion didn't have really anything to do with regexes but the two different approaches, one which was more peculiar to my only-coding-in-Ruby-recently brain:

Class Approach

import re
class Searcher(object):
  def __init__(self, rex):
     self.crex = re.compile(rex)
  def __call__(self, txt):
     return self.crex.search(txt)

s = Searcher("dog")
print s("dog").string

After I remembered what the __call__ was used for (which I actually like) and got used to the __'s (which I don't like) and I've never liked the self's in Python method arguments -- this made sense.

Function Returning a Function

import re
def searcher(rex):
  crex = re.compile(rex)
  def _(txt):
     return crex.search(txt)
  return _
s = searcher("dog")
print s("dog").string

At first this didn't make much sense and I got tricked by the underscore (thinking it was some sort of Perl-like special function name or something, it isn't though!). Wny would I call a function returning another function that I would use over and over again. Why would you do that? What is interesting though is if that s("dog")("dog") also produces identical results although I have no idea why (except that the first nested function within a function always executes the second parameter).

Names, Objects, Bindings, and Thinking Like a Pythonista

2007-12-24T19:48:00.000-08:00

On the Python Mailing list there was a discussion about "references" and someone posted a link to How to think like a Pythonista with some nice ASCII art.

Names look like this:

,-----.
| foo |
`-----'

Names live in namespaces, but that's not really important for the
matter at hand as the only namespace in play is the one associated
with the read-eval-print loop of the interpreter. In fact names are
only minor players in the current drama; bindings and objects are the
real stars.
...

I haven't done much Ruby coding this month (cause of the baby so I'm trying to use the break to brush up on my Python. So a nice article for that.

Angry Ruby Meet Happy Python!

2007-12-23T16:20:00.000-08:00

Because I don't seem too angry anymore.