Google Protocol Buffers

Google Protocol Buffers was mentioned on the Python LinkedIn Community.

Protocol buffers are a flexible, efficient, automated mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages. You can even update your data structure without breaking deployed programs that are compiled against the "old" format.

Here is the Link to the Python API.

Talking 3.0

I'm embarrassed to say I've hardly touched 2.6 let alone 3.0 but Let's talk about Python 3.0 has some interesting background on the limitations of Python 2.x as well as thoughts on process improvement

This is a startlingly good analogy for the way lots of corporations do things; once a particular process is entrenched (and especially after a couple rounds of employee turnover), there’s nobody left who remembers why the company does things this way. There’s nobody who stops to think about whether this is still a good way to do things, or whether it was even a good idea way back at the beginning. The process continues through nothing more than inertia, and anyone who suggests a change is likely to end up viciously attacked by monkeys.

Of course CentOS5 still only comes with Python 2.4.3, which is what I'm stuck for much of my development unless is compile from scratch.

Back to my Labor Day App

One of the best rules of programming is to leave your code alone if you are banging your head against a problem. When upgrading to 1.0 Well I finally fixed the issue with my admin interface. It helps if you put admin.py in the right directory!

Within the app instead of the project.

DOH!

Func: Python Systems Management

Func: Fedora Unified Network Controller looks pretty interesting.

A lot of programs, systems, and tools need some way to communicate. Func provides a two-way authenticated system for generically doing these sort of things. You can build your own applications on top of it, and easily expand func by adding in additional modules, whether you want these to work through the func command line or by means of some other application. If you just want to use the func command line, that's great. If you want to build apps on the func tools, that's great too. If this sounds generically vague, it's only because it really is that expandable.

With some examples

# func target.example.org call hardware info
# func "*.example.org" call yumcmd update
# func "*" call moduleyouwrote functionname 1234 5678 firetruck acmelabs hike! 

Converting Subnet to CIDR in Python

A nice recipe, here is the meat of an even faster version in the comments.

def calcDottedNetmask(mask):
    bits = 0xffffffff ^ (1 << 32 - mask) - 1
    return inet_ntoa(pack('>I', bits))

Which also led me to the Python netaddr project. Very cool!

Rundown on Django 1.0 Upgrade Problems (and Solutions)

I've been too swamped with my teaching lately to make much progress on a little app I initially wrote in 0.96.2 and have been trying to port to 1.0.

Besides some bonehead typos that wasted a lot of time, here are the summary of issues I've run across:

1. Creation of the new admin.py
2. Modifications of urls.py
3. Conversion between max_length and maxlength in your modles

I still don't have my admin interfaces working even though I've followed the exact steps, so I'm sure it is something else.

The main problems related to the admin interface, but there are a number of resources I've run across including Porting your apps from Django 0.96 to 1.0 and the complete list of backward incompatible changes and a conversion script to create the new admin interface conversion.

And how easy is Django CSV?

I'm a few hours (and of course I already have a functional app) into converting a spreadsheet "database" we currently store on Sharepoint with a Django app and I was pleased to see how easy it was to do CSV dumps of data. And it worked on the first time. Amazing.

And CherryPy Looks to be the Winner

So I've been looking for a pure Python lightweight HTTP server for serving up a small django app.

Here are the steps I used (assuming Django is already installed however you installed it) for Ubuntu Hardy LTS:

1) Install CherryPy 3.x (I installed the python-cherrypy3 package)

2) Get DjangoCerise and follow the docs. Documentation and these wrapper scripts were the make or break difference.

3) Increase number of threads in th e SERVER_THREADS file. My crude app worked fine but the django admin interface was a bit sluggish.

While it properly daemonizes (possibly because I screwed up the scripts) shutdown wasn't working for me and (quite obviously, in hindsight) unless you run the startup scripts as root (which I will be) once this app qoes into "production" it won't properly set the user to nobody.

From Aspen to Karrigell and web.py

Been struggling to get Aspen working with Django. My app works fine, still having issues with "static" apps, which are necessary for the media directory and I ran across two framworks I hadn't seen before.

The first was Karrigell

Karrigell is a flexible Python web framework, with a clear and intuitive syntax. It is independant from any database, ORM or templating engine, and lets the programmer choose between a variety of coding styles

The package includes a powerful built-in web server, so there's no need to download, install and configure a separate one, and a pure-Python database engine, PyDbLite, which is used for the demos

and another was web.py which is so barebones it is not worth quoting the website.

I doubt I'll use either of these, in particular I'm not sure the point of using a web framework unless there is an ORM?

No Model Inheritence in Django?

Based on this wiki page on Model Inheritance and this hack on extending models (yuck) not looking too good. I would guess this "just works" in Rails, since I've just redesigned my database now would be the time to switch but I really want Django's admin interface.

Not Too Happy with Django Migration

So after playing around with dmigration and dbmigration I'm probably going back to mucking with SQL (which I'm even less happy) or maybe hacking something together that meets my needs and actually works.

fun with ieeemac

This week I count have actually used ieeemac when I was writing some scripts to automate packet generation using Click Router.

But of course I was using Ruby so I couldn't take advantage of this.

>>> import ieeemac
>>> import commands
>>> o = commands.getoutput("ifconfig -a")
>>> o
'lo0: flags=8049 mtu 16384\n\tinet 127.0.0.1 netmask 0xff000000 \n\tinet6 ::1 prefixlen 128 \n\tinet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 \ngif0: flags=8010 mtu 1280\nstf0: flags=0<> mtu 1280\nen0: flags=8863 mtu 1500\n\tether 00:11:24:45:4b:1a \n\tmedia: autoselect (none) status: inactive\n\tsupported media: none autoselect 10baseT/UTP 10baseT/UTP 10baseT/UTP 100baseTX 100baseTX 100baseTX \nen1: flags=8863 mtu 1500\n\tinet6 fe80::211:24ff:fea4:9fc7%en1 prefixlen 64 scopeid 0x5 \n\tinet 192.168.1.103 netmask 0xffffff00 broadcast 192.168.1.255\n\tether 00:11:24:a4:9f:c7 \n\tmedia: autoselect status: active\n\tsupported media: autoselect\nfw0: flags=8863 mtu 2030\n\tlladdr 00:11:24:ff:fe:45:4b:1a \n\tmedia: autoselect status: inactive\n\tsupported media: autoselect \nwlt1: flags=41 mtu 1500\ntap0: flags=8842 mtu 1500\n\tether 74:61:70:00:00:00 \n\tclosed\ntun0: flags=8850 mtu 1500\n\tclosed'
>>> ieeemac.find_macs(o)
[, , , ]
>>> ml = ieeemac.find_macs(o)
>>> ml
[, , , ]
>>> ml[0]

>>> dir(ml[0])
['__doc__', '__eq__', '__getattr__', '__init__', '__module__', '__str__', '_formats', 'format', 'formats', 'groups', 'groups_need_fixing', 'to_format']
>>> ml[0].to_format("cisco")
'0011.2445.4b1a'

Easy Win32 Python and Registry

So I have some experience with APIs for accessing the registry (when I wrote some SCADA Nessus Plugins) but it can't get any easier than this recipe.

NetworkX: This I might actually find time to play with

Just ran across NetworkX tonight:

NetworkX (NX) is a Python package for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks. Features: Includes standard graph-theoretic and statistical physics functions, Easy exchange of network algorithms between applications, disciplines, and platforms, Includes many classic graphs and synthetic networks, Nodes and edges can be "anything" (e.g. time-series, text, images, XML records)

A Better Python Cmd Library

The Python cmd module has always been been better than the Ruby imitation (after all like most things about Ruby, it is cheap copy of the original, distinctive feel?) but cmd2 turns this into a real ass-kicking with the following features:

 * Searchable command history (commands: "hi", "li", "run")
 * Load commands from file, save to file, edit commands in file
 * Multi-line commands
 * Case-insensitive commands
 * Special-character shortcut commands (beyond cmd's "@" and "!")
 * Settable environment parameters
 * Parsing commands with flags

Django on Jython?

I previously whined about the non-progress of Jython (especially relative to JRuby) but this is great news.

The most important thing to know about Django on Jython is that we are almost there, and with clean code. End-to-end functionality is demonstrated by the admin tool running in full CRUD, along with a substantial number of unit tests and syncdb.

Maybe someday I'll have time to read on Python Bytecode

Maybe someday I'll get a chance to fully digestExploring Python Bytecode.

For the past month or so, I’ve been trying to understand what appears to be a black art mostly because of lacking documentation - Python bytecode generation and peephole optimization. Some notes from the study for the benefit of IRC-mate ‘jstatm’ and anyone else living on similar planes of insanity.

Although bytecode applies to objects other than functions such as tracebacks, dictionaries and strings, I am only interested in optimization and flow analysis of class methods, functions and lambdas. Lets get to the action straight away with an example. Here is a small python program to disassemble and display the bytecode of a function in human readable form.

Decorators, Python Black Magic and the Residue of Past Programming

While trying to figure out decorators I ran across a really cool presentation called Python Black Magic (or how I learned to stop writing Java in Python). I understand this, since much of the Python I've written over the years, was even worse -- I was writing Perl in Python. For the longest time Python OO conventions seemed strange compared to Java/C# and I refused to use them sticking to horrific Perl-like hash of a hash of a hash data structures. On the one hand if you are sort of programming-ADD-like-me, I definitely see the advantage of developing a set of crisp clean, language agnostic design patters. Who cares if your code isn't completely Pythonic or if you aren't exploiting all the Ruby functional programming fu, your code will be accessible to the largest possible audience.