Tag Archives: hacking

Hacking on OpenSim Infrastructure: Mantis Improvements For The Win!

Among other roles in the OpenSim project, I’m reluctant admin for most of the opensimulator.org infrastructure. Infrastructure being defined as: scm repo (subversion and mercurial), bug tracker (mantis), and wiki (media wiki).

Recently I decided to do some hacking on our mantis tracker to make it work better for the project. There was a very real reason to do this, our mantis has over 350 open and unassigned issues, and was starting to get massively ignored by most developers as it was far too overwhelming to route out real issues in mantis vs. stale issues vs. user errors in such a large sea.

The first change in this area was creating an osmantis bot. This is an IRC bot that spits out a message in our #opensim-dev channel on every mantis change. This brings the same level of visibility to our bugs as to our svn commits (which get the same treatment). This involved a rather brutal amount of hackering in mantis, and a perl IRC bot that runs on opensimulator.org. If you’ve been on IRC at all in the last couple of weeks you’ve seen it.

This morning I introduced 3 new states related to patches (patch included, patch feedback, patch ready). The agreed policy of OpenSim is that patches come in via mantis. However, as nothing really makes patches stand out more than regular issues, we were loosing a lot of patches in mantis. I believe our oldest patch in mantis which clearly hadn’t been looked at by anyone was over 3 months old. Not a great state to be in. With the new set of states mantis issues with patches can be set to patch included as part of triage. This will make them pop up to the top of everyone’s attention.

Another issue we’ve had is that users can’t close their own issues. I just fixed that (I think), so if you have reported issues via OpenSim mantis I’d ask to go check to make sure they still look valid. If they aren’t, please close them, to help us clean up the current state of things.

I’m hoping these changes make it easier for us to manage issues coming into the project, and make using mantis more effective for everyone.

Publishing my first gem: thermostat 0.0.2

When not working on OpenSim, I’ve been doing a bunch of random side projects in Ruby. One of the more recent ones has been a more advanced control and monitoring tool for my thermostat.

Along the way I created a module for the thermostat control. Last night I finally figured out how to package this work up as a gem file, which makes this easily installable to the world. Right now, the code is really raw, and I haven’t put in any documentation yet. But, for those interested, it’s available.

> gem search -r thermostat

*** REMOTE GEMS ***
Bulk updating Gem source index for: http://gems.rubyforge.org

thermostat (0.0.2)
    Ruby modules to control Prolipix IP Thermostats

Enjoy.

Graphing with Gruff

The house monitoring project has made a little bit of progress, as I’ve now got data being collected into a rails app using backgroundrb, and can get that data back out into very pretty graphs with gruff.  (I also looked a little bit a sparklines, but that’s specifically for graphs without labels.)
 

We’re running above target temp as it’s the weekend, so the wood stove is on.  As is the furnace fan to spread the heat through the house.  I’m overloading the values for heat on an fan on to be either the bottom of the graph or a specific small value.  I need to sort out a better way to put that into the graph, which may require some hacking on gruff itself.

Early working thermostat code

I definitely don’t have this anywhere near I like it, but I did manage to just dump out a bunch of info from my thermostat and turn off the fan with this script:

#!/usr/bin/ruby

require 'thermostat'
require 'pdp_constants'
include Proliphix

t = Thermostat.new("192.168.1.30","admin","XXXXXXXX")
t.set_senors(ThermHvacMode, ThermHvacState, ThermFanState, ThermFanMode, ThermAverageTemp, ThermHeat1Usage)
t.fetch_data

# dump out what we have
puts t

# turn off the fan
t.set_data(ThermFanMode, 1)

I need some nicer symbolic constants for state setting, and pull together a rails site just to keep track of thermostat data over the course of the day.  All this code is going on my newly registered rubyforge project.

The Earth in OpenSim

Xplanet is this great program on Linux that does projections of the earth (including all kinds of possible overlays to include things like real time clouds, earthquake activity, and major storms.)  For years I’ve used it as the background image on my Linux desktop.  On the way home last night, while chatting with my friend Trey, it occured to me that through creative use of OpenSim, we could easily do this this in world with dramatic effect.

Below is a couple of screen shots of my 10m globe.  I had to modify the code paths for osSetDynamicTextureURL to get 1024×1024 textures into the pipe, as 512×512 doesn’t actually look good wrapped on a 10m globe.  There is still a bit of work to make this really good (like completing osSetDynamicTextureURL so we don’t need another timing loop in LSL to do refresh), but the initial results are quite nice. 🙂

The circles with numbers are earthquakes in the last 24 hours plus magnitudes (this information comes from the TotalMarker project).

Here you can see the storm track of the named storm Ivan (also from TotalMarker).

More fun with dbus

Since my dbus post last week, I’ve been playing around more with dbus whenever I get a few minutes. The modern Linux desktop is pretty good, but with minor tweaks, you can make things even better. (all this code is now up in a mercurial repository called dbus-hackery).

Automating Inactivity

Pidgin makes sounds on every message to me, xchat makes sounds on certain key words. Without these cues, I’d never remember to go check these applications. Because I have a tendency to leave my laptop on overnight, I found that I’d often have xchat ringing away at midnight when someone was looking for me. If I forgot to mute my machine before that, it would often wake me up.

One of the programs sending signals on dbus is gnome-screensaver.

def connect_screensaver(session_bus)
    ss_dbus = session_bus.service("org.gnome.ScreenSaver")
    ss = ss_dbus.object("/org/gnome/ScreenSaver")
    ss.introspect
    if ss.has_iface? "org.gnome.ScreenSaver"
        ss.default_iface = "org.gnome.ScreenSaver"
        puts "Connected to screensaver"
    end
    return ss
end

def mute()
    IO.popen("aumix -vq") {|r|
        r.read.scan(/(d+)/) {|m|
            @@vol = m
            puts "saved volume: #{@@vol}"
            break
        }
    }
    puts "muting"
    system("aumix -v 0")
end

def unmute()
    puts "unmuting"
    system("aumix -v #{@@vol}")
end

ss = connect_screensaver(session_bus)

ss.on_signal("ActiveChanged") {|s|
    if s
        mute
    else
        unmute
    end
}

The connect will look exactly as expected from the previous look at dbus. The ActiveChanged signal outputs a single parameter, a boolean, which is true when the screensave goes active, false when the screensaver is deactivated.

Volume control on the command line is most easily done with aumix (though if you are on Ubuntu Gutsy you’ll have issues until you rebuild aumix yourself. Hopefully they’ll fix that bug soon.) A little regex fun captures the current levels to a package variable, and restores them back on unmute.

Now I’ve got global mute when the screensaver fires, restored when I return.

Better Away with Pidgin

My screensaver being locked is a pretty clear indication that I’m away, though it being unlocked isn’t a clear indication that I’m back. Especially on weekends, I pop back for a quick check of something, then the computer is put away again.

def set_away(pidgin)
    puts "trying to set away"
    name = "screensaver"
    status = pidgin.PurpleSavedstatusFind(name)[0]
    if not status > 0
        status = pidgin.PurpleSavedstatusNew(name, 5)[0]
    end
    puts "Status #{status}"

    pidgin.PurpleSavedstatusSetMessage(status, "screen saver auto away")
    pidgin.PurpleSavedstatusActivate(status)
end

ss.on_signal("ActiveChanged") {|s|
    if s
        mute
        set_away(pidgin)
    else
        unmute
    end
}

In order to set a status with a message, it has to be a saved status. To prevent growing that to infinity, I first look to see if it is defined, creating a new saved status if not. 5 is a magic number here meaning STATUS_AWAY (reference the pidgin status.h for more info). Then we set the message on that status, and activate it. A single line change on our screen saver signal adds this into play.

Keep on Hacking

One of the things I’m hoping to impress in these posts on dbus is that with a highly functional language like ruby, linking applications on a modern gnome desktop can be done even by mere mortals. Linking sound to your screen saver is something that would have required a reasonable chunk of c code. Now you can do it in 20 lines of ruby, thanks to dbus.

I’ve thought about creating some sort of extended control panel to enable the features I’ve hacked together, but the reality is the code is so small, and so simple, it seems like overkill. With code this easy, you should just jump in and hack it to your own needs.

As I keep playing with dbus, I’ll post more bits here. Twitter integration is still on my list of things to do, and maybe something I’ll even manage to get to this week.

Pidgin, Network Manager, Dbus, Ruby, oh my!

A few weeks ago I was chatting with Dan about my one great annoyance with Pidgin: it takes up to 15 minutes to realize that it’s network connection isn’t valid any more and to automatically reconnect. With NetworkManager and dbus, this should be a reasonably simple feat. Dan got curious about this and wrote himself a python program that sets pidgin status in this way.

D-What?

A bit of background. One of the Freedesktop.org standards is d-bus:

D-Bus is a message bus system, a simple way for applications to talk to one another. In addition to interprocess communication, D-Bus helps coordinate process lifecycle; it makes it simple and reliable to code a “single instance” application or daemon, and to launch applications and daemons on demand when their services are needed.

The basic idea is that d-bus allows applications to publish interfaces that other applications on the desktop can interface with. Right now, only a few applications really support d-bus in a significant way. Fortunately, one of those is pidgin, which has an extremely rich d-bus interface.

Ruby & DBUS

Here’s a place where I know I’ll draw Dan‘s ire. I’ve been on a Ruby kick recently, so my first reaction to Dan’s post was “great, I’ll have to figure out how to do this in ruby now.” It sounds like typical language bias, but I’ll try to justify it at least a little.

Right now I’m writing code on a weekly basis in 3 languages: C# (OpenSim), Java (Grad School Project), and Ruby (side web projects done in Rails). 3 languages is a lot to remain fluent in, and causes some interesting syntax errors when jumping back and forth between them. One less context switch seemed like a good idea. At some point I’ll bother writing up why ruby has seduced me, but that is for another day.

While finding python-dbus bindings are extremely straight forward, the ruby-dbus front is a little more of a wandering path. After finding a couple of abandoned efforts, I finally came to the active ruby-dbus project. Make sure to grab the latest and greatest code from there before proceeding.

Connecting to DBUS

Before I get into actually using dbus, I need to set up connections to it. There are 2 different buses, the system bus (which is shared for all users), and the session bus, which is unique per login session. Pidgin uses the session bus: it’s a user application. Network Manager uses the system bus: it’s events are system wide and affect all users.

#!/usr/bin/ruby

require "dbus"

bus = DBus::SystemBus.instance
session_bus = DBus::SessionBus.instance

# Get the Pidgin Service
pidgin_dbus = session_bus.service("im.pidgin.purple.PurpleService")

# Get the object from this service
pidgin = pidgin_dbus.object("/im/pidgin/purple/PurpleObject")

# Introspect it
pidgin.introspect
if pidgin.has_iface? "im.pidgin.purple.PurpleInterface"
    pidgin.default_iface = "im.pidgin.purple.PurpleInterface"
    puts "We have Pidgin interface"
end

n_dbus = bus.service("org.freedesktop.NetworkManager")
netman = n_dbus.object("/org/freedesktop/NetworkManager")
# Establish a proxy interface object for NetworkManager as it doesn't support introspection
poi = DBus::ProxyObjectInterface.new(netman, "org.freedesktop.NetworkManager")

Basically the pattern is clear. Get a service handle, get an object definition from that service handle, then get an interface from that object. There are 2 flavors for this, one where we’ve got introspection information, and one where we’ve got to go blind because introspection isn’t supported.

DBUS Introspection

DBUS interfaces come in 2 flavors, those that support introspection, and those that don’t. If an interface supports introspection you can get an interface definition off the dbus wire itself, otherwise you need to know the interface a priori. Pidgin supports introspection, Network Manager does not. The major short coming of the ars technica article on Pidgin and DBUS was the lack of information on using introspection to show all the other pidgin interface functions (of which there are > 600). Python dbus introspection throws an exception on my Ubuntu 7.10 environment, so it wasn’t helpful here. However, ruby, as usual, came to the rescue.

The ruby-dbus code contains an example application called gd-bus, which performs introspection on all dbus interfaces it can find, and prints them out nicely. The cheat sheet goes something like so:

M PurpleAccountsFind(in name:s, in protocol:s, out RESULT:i)

  • M – DBUS Method, aka Function (might also be S meaning it’s a Signal you can register to listen to)
  • PurpleAccountsFind – the method name
  • in / out – whether this is an input or output parameter
  • NAME:type – the name and type of the parameter. Types can be s – string, i – integer, u – unsigned int, ai – array of integers

Note: the ruby interface seems to always wrap arrays around the output parameters. I have no idea why, but it’s consistent, so a few extra “[0]”s get you a long ways. If you know why, please comment.

The translation of this interface specification into ruby gives you something like this:

def recycle_pidgin(pidgin)
    accounts = pidgin.PurpleAccountsGetAll
    for account in accounts[0]
        if pidgin.PurpleAccountIsConnected(account)[0] > 0
            pidgin.PurpleAccountDisconnect(account)
        end
        pidgin.PurpleAccountConnect(account)
    end
end

This cycles through all pidgin accounts, disconnects all the connected ones, and then attempts to connect all accounts. This is effectively what I end up doing by hand every time I switch networks with my laptop.

Bringing it all together

I now had connections to the 2 buses, and code to cycle the pidgin accounts. Last bit is actually watching for the Network Manager signal that I’ve got a new active network device.

poi.on_signal(bus, "DeviceNowActive") {
    recycle_pidgin(pidgin)
}

main = DBus::Main.new
main << bus
main.run

Network manager doesn’t support introspection. However, it does have pretty decent docs to figure out what the interface is. I’m still sad it doesn’t show up nicely in gd-bus though.

The proxy interface object sets up a signal using 2 parameters and a code block. Every time there is a DeviceNowActive signal on the system bus, I recycle pidgin. Pretty straight forward.

The last little bit is making this thing go into a loop. Ruby dbus contains it’s own main loop for just this task. I created a new main loop, tell it to watch the system bus, and then start it. And, we’re done.

The future’s so bright…

DBUS has been on my list of “I need to go figure this out” for a while. A morning of reading docs, hacking a bit, and crashing network manager a few times, and voila, you’ve got this blog post.

I’ve got lots of ideas floating around in my head now for other things that I can do with dbus to make my applications work better for me. As I bang a few of those out into code, expect to see more here about it. Pidgin is an especially target rich environment given how rich and interface they expose (nice job guys!).

Ruby gymnastics

@strips = @user.comics.collect {|c|    
    c.strips.find(:all, :conditions => [“date > ?“, 7.days.ago])}.
    flatten.   
    delete_if {|c| c == nil}.   
    sort_by {|c| c.date}

The previous shows ruby in much of it’s glory, and is code that I needed this weekend for one of my side hacking projects.  I’ve colored it to match what xemacs shows me, just to make it a little more clear.

First off, it shows off the power of mixins. 7.days.ago does exactly what you would expect, providing you with a date object.

Second, it shows off the power of collect (aka map in many other languages).  Collect lets you iterate through a list you have, and return a new collection based on an arbitrary transform.  In this case returning a list of strip objects for each comic.

And lastly, it shows the fact that collection operations can be chained.  My list of lists becomes a single order list, I purge out nils (probably redundant at this point), and then sort all the objects by their date field.

Ruby is such a fun language to program in. 🙂

A semester of search

My grad school class this semester is the Project Course, where the whole semester is spent on a group project.  No tests, no other grading besides the project, which is actually what I expected more of when starting grad work at Marist (that’s a different post though).  The project domain is search.

We have to build an application with an integrated search engine tuned for a specific problem.  Our group problem is an user driven online restaurant review site.  Our canonical example is searching for “boston seafood”, which should return all the posts that a human would, given the same tasks.  That means “the best lobster in bean town” counts as a hit that you’d want.  Guess what, SQL like clauses and regex’s aren’t going to cut it here.

But that’s ok, we don’t have to do everything from scratch.  We’re expected to base our solution on Lucene, which is a search SDK.  You build custom indexer, analyzer, and searcher classes from the Lucene base classes, and feed it documents.  Lucene does the heavily lifting of building the inverted index, and scoring the results based on the rules, weights, and policies you’ve given it.  A project like this is pretty open ended, as you can always make it better given more time, and more interesting analysis tricks.

The whole team is making nice progress, so for the last two weeks I’ve been able to focus squarely on Lucene integration code itself.  Pass one got some basic queries working in Lucene.  Pass two was earlier this week, when scoring started to be useful.  Pass three will be tonight, where I’ll start to integrate synonym support so that lobster is understood as a type of seafood, burgers are understood to be american food.  Though I’ll have to think about how to make sure crab cakes don’t show up in the desert category, though maybe we just need a hybrid seafood desert category.

A few interesting lessons have come out of the work so far.  First, search is way harder than most people think.  While Lucene gives you lots of nobs and levers to tweak how documents are ranked, the results of those tweaking aren’t always what you think.  It’s sort of like moving furniture by throwing bowling balls at it, you may get things close, but you do a lot of collatoral damage in the process.  Recently I was attempting to boost scores based on terms showing up in the subject of posts, which completely overwhelmed our post rating scoring, making low quality posts show up at the top of the list.

You also notice when people are using search badly, or more specifically using bad search.  Using SQL Like clauses is not search, it’s grep.  Unfortunately most php sites do that because they don’t have anything better (Lucene has been ported to a lot of language environments, php is not one of them).  The gentoo wikis fall into this category.

Finally, you realize that google’s scoring, while good in general, may not actually be what you want for your problem domain.  The fact that the word seafood shows up 3 times in a post doesn’t make it a better post, but default scoring gives it a boost based on the number of times relevant terms show up.  Badger, Badger, Badger, while being non kosher, shouldn’t be scored highly in our results, even if we had a category fully dedicated to badgers and mushrooms.