The Best Sci Fi You've Never Seen is coming to SciFi Channel

I nearly lost it when I saw that Charlie Jade is coming to SciFi channel next week. This was a south african production a few years ago, and is one of the most innovative shows I've seen in a long time. The series has a bit of a slow start, but stick with it. Then run up to the end is incredible.

What I wouldn't give if this generated enough interest for SciFi to restart this series. It was incredible.

Pictures from The 5th Annual Dague / Tveekrem memorial day weekend party

I found myself surprised to realize that we'd been doing our memorial day sunday party What started as our house warming party 4 years ago, has now turned into our big annual event. We had another good turn out year, with ~ 35 folks, good beer, and great food. Ladder Ball was definitely the hit of the year.

Adrianne posted pictures on smugmug, check them out if you are interested.

The switch from xemacs -> emacs

I've always felt the root of the emacs vs. vi holy war (which is one of the longest standing holy wars in free software) basically came down to the following key point:

When you first were learning Linux / Unix, did your mentor use vi or emacs?

The answer to that is at least 90% correlation to your preferences. Much like most people share the politics of their parents, most people share the editor preferences of their mentors. Switches don't tend to happen unless mentors switch as well.

And in that camp, I'm an emacs guy. I learned it in college when I took my first programming class (which was in lisp). Our professor gave us a starting .emacs file, pointed us at the tutorial, and built macros that helped us out in our efforts.

A near decade with XEmacs

Then I graduated from college, and my first mentor at IBM was also an emacs guy, except he was an xemacs guy. It was emacs, but prettier. So I piled on, and was there ever since. Over the years I tried a couple of times to go back to emacs, but their font handling was never as good. I love programming in arial, as it's just really pleasant on the eyes (this shocks and horrifies people that line up = signs in declarations, but I don't much care. :) ).

A few years ago, just as emacs was getting reasonable variable width font support, xemacs integrated anti-aliased fonts into their CVS tree, and now I had another reason to stay on xemacs, because now everything looks pretty. Using xemacs was sometimes a pain, as a number of modes didn't really work right on it. I never had a reasonable html mode working that did indentation like I wanted.

Steve Yegge's Rant

Last month Steve Yegge had a post entitled Xemacs is Dead, Long Live Xemacs which was basically a call to unify around emacs because it had finally caught up, and it is being very actively maintained. I was skeptical, but decided to try again. Using the Ubuntu packages I lost my anti-aliasing, which meant this was a failed experiment.

But, after some research, I realized that emacs cvs not only has xft support in the tree, but that since March it's been the default. This is what will be emacs 23. I was already running xemacs out of cvs, so taking the same leap with emacs cvs wasn't such a big deal.

It's taken a couple of weeks to tweak my configuration to get me the same, or better, results with emacs as I had with xemacs. Last night I finally understood what I needed to get nxhtml to do my html.erb files correctly (ruby and html bits independently highlighted, and mode switching automatically when moving between code blocks). Minus 1 font issue with planner, I can definitely say I'm fully converted.

I'm also enjoying diving into elisp again. For whatever reason, life seems a bit more stable on emacs than it did on xemacs. And once emacs 23 actually makes it to distros, I won't even need to have my own binary builds. :)

Perfect Rails Development / Deployment environment with mercurial and passenger

A couple weeks ago I found phusion's passenger (aka mod_rails), and it's great. Passenger compiles an apache module which manages a rails app server for each application. It is on par with mongrel on speed, and so much easier to integrate into an average Linux environment for rails app hosting. Here's my new standard rails development environment (3 applications right now, with a couple more in the works).

Development with Mercurial

Distributed Source Code Management is the wave of the future, and I can't say enough good about it. My prefered system is Mercurial (aka hg), which I got to know why working on the Xen project. While git is getting a lot more hype of late, I still think hg is easier to use, and an easier switch for people that know subversion.

Rails makes it very easy to run a server locally for development, so having versioning locally makes perfect sense. I can hack away for days making changes on my laptop until I get to a point where I want the code to see the light of day. Then it is an easy hg push to put my code either into production, or into a repository for sharing.

Deployment with Passenger

Passenger
is a god send. Getting mongrel to do the right things on init on an ubuntu box was just a pain in the rear end. I like apache, I know apache, having to configure a web app outside of apache sucks. Passenger builds an apache plugin that is an rails app server. You don't need to know any more than that, because it just works.

Actually, you need to know 2 things, both of which add to passenger's awesomeness.

  • The rails app will run as the uid of the owner of your environment.rb file. This means you need to take care on how your permissions work on you deployed rails app. This is a good thing, as we'll see in a minute.
  • You don't need to restart apache to restart the rails app. You just need to touch tmp/restart.txt in the rails directory, and the app server restarts. This is very handy.

Bringing it all together with Mercurial Hooks

On my box where I'm deploying applications I created a new rails user, with a scrambled password, and just my ssh key to get in. This is the account in which I push production versions of apps to. It means that the rails apps run as user rails, which is ever further issolated than user www-data. You could even go a step further and have each rails app under a different user, but for me that's overkill.

The .hg/hgrc for one of my local projects looks as follows (with myhostname, myrailsapp, and myrailssite replaced to protect the innocent):

Now my push targets are in place.<br /><br />The last bit is adding a set of hooks on the target repository so that on each push the repo is updated, migrations are run, and the app is restarted. I've created /data/site/myrailssite/.hg/hgrc as follows:

And that's it. Now an hg push production force syncs the remote repo, runs any new migrations, and restarts passenger. A 1 step command, and everything is live.

While I've been a huge Ruby on Rails fan for the last year, deployment always sort of sucked with it. Now I'm a very happy camper with this setup which makes for very seemless development of rails applications.

Volunteer Motivations

The whole OLPC goes windows debacle has been going on for months now, creating incredible polarization on many fronts. A huge part of what actually excited many of the XO laptop volunteers was the chance for a Linux breakout market. I really think that senior leadership lost track of the fact that those blogging up the XO effort to its launch were largely in it for the Linux angle. By deciding to go the "natural route" and replace Linux with XP, lots of people have lost interest in the effort, including myself.

Putting more XP machines into the world isn't something very interesting to me, and it seems to go against the whole notion of the computer being a learning tool at all levels. I guess now the children of developing nations will be learning power point instead of python programming. What a shame.

Sculptie Physics in OpenSim

In secondlife sculpties only collide on bounding boxes, which make them really only suitable for visuals, not for part of complex builds. Due to some early work done by Teravus this week, that's no longer true for OpenSim. We're now creating a tri-mesh collision surface for sculpties and passing that into our physics engine. This code is young (only a week old), but you can see a demo of results below.

Sculptie Physics on OpenSim from Dahlia Trimble on Vimeo.

llTargetOmega in OpenSim, an epic journey in OpenSim prim updates

A few weeks ago I had an email conversation with Dale Innis about llTargetOmega support in OpenSim. This script function lets you set the angular velocity on a prim, which the client then interprets and displays spinning objects. It is not guarunteed to be synchronized between all clients, but it provides a rather useful visual effect regardless.

llTargetOmega didn't work for us a week ago, which confused me, as I saw that in the LSL portion of our code it was doing exactly the right thing and setting the angular velocity correctly. I should work, but it didn't. In the lack of it working people were setting fast timers that pushed out rotation updates. This caused a lot of extra load on the server, and was really the wrong approach for this.

Take 1: Terse Updates

OpenSim has 2 paths to sending information about Prims to the client (we'll get to the first one later). Terse Updates are a small update packet that contains just a bit of information on updated textures and some of the vectors used to establish prim position, velocity, acceleration, rotation, and angular velocity. When a prim is updated in the environment, Terse Updates are used to tell all the other clients about that change. One of the heavy users of the Terse Update path is the physics engine, as all the vectors the physics engine changes are in there. We've seen a lot of work on the Terse Update path as physics have gotten more and more tested.

On Tuesday I finally dug in and traced our Terse Update path, and found an interesting thing. When the object was physical (i.e. movement coming out of the physics engine) we did the right thing for Terse Updates. When it wasn't, we hard coded all the velocities to zero. So even if things were rotating, any time we sent an update we'd stop them.

Author: sdague
Date: 2008-05-06 15:17:00 -0700 (Tue, 06 May 2008)
New Revision: 4543

Modified:
trunk/OpenSim/Framework/IClientAPI.cs
trunk/OpenSim/Region/ClientStack/LindenUDP/LLClientView.cs
trunk/OpenSim/Region/Environment/Scenes/SceneObjectPart.cs
trunk/OpenSim/Region/Examples/SimpleModule/MyNpcCharacter.cs
Log:
send actual velocity and angular velocity in terse updates
instead of hardcoding to zero when the primitive is non physical.
llTargetOmega should work now.

Ok, so life is good, the issue is fixed, and we move on.

Except... it wasn't.

CSI: OpenSim, getting to the bottom of this

At this point a whole bunch of people on the IBM side jumped in. Mike Osias had a build that was on it's knees due to use of fake rotation, so he had all the good test cases, and opened mantis 1166. I'm not a scripter, so I needed some examples to know what should work. Alan Webb started to dive in and try to figure what was going on as well. I figured I'd spend an hour on it to try to figure out where things were at before getting back to avatar appearance bits.

After abount an hour Alan and I started comparing notes. The code in this area is extra confusing because we've got 2 vectors for angular velocity. An, no, they aren't actually different in any real way. Lots of people have tried to rationalize that they do different things, but they don't. This is cruft, and is part of what happens in an organically growing open project. The AngularVelocity / RotationalVelocity thing an opensim appendix, and should be surgically removed at some time in the near future.

But the behavior was even odder. I could set llTargetOmega on an object, and it wouldn't move. Then I'd touch it, and it would. I got Mike into my test environment and was looking at a spinning cube.

"Ok, you see that cube spinning?"

"No"

I grab it and move it. "What about now?"

"Yes, spinning now."

At this point I was confused a lot. Why would that be?

Take 2: Full Updates

I said there were 2 ways of a client finding out about prims, and this gets us back to the first one. In addition to Terse Update, there is what we call Full Updates, which are really just the full prim definition being sent down the wire. This is everything we know about the prim. This packet is also marked as reliable, to make sure the client doesn't drop it (terse updates are droppable).

And now we get back to organic code bases. One of the big activities since October was working physics in opensim. Lots and lots of work were spent on Terse Updates. Very little work was spent on full updates. It turned out that Full Updates were always hardcoding all the motion vectors to zero. The SendPrimitiveToClient function predates both physics and scripting by months. In a pre-physical opensim world passing the motion vectors didn't make any sense, as there wasn't anyway to set those values. The code worked well, so no one was really looking at it again, at least not in this specific area.

TerseUpdates (sent on minor prim movement) would make things spin. Full Updates (sent on initial prim rez, or after calls to osSetDynamicTextureURL) stopped the spinning. My earth projector turned out to be the perfect test case for this once I added rotation to the globe.

Originally I was going to punt on this and leave it to someone else, but then the thrill of the chase got to be too much. But there was one problem. This information is sent to the client in a 60 byte array, with basically undocumented positions. It was easy to fix terse updates because someone had already sorted it out, and I just needed to copy the decoding pattern there. For FullUpdates, it was more of a trial and error approach, represented by a series of checkins, reverts, and new attempts.

You know what happens when you get that array wrong? Spectacular fail. 3/4 of prims aren't in the right place, and touching an image board (user of osSetDynamicTextureURL) makes it fly away to some other part of your sim. Maybe in space. I eventually figured out a workable serialization:

Author: sdague
Date: 2008-05-07 12:44:22 -0700 (Wed, 07 May 2008)
New Revision: 4566

Modified:
trunk/OpenSim/Region/ClientStack/LindenUDP/LLClientView.cs
Log:
seriously hope this gives us rotation and rotational velocity

As you can see, I was getting a little punchy on changelog entries.

So we're done and fixed, and back to work.... well not quite.

Take 3: Deselected Objects

When you edit an object the client stops it's motion, as nothing would be more evil than trying to edit an object that is flying away from you at 60 m/s. When you've deselected the object it tells the server. But the object is stopped. The client needs to be told again that it is spinning. I got that critical information from melanie on IRC, which was enough to pass on the buck.

We had Mike almost working, and Mike is no slouch on our code base (he's sent in a couple dozen patches in the past), so I flipped this one back to him with "we're almost done, but you'll need to find the right place in the deselect path to generate a Terse Update. Then I think we've got full llTargetOmega support."

A day later Mike sent in this final patch:

Author: sdague
Date: 2008-05-08 05:48:29 -0700 (Thu, 08 May 2008)
New Revision: 4585

Modified:
trunk/OpenSim/Region/Environment/Scenes/Scene.PacketHandlers.cs
Log:
From: Michael Osias &lt;mosias@us.ibm.com&gt;

Patch to schedule terse update on deselect, specifically so llTargetOmega
sets rotational velocity on deselect.

This should complete our llTargetOmega support and fix:
http://opensimulator.org/mantis/view.php?id=1178

And now. For real. llTargetOmega works.

Final Thoughts

Avartar Appearance as a User Service isn't coming this week, sorry folks. The above epic took much of my hacking time this week. It was a pretty solid educational experience for me in the way we actually communicate the contents of the Scene to the client, which was good to learn after a year on the project. :)

Something else to take away from this. Lot's of focus is currently on the OpenSim scripting implementation, as it should be, as that's a huge user visible portion of our function. llTargetOmega it self is &lt; 6 lines of implementation. But our supporting scene model needed some work to actually get that info to the client.

I get asked all the time "how long until my favorite feature X is implemented", and the answer is always an unsatisfying (to me and them), "I'm not sure". Sometimes the plumbing is already there, and it's quick. Other times we're doing deep dives into our code base to implement what seems to the user to be a very simple function.

We're making constant forward progress, I'd even say rapid constant forward progress, but patience is always a good thing. Also, if you want OpenSim to work for whatever you application is, you should be trying to use it now and filing bug reports. That's how we function, personal itches, and knocking of mantis reports. Any ability that you have to narrow the bug to a specific section of code (even if you don't have a fix), helps a lot as well, as it removes possibly hours of core developer time trying to track down where things fail.

Weekend Rails Hacking

For the past 4 years I've been using evite to manage the RSVPs for our memorial day weekend party. Given that it's a pretty large scale pot luck event, it's helpful to have a system where people can respond with a message that others can see. The reduces my need to field "what should we bring" questions, as you can easily see what everyone else is bringing and react accordingly.

Evite sucks. While it doesn't force attendees to make accounts, it makes it look like it does. The evite.com emails tend to catch as spam. And the interface is now dubious under firefox. The idea is still good, but it hasn't really ramped with the trends in the rest of the service web application space.

One of the key things I wanted in an evite replacement is getting rid of user logins. Given an event, and an email address, you can come up with a unique key that qualifies that person for that event. That means the user just follows a link, and they are in. Links are unique for people. If you make the key a hash of the person's email and some secret seed key for the event, you've got something cryptographically strong as well. No one can modify another person's entry because the key needs to match before you get any info.

Saturday was a rainy day, so I built this system. By Saturday night I had most of it working, and had rolled this out live by Sunday afternoon. This was my first rails 2.0 app, so I needed to catch up on a few things along the way. Things I learned:

  • Rails 2 creates scaffolds in a slightly new way. That threw me for a bit, as I had already built models for most of my objects before creating scaffolds. The new way (putting attributes on the command line) looks like it is designed to make rails tooling easier.
  • ActionMailer is crazy easy. It even does multipart mixed emails really easily. My mhvlug mailer script for month announcements is going to need to be converted to this at some point.
  • Rails has a word_wrap function in the view context. Of course it does, why did I even doubt that.
  • The google maps API is impressive. I had maps based on event location within 60 minutes of signing up for my Google Maps API key.
  • The f= param on maps.google.com is which fuction to drive. q: location query, d: directions. That took a little bit of reading urls to realize.
  • It's pretty easy to integrate mercurial push to auto restart a rails app if it's running under passenger.
  • If you are running multiple versions of rails applicatoins under passenger, delete all the rails links in vendor/ so that it picks up the right rails environment.
  • arround_filter in rails is really handy to catch generic exceptions and dump people off to an error page that isn't the default rails one.

All in all, I was really happy how this turned out. As soon as I get some free time I'll genericize the bits of the app that I coded just for our event, and get this out on rubyforge. I only wish there was a rails equiv of gems, as I've still found that it isn't entirely clear how to best package a rails application as an easy to download open source component.