Tag Archives: drupal

Drupal Camp Western Mass

When your alarm goes off at 5:30am to go to a conference, part of you wants to skip and go back to bed. When it also means you’ve got to drive 2 1/2 hours in below zero conditions, there is even more inertia to just bail. I’m really happy I fought those primal urges and made it to Drupal Camp Western Mass yesterday, because it was pretty awesome.

Over the course of the day, given the 5 tracks that were being run, I could always find a really good talk. At one point it meant walking out of a bad talk, but I landed somewhere good eventually. I learned quite a bit more about some of the core of drupal, and hit up some of the more design oriented talks, which has given me a new reading list. There were 300 attendees there yesterday, which is a heck of showing for such grass roots organized conference far from a big city. It was also nice to be at a tech conference that wasn’t just a bunch of male geeks. At least a third of the audience was female.

I’ve got about 8 pages of notes, and a new bag of tricks to approach some of the more complex things we’d like to do for the Poughkeepsie Farm Project. I hope they run it again next year, as it’s definitely worth the drive and the time.

Drupal Talk Roundup

A couple weeks ago I gave the first MHVLUG talk of the year on the work I’ve done with Drupal for the Poughkeepsie Farm Project, Mid-Hudson Astronomical Association, and MHVLUG. I think it went well, but it’s sometimes hard to tell, especially because I’ve been trying out some new approaches on talks, so feedback comes in different ways.

When I was building the Android presentation for CPOSC, I realized I was building a generic Android tutorial, and I stopped myself. Why would anyone want to hear me give that presentation? You can get that all over the internets, on the youtubes and vimeos of the world. So, I thought long and hard about what I could uniquely bring to the table. The answer was pretty simple, talk about the things that I racked my brain on, in the context that I ran into them when building my application.

This drupal talk was much the same. Two days before the talk, while doing a dry run at home, I realized that I had a tutorial that was better covered by the internets. So, I started over, and edited heavily, and managed to produce a personal narrative of working with the PFP that showed the nuts and bolts of Drupal along the way. It was a much better presentation. It also made it easier to put a call for action in the presentation, for people to get involved with local non profits.

Because this was a narrative, and not a tutorial, the audience interaction is very different. People stop and ask clarification during a tutorial, but questions during a story are often considered rude. This led to a very distinct boundary between talk, running just about an hour, and questions, which carried for 30 minutes after that. Without questions during the talk, it’s a little harder to tell if people are engaged. I do have a vivid image in my head of looking out out on the audience and even the laptop folks were all eyes up and watching, so I think I succeeded there. The 30 minutes of questions, coming from at least 8 different people, helped reinforce that.

I also made a few mechanical changes to the presentation in Open Office, which others might find interesting. I’ve moved to using Fade Smoothly (Fast) for slide transitions, and Fade In (Fast) for element animation. Getting rid of the hard appear makes the whole thing feel more fluid, and as long as it’s fast, it doesn’t get in the way.

I also realized that when you put up a slide you feel like it should get your time, but sometimes there is nothing really to add. I had a brief walk through of drupal installation. During dry run I realized I spent way too much time talking about slides with relatively little talk worthly content. To keep myself honest during the talk I made those slides automatically advance after 5 or 10 seconds (depending on complexity). That meant I got all of that on screen, but capped it to 40 seconds, and didn’t find myself saying something like “what more can I say about this slide”.

Overall I think things went quite well, and am now looking forward to the ACM talk tomorrow night.

Upcoming Talk: Building a Community Site with Drupal

On Wednedsay, January 5th, I’ll be giving the MHVLUG lecture on Drupal.  It’s been two years since I started poking at Drupal in order to overall the Poughkeepsie Farm Project website, and this talk is largely going to be about that experience, and some of the basic lessons I’ve learned along the way.  While there will be plenty of technical bits, covering basics of getting deep enough into a drupal project to make it interesting, there is also another interesting story about getting involved with non profits.

The talk is coming along nicely, and I’m quite excited for giving it next week.

New MHAA Website

Over the past few weeks I’ve been redoing the Mid Hudson Astronomical Association website.  It looks like this:

Yes, it’s dark, but that’s when astronomy happens.  The site is built on Drupal, as I’ve gotten some experience there recently doing sites for the Poughkeepsie Farm Project and MHVLUG.  For people that want to know more about the tech side, I’ll be giving a talk in January at MHVLUG.

I did come across one really odd thing in working on the three boxes (I was calling them chicklets, but they look less like that with content in them). Round corners in CSS are awesome (thank you w3c). IE9, at least the version in Adobe Browser labs, still doesn’t support it (really Microsoft? I thought you were getting down with the standards). While a TD can have a round background, it’s border is always square (I almost understand why, but it definitely limits what you can do. It also took me a while to realize this was happening as the round is subtle enough on the front page). Div height 100% doesn’t work inside a TD (it seems like it should, but no one implemented it that way).

So the only way to get 3 columns that correctly degrade to 2 columns (50% of screen each) when one is missing (there will not always be a special event), have round borders, and be the same height is…. jquery. While on the one hand, it seems crazy, on the other hand, yay for jquery. You can view the source on the website to see how I did it.

Sunday Morning Source Code

I’m trying to figure out why duplicate key errors keep getting generated for the drupal twitter module, because it’s all open source, I started reading code to figure out why it didn’t seem to be using mysql BIGINTS.  I came across this comment:

* Dear PHP, I hate you so bad. Love, Jeff.
* Workaround for drupal_write_record(), which treats a DB bigint as a signed 32
* bit int on 32 bit PHP builds. We can STORE bigints, and PHP automatically
* converts them to floats in memory while we work with them, but db_placeholders()
* always treats them as %d and casts them back to dumb signed ints.
* See http://drupal.org/node/333788 for more info.
* Instead we’ll set the column type to ‘string’ which is a little like jumping
* off the Sears Tower because the elevator’s broken. But that’s life.

I laughed.

Scratching an itch

Apparently I’m now writing a drupal drush module for patch management, and slowly understanding what that entails.  I’ve looked around and surprisingly there really isn’t a good solution for this yet.  There are a number of solutions for applying other people’s patches that are posted somewhere, but what I really care about is being able to easily keep, and reapply the dozen or so patches I’ve made to drupal modules to make mhvlug.org work.  Some of these were due to bugs that don’t seem to be getting fixed any time soon.  Some are due to lost of drupal modules not working with PHP 5.3.

Regardless of the reason, I’ve apparently found a new itch to scratch, which hopefully isn’t going to take me too long, because I really need to get back to android hacking.

The challenge of upstream in Drupal

I’ve been using Drupal for a number of community websites for the past year.  Overall I quite like the system and how customizable it is.

What I’m not really thrilled by is the way the module structure exists on drupal.org.  The problem is that drupal core is a very small number of modules.  The bulk of useful functionality comes from user contributed modules.  So far so good, many projects run the same way.

The real issue is that the drupal project maintains a centralized CVS infrastructure for module developers.  There is a pretty formal process to get accepted as a module author, and no real way to get CVS access to just fix bugs.  The net result is that after finding 3 bugs in the date and callendaring modules, creating patches for each on my own environment, and submitting them upstream, only 1 has gotten looked at.  The fixes were submitted about 8 months ago, and I’ve pinged a few times.  I also wrote the author directly on one of the modules.

The root cause of this issue is that drupal has centralized infrastructure, but decentralized decision making.  It is a classic community wherein github would be a good solution.

I’ve been an absentee maintainer before, I know that sometimes interests change, and you move on to other things.  But at least with something like github, other folks can very easily extend what you’ve got and make the fixes themselves.  Hunting out RCS style patches in drupal discussion forums is just really problematic.  At this point I’ve got quite a number of local fixes, so module upgrade and maintenance is tricky.  The only saving grace is that most of those fixes are on modules that are effectively abandoned so I shouldn’t have to worry about an update coming down the pipe.

The Drupal community is soldiering on even with bad developer infrastructure.  I just wonder how much further along they’d be with better developer infrastructure.

The new Poughkeepsie Farm Project site is live!

About 14 months ago I raised my hand to help with a more interactive web presence for the Poughkeepsie Farm Project.  This kicked off a large discussion over the course of the year, a web committee, and a great pro bono new graphic design.  Many many people were involved to get this project to completion, I just consider myself a catalyst.

Today, after a year of work, we launched the new farmproject.org:

Go check it out.  Now that we’re on a drupal platform, we’ll be rolling in smaller features over time.  I’ve got a few ideas queued up that I’ll try to get out there for the first member pickup at the end of May.

Getting my head around Drupal: mhvlug.org version 4 a detailed guide

Over the past couple of weeks I redid the MHVLUG site as a Drupal site.  Drupal is a content management system, which is a fancy way of saying it’s a website, that lets you modify most of it’s parts via a web interface, and contains semi structured data.  I wrote a bit about it in the past.

The Motivation

I got motivated to redo the MHVLUG website after working on the farmproject website redesign, which also uses Drupal.  The MHVLUG website had had 3 previous iterations: static html stored in CVS, MoinMoin wiki, and Mediawiki.  The reasons for each previous switch are beyond the scope here, but each time I felt we took a step forward.

While the wiki approach worked ok for the LUG, the biggest set of edits on the wiki was around monthly meetings.  Before each meeting I’d need to move the meeting content into the front page.  After each meeting someone would need to copy and paste that into it’s own page (sometimes this got lost).  Meetings would get presentations added after the fact, but because it was a wiki, content and presentation were all wrapped up together.  Having the meeting data stored separate from presentation, and being able to create different slices of it for the site (next meeting on frontpage, full meeting pages for the archives, lists of past meetings, lists of future events, in calendars) was enough to get me over the hump.

Drupal Basics

The basic Drupal environment is a lot more bare bones than you would imagine.  It would make an ok blog out of the box, but that’s about it.  Before you get started with any real Drupal project you’ll need at least the following addon modules:

  • cck – content contruction kit, this lets you define custom types with custom fields
  • views2 – this is a basic query builder that lets you create custom slices of data to display as pages, blocks, or in a number of other ways.
  • devel – this gives you a really handy set of add on functions for debugging your types and views
  • admin menu – this will save you a lot of time

I’m giving you my wisdom in hindsight here.  I didn’t have devel or admin menu during the first bits of launching the new mhvlug.org… and man would they have saved me a bunch of time.

Meetings and Events

Beyond basic pages and stories (aka news), the mhvlug site has 2 main special types of data: Meetings and Events.  You could coax the 2 into 1 type, which simplifies some things, but meetings have enough extra bits of data (uploaded presentations, presenter info) that I chose to not do it that way.  Meetings and events are both a collection of a place, a time, and content.  After installing the date module, I had the functionality that I needed on the time front.  I added some custom fields for presenter and presentation on the meeting front, and with a not too complicated set of views had it so the next meeting showed on the front page automatically.  When Jan 7 rolls around, no one will have to go adjust the front page any more.  At this early stage I already had the win I was looking for.


While I had the time and content portions worked out pretty quickly for our Meetings, location was a bit more challenging.  We only have about 6 locations that we ever use for MHVLUG events or meetings, so I didn’t want to add addresses manually to each piece of content.  Fortunately with cck Drupal has the concept of a node link, which lets you build a relationship to another piece of content.  So now I had a 3rd custom type, Location.

Given that this is 2009, the minute you have an address, you want to have a google map to go along with it.  I spent a couple of days trying all manner of various google map modules before I threw up my hands on this one.  Every single module I experimented with didn’t do quite what I wanted, and needed an aweful lot of configuration.

Eventually, I went for the simple way out.  I just made a text field that I could embedded a google my map chunk of iframe code.  When the field is printed out, you get the map.  This turns out to be particularly useful as some of our locations don’t really have geocodable addresses.  So this challenges was overcome without any new modules.

Custom Display of Meetings

Although you can get a certain amount of customization out cck for custom node types, I found I couldn’t really get what I wanted when it came to the display for meetings.  Fortunately Drupal provides a mechanism for dealing with this, which is custom templates per node type.  By default the core content of each page is rendered using the node.tpl.php in your theme.  If you create note-meeting.tpl.php, it will use that to render meeting types instead.  This works for any custom node type.

It occurred to me that if someone was looking at a meeting or event in that was coming up, they’d immediately want to know where it was.  I wanted to do more than just print the name of the location, I wanted to pull in and display the map as well.  This was where I learned a few really important tricks.

dprint_r is the first one.  If you have the devel module enabled, you get access to some functions that you can use to help debug what’s going on in drupal.  dprint_r spits out a nicely formated version of the data structure you pass it in html.  You can thus use it in your templates to see whats going on.  As someone that thinks in data, this was critical to getting my head wrapped around what drupal was actually doing.

When you load an event page, drupal loads all the data for the event object you are accessing, and it loads the id, tile, and url for any referenced objects, in my case location.  To get more you use the node_load function, which loads any arbitrary object in drupal by it’s node id.  This let me pull in the whole location object and embed the map on meetings pages.  node_load has performance implications, so don’t use it everywhere all the time, but in this case it turned out to be cheap and powerful.

The php templates are just php code, so you can get even trickier.  Meeting location is only interesting prior to the meeting, so I adjusted the template so it only displayed when the meeting/event date was in the future.  Then the archive isn’t polluted with maps.  Works great once you figure out the chaining of date, time, datime objects you need.  Plus, make sure to get your timezone right! 🙂

The Calendar

I went live with basically the functionality above, but I realized that if I could get drupal to spit back out ical, I could get rid of my parallel calendar site I was maintaining.  There is a lot of documentation on how to do this with the calendar module… it’s all very confusing.  Once you have both date and calendar installed you’ll have the option in the administration panel to use the Date Wizard to create your calendar.  Do it!  It creates 8 linked views that give you a calendar at all levels (year, month, week, day), upcoming lists, an ical feed.  If you don’t like the date field it’s creating, just get rid of it after the fact.  Building those views by hand is just going to be a pain, and it’s much easier to tweak them after the fact.

I finally hit a point here where I needed to dive into code, because the ical specific portions of drupal are lacking.  Here are the bugs I found, fixed, and submitted patches upstream for.

In english, Drupal isn’t doing the required wrapping for multiline fields like it should in the spec.  ical wrapping is odd, so I understand why people haven’t fully implemented it yet.  You wrap at > 60 characters, and not on wordbreaks.  That’s right, cutting up a word in the middle is part of the spec, and things actually work better when you do that.  New lines need to start with “rn “, which is really important.  The other issue is that no one tested the recurrence rules much in drupal.  It turns out that in default drupal you can only have 1 event that follows a recurrence rule, i.e. every 2nd Tuesday of the month.  The processor bails before it gets to the second rule.

I fixed all this locally, built the patches, and sent them upstream.  This is the only time I had to dive into the code and fix things.  While it would have been great to not even have to dive in here, I’ve yet to pick up an ical base that I didn’t need to go tweek yet.  I had the same amount of work on the ruby icalendar stack when I played with it.  iCal is just a weird specification, that doesn’t look like what you’d expect.  This is what you get when Lotus and Microsoft build an interchange protocol in the late 90s.

Why iCal?  We found that 50% of our users are using Google Calendar now.  Export to iCal is required for anything that is time based, as Google Calendar is the defacto client for this (though Lightning in Thunderbird does a quite nice job as well).

Mailing List integration

MHVLUG has a mailman mailing list where most of our communication takes place.  Previously you needed to have an account on the website to edit, and a different account on the mailing list.  Through the user mailman manager you can let people easily subscribe and manage their list subscriptions.  This works really nicely, and has already gotten a few people on our mailing list that didn’t ever join before.

Fighting the Spam Bots

The moment you open up registration to the web, you get spam bots trying to login in, and post Chinese drug company links on your website.  It is the price we pay for such an open medium.  Spam bots nearly killed us under the moin wiki.  Media wiki faired better, but I still needed to go and revert a couple of pages every couple of months.  I found that in the first 2 days of drupal being up we had 3 bots signing up, never a good sign.

So here is the formula I’m using for drupal that seems to be quite effective:

  • Require that users confirm their registration via email activation.  This is the default for drupal, and helps quite a bit.
  • Adding Captcha and Recaptcha modules to prevent bots from bothering you with partial registrations in the first place.
  • And finally, use LoginToboggin to put non confirmed users into a penalty box group, which it will automatically purge after 30 days.

While the first 2 actions will protect your site, you’ll still pile up plenty of partial registrations, which just clutters up user management.  Having the system auto purge nonconfirmed accounts makes it all the more self tending.

Better URLs and URL migrations

Out of the box drupal has this totally ugly node/# model for urls.  While it is valid, it really sucks to look at, and google often penalizes you for that as well.  While there is some support for pretty urls out of the box, you really want to install path_auto right off the bat, which builds the url from the title of the node.

Specific to this migration was that we had over a hundred pages in the old media wiki (mostly meetings).  Which means people are going to have linked to something in the past, and not find it in the new site.  There was no way I was going to url map every old url to the new ones.  There is a interesting partial solution which seems to be work well, the search404 module.  When an unknown url comes in it breaks up the url path into words and runs it through the search engine.  If it’s 1 hit, it just takes you there.  If it’s more or less, it leaves you at the search404 page with the search results provided.  It’s not perfect, but I’m hoping it eases the transition (though I’m still getting hits from search for urls from the moin wiki, so some amount of that isn’t going to go away any time soon.)

Sending out Announcements

This is one place where I didn’t find anything in Drupal out of the box that did what I wanted, which is to take the meeting or event text, wrap it in a template with standard boiler plate, and email it to the mailing list.  I tried a few things, like simplenews, which turned out to be anything but.  I wasn’t at the point where I yet want to build a module from scratch (though I’ll probably get there at some point), so I did the next best thing, and hacked the crap out of it do what I want.  The module I hacked up with the print module.

Print provides printer friendly pages, as well as send by email functionality.  I just exposed the print function to the users, but send by email I kept as admin only, and gutted it so that for meeting node types it built me exactly the kind of boiler plate I wanted.  Using mimemail it sends it in both html and text, and looks pretty good.  It is a hack, but one I’m willing to live with for now.

Making Editing Pleasant

The last thing to mention is the fckeditor integration.  I added this early on just to make editing a lot easier on folks.  Drupal doesn’t use any special markup, it uses html under the covers.  Fckeditor is a quite good wysiwyg editor.  The only gotcha here is to make sure that you exclude certain administrative fields, because fckeditor will fix up your text into proper html (like wrapping it in P tags) on submit.  Some times you really don’t want that.  It’s actually quite amazing how good visual editors in javascript have gotten now a days, and what will come over in a copy and paste.

Pulling it all together

With the modules and configuration I’ve layed out here, I now have a quite good community site that supports events, calendars, users that edit pages, users able to manage their mailing lists subscriptions, and a front page that is always going to show the next meeting.  And beyond that, it’s just pretty.  I’ve also started playing with things like twitter integration, and sending email to the list on new news stories (which I’m manually doing now).

I learned a couple of lessons on Drupal in the process.  First, don’t be afraid of modules.  Drupal modules, especially the good ones, do a small amount of specific function.  To have a robust site that does what you want is going to take adding quite a few.  Building this up from scratch is what Drupal often gets dinged for compared to Joomla, but I actually like it better this way, as it provides more flexibility.

Second, sometimes there is no module to do what you want.  In that case you have 2 options, see if you can do it simpler (like I did on the map field), or see if you can hack up the function you need on a related module (like I did with the email announcement).  Both work, depending on what you are going for.

Lastly, the keystone to any site is really Views, CCK, and how you create your custom node types.  Think about this one the most.  What is a meeting?  What is an event?  You can always modify them later, but consider figuring out what your custom types are to be your first and most important mission when building a site.

Mediawiki vs Drupal for a community site

After our android hack-a-thon, Frank put a page in the MHVLUG wiki to start to stub out Android information that we’re all finding useful.  It’s small, and largely a stub, but it’s a start.  When I went back to shift a couple things around, I started to really wish this whole things was Drupal instead.

Drupal gets a lot of hate in the tech world, so I’m sure that I’ll get complaints or at least scoffs.  I’ll get into why I wish this was drupal in a bit, but first, some theories on why people hate drupal.

It’s popular.  There are over a million drupal sites out there, and you’d loose your tech street cred if you didn’t scoff at that which is popular.  I suspect that our high school experiences dealing with being unpopular probably shape that point of view.

It’s a big php application.  PHP has a low barrier of entry.  This is a double edge sword, because your average php developer is probably less skilled than your average developer in another language (Visual Basic has this even worse, as some time with the daily wtf will show.)  Being both php and big means that drupal security issues come up relatively regularly.  Like with wordpress, you need to keep on top of updates.  A 2 year old drupal install, like a 2 year old wordpress install, is basically a rusty door waiting to fall off.

People learn one tool, then try to make the world out of it.  I’ll brush off the first 2 complaints, as I think they are both addressable.  When talking to O’Connor recently, I found a much more reasonable complaint, which he was in the middle of.  Drupal is a content management system, and if you use it for semi structured content storage and display (i.e. a ghetto database), it’s great.  But because it has a module structure, a lot of people turn Drupal into a web application development framework…. which it really is not.  If you want that, check out rails, django, cake-php or something that actually gives you that level of control.  This is a standard issue – if you have a hammer, everything looks like nails.

If I was doing mhvlug.org over from scratch, it would be with Drupal.

What I’ve found over the last almost 7 years with MHVLUG is that being a wiki was better than being a static website, but that the number of edittors for the site is still in the single digits.  We’re a group of 150 people on the mailing list, 20 – 30 at a monthly meeting, 10 – 12 at our monthly lunch.  It’s a solid community, but it’s one with a pretty well defined reach, that’s been more or less constant now for at least the last 4 years.  People are fading in at basically exactly the same rate as people are moving away or fading out.

The bulk of what gets updated is information around the meetings.  If that was stored in a semi-structured way, it would be a whole lot easier to update in one place, and have it viewed different ways in different places.  If I didn’t have a project list queued out the door, around the block, down the hill, and… well you get the point, I’d seriously think about this one.

I’m pretty happy with how things are coming together in exactly this way when it comes to the farm website build on drupal.  Now that I understand what the right “types” structure is for the farm, the rest of it is just falling into place.

So for people looking to put up a website for a community in this day and age, I’d highly suggest that drupal is a good place to start.  Yes, you’ll catch grief from your other tech friends, but such is life.  Long term, I think it’s a pretty good call.

Update: I ended up redoing the MHVLUG website as a drupal site, with an extensive writeup on how.