Tag Archives: drupal

Sunday Morning Source Code

I’m trying to figure out why duplicate key errors keep getting generated for the drupal twitter module, because it’s all open source, I started reading code to figure out why it didn’t seem to be using mysql BIGINTS.  I came across this comment:

/**
* Dear PHP, I hate you so bad. Love, Jeff.
*
* Workaround for drupal_write_record(), which treats a DB bigint as a signed 32
* bit int on 32 bit PHP builds. We can STORE bigints, and PHP automatically
* converts them to floats in memory while we work with them, but db_placeholders()
* always treats them as %d and casts them back to dumb signed ints.
*
* See http://drupal.org/node/333788 for more info.
*
* Instead we’ll set the column type to ‘string’ which is a little like jumping
* off the Sears Tower because the elevator’s broken. But that’s life.
*/

I laughed.

Scratching an itch

Apparently I’m now writing a drupal drush module for patch management, and slowly understanding what that entails.  I’ve looked around and surprisingly there really isn’t a good solution for this yet.  There are a number of solutions for applying other people’s patches that are posted somewhere, but what I really care about is being able to easily keep, and reapply the dozen or so patches I’ve made to drupal modules to make mhvlug.org work.  Some of these were due to bugs that don’t seem to be getting fixed any time soon.  Some are due to lost of drupal modules not working with PHP 5.3.

Regardless of the reason, I’ve apparently found a new itch to scratch, which hopefully isn’t going to take me too long, because I really need to get back to android hacking.

The challenge of upstream in Drupal

I’ve been using Drupal for a number of community websites for the past year.  Overall I quite like the system and how customizable it is.

What I’m not really thrilled by is the way the module structure exists on drupal.org.  The problem is that drupal core is a very small number of modules.  The bulk of useful functionality comes from user contributed modules.  So far so good, many projects run the same way.

The real issue is that the drupal project maintains a centralized CVS infrastructure for module developers.  There is a pretty formal process to get accepted as a module author, and no real way to get CVS access to just fix bugs.  The net result is that after finding 3 bugs in the date and callendaring modules, creating patches for each on my own environment, and submitting them upstream, only 1 has gotten looked at.  The fixes were submitted about 8 months ago, and I’ve pinged a few times.  I also wrote the author directly on one of the modules.

The root cause of this issue is that drupal has centralized infrastructure, but decentralized decision making.  It is a classic community wherein github would be a good solution.

I’ve been an absentee maintainer before, I know that sometimes interests change, and you move on to other things.  But at least with something like github, other folks can very easily extend what you’ve got and make the fixes themselves.  Hunting out RCS style patches in drupal discussion forums is just really problematic.  At this point I’ve got quite a number of local fixes, so module upgrade and maintenance is tricky.  The only saving grace is that most of those fixes are on modules that are effectively abandoned so I shouldn’t have to worry about an update coming down the pipe.

The Drupal community is soldiering on even with bad developer infrastructure.  I just wonder how much further along they’d be with better developer infrastructure.

The new Poughkeepsie Farm Project site is live!

About 14 months ago I raised my hand to help with a more interactive web presence for the Poughkeepsie Farm Project.  This kicked off a large discussion over the course of the year, a web committee, and a great pro bono new graphic design.  Many many people were involved to get this project to completion, I just consider myself a catalyst.

Today, after a year of work, we launched the new farmproject.org:

Go check it out.  Now that we’re on a drupal platform, we’ll be rolling in smaller features over time.  I’ve got a few ideas queued up that I’ll try to get out there for the first member pickup at the end of May.

Getting my head around Drupal: mhvlug.org version 4 a detailed guide

Over the past couple of weeks I redid the MHVLUG site as a Drupal site.  Drupal is a content management system, which is a fancy way of saying it’s a website, that lets you modify most of it’s parts via a web interface, and contains semi structured data.  I wrote a bit about it in the past.

The Motivation

I got motivated to redo the MHVLUG website after working on the farmproject website redesign, which also uses Drupal.  The MHVLUG website had had 3 previous iterations: static html stored in CVS, MoinMoin wiki, and Mediawiki.  The reasons for each previous switch are beyond the scope here, but each time I felt we took a step forward.

While the wiki approach worked ok for the LUG, the biggest set of edits on the wiki was around monthly meetings.  Before each meeting I’d need to move the meeting content into the front page.  After each meeting someone would need to copy and paste that into it’s own page (sometimes this got lost).  Meetings would get presentations added after the fact, but because it was a wiki, content and presentation were all wrapped up together.  Having the meeting data stored separate from presentation, and being able to create different slices of it for the site (next meeting on frontpage, full meeting pages for the archives, lists of past meetings, lists of future events, in calendars) was enough to get me over the hump.

Drupal Basics

The basic Drupal environment is a lot more bare bones than you would imagine.  It would make an ok blog out of the box, but that’s about it.  Before you get started with any real Drupal project you’ll need at least the following addon modules:

  • cck – content contruction kit, this lets you define custom types with custom fields
  • views2 – this is a basic query builder that lets you create custom slices of data to display as pages, blocks, or in a number of other ways.
  • devel – this gives you a really handy set of add on functions for debugging your types and views
  • admin menu – this will save you a lot of time

I’m giving you my wisdom in hindsight here.  I didn’t have devel or admin menu during the first bits of launching the new mhvlug.org… and man would they have saved me a bunch of time.

Meetings and Events

Beyond basic pages and stories (aka news), the mhvlug site has 2 main special types of data: Meetings and Events.  You could coax the 2 into 1 type, which simplifies some things, but meetings have enough extra bits of data (uploaded presentations, presenter info) that I chose to not do it that way.  Meetings and events are both a collection of a place, a time, and content.  After installing the date module, I had the functionality that I needed on the time front.  I added some custom fields for presenter and presentation on the meeting front, and with a not too complicated set of views had it so the next meeting showed on the front page automatically.  When Jan 7 rolls around, no one will have to go adjust the front page any more.  At this early stage I already had the win I was looking for.

Locations

While I had the time and content portions worked out pretty quickly for our Meetings, location was a bit more challenging.  We only have about 6 locations that we ever use for MHVLUG events or meetings, so I didn’t want to add addresses manually to each piece of content.  Fortunately with cck Drupal has the concept of a node link, which lets you build a relationship to another piece of content.  So now I had a 3rd custom type, Location.

Given that this is 2009, the minute you have an address, you want to have a google map to go along with it.  I spent a couple of days trying all manner of various google map modules before I threw up my hands on this one.  Every single module I experimented with didn’t do quite what I wanted, and needed an aweful lot of configuration.

Eventually, I went for the simple way out.  I just made a text field that I could embedded a google my map chunk of iframe code.  When the field is printed out, you get the map.  This turns out to be particularly useful as some of our locations don’t really have geocodable addresses.  So this challenges was overcome without any new modules.

Custom Display of Meetings

Although you can get a certain amount of customization out cck for custom node types, I found I couldn’t really get what I wanted when it came to the display for meetings.  Fortunately Drupal provides a mechanism for dealing with this, which is custom templates per node type.  By default the core content of each page is rendered using the node.tpl.php in your theme.  If you create note-meeting.tpl.php, it will use that to render meeting types instead.  This works for any custom node type.

It occurred to me that if someone was looking at a meeting or event in that was coming up, they’d immediately want to know where it was.  I wanted to do more than just print the name of the location, I wanted to pull in and display the map as well.  This was where I learned a few really important tricks.

dprint_r is the first one.  If you have the devel module enabled, you get access to some functions that you can use to help debug what’s going on in drupal.  dprint_r spits out a nicely formated version of the data structure you pass it in html.  You can thus use it in your templates to see whats going on.  As someone that thinks in data, this was critical to getting my head wrapped around what drupal was actually doing.

When you load an event page, drupal loads all the data for the event object you are accessing, and it loads the id, tile, and url for any referenced objects, in my case location.  To get more you use the node_load function, which loads any arbitrary object in drupal by it’s node id.  This let me pull in the whole location object and embed the map on meetings pages.  node_load has performance implications, so don’t use it everywhere all the time, but in this case it turned out to be cheap and powerful.

The php templates are just php code, so you can get even trickier.  Meeting location is only interesting prior to the meeting, so I adjusted the template so it only displayed when the meeting/event date was in the future.  Then the archive isn’t polluted with maps.  Works great once you figure out the chaining of date, time, datime objects you need.  Plus, make sure to get your timezone right! 🙂

The Calendar

I went live with basically the functionality above, but I realized that if I could get drupal to spit back out ical, I could get rid of my parallel calendar site I was maintaining.  There is a lot of documentation on how to do this with the calendar module… it’s all very confusing.  Once you have both date and calendar installed you’ll have the option in the administration panel to use the Date Wizard to create your calendar.  Do it!  It creates 8 linked views that give you a calendar at all levels (year, month, week, day), upcoming lists, an ical feed.  If you don’t like the date field it’s creating, just get rid of it after the fact.  Building those views by hand is just going to be a pain, and it’s much easier to tweak them after the fact.

I finally hit a point here where I needed to dive into code, because the ical specific portions of drupal are lacking.  Here are the bugs I found, fixed, and submitted patches upstream for.

In english, Drupal isn’t doing the required wrapping for multiline fields like it should in the spec.  ical wrapping is odd, so I understand why people haven’t fully implemented it yet.  You wrap at > 60 characters, and not on wordbreaks.  That’s right, cutting up a word in the middle is part of the spec, and things actually work better when you do that.  New lines need to start with “rn “, which is really important.  The other issue is that no one tested the recurrence rules much in drupal.  It turns out that in default drupal you can only have 1 event that follows a recurrence rule, i.e. every 2nd Tuesday of the month.  The processor bails before it gets to the second rule.

I fixed all this locally, built the patches, and sent them upstream.  This is the only time I had to dive into the code and fix things.  While it would have been great to not even have to dive in here, I’ve yet to pick up an ical base that I didn’t need to go tweek yet.  I had the same amount of work on the ruby icalendar stack when I played with it.  iCal is just a weird specification, that doesn’t look like what you’d expect.  This is what you get when Lotus and Microsoft build an interchange protocol in the late 90s.

Why iCal?  We found that 50% of our users are using Google Calendar now.  Export to iCal is required for anything that is time based, as Google Calendar is the defacto client for this (though Lightning in Thunderbird does a quite nice job as well).

Mailing List integration

MHVLUG has a mailman mailing list where most of our communication takes place.  Previously you needed to have an account on the website to edit, and a different account on the mailing list.  Through the user mailman manager you can let people easily subscribe and manage their list subscriptions.  This works really nicely, and has already gotten a few people on our mailing list that didn’t ever join before.

Fighting the Spam Bots

The moment you open up registration to the web, you get spam bots trying to login in, and post Chinese drug company links on your website.  It is the price we pay for such an open medium.  Spam bots nearly killed us under the moin wiki.  Media wiki faired better, but I still needed to go and revert a couple of pages every couple of months.  I found that in the first 2 days of drupal being up we had 3 bots signing up, never a good sign.

So here is the formula I’m using for drupal that seems to be quite effective:

  • Require that users confirm their registration via email activation.  This is the default for drupal, and helps quite a bit.
  • Adding Captcha and Recaptcha modules to prevent bots from bothering you with partial registrations in the first place.
  • And finally, use LoginToboggin to put non confirmed users into a penalty box group, which it will automatically purge after 30 days.

While the first 2 actions will protect your site, you’ll still pile up plenty of partial registrations, which just clutters up user management.  Having the system auto purge nonconfirmed accounts makes it all the more self tending.

Better URLs and URL migrations

Out of the box drupal has this totally ugly node/# model for urls.  While it is valid, it really sucks to look at, and google often penalizes you for that as well.  While there is some support for pretty urls out of the box, you really want to install path_auto right off the bat, which builds the url from the title of the node.

Specific to this migration was that we had over a hundred pages in the old media wiki (mostly meetings).  Which means people are going to have linked to something in the past, and not find it in the new site.  There was no way I was going to url map every old url to the new ones.  There is a interesting partial solution which seems to be work well, the search404 module.  When an unknown url comes in it breaks up the url path into words and runs it through the search engine.  If it’s 1 hit, it just takes you there.  If it’s more or less, it leaves you at the search404 page with the search results provided.  It’s not perfect, but I’m hoping it eases the transition (though I’m still getting hits from search for urls from the moin wiki, so some amount of that isn’t going to go away any time soon.)

Sending out Announcements

This is one place where I didn’t find anything in Drupal out of the box that did what I wanted, which is to take the meeting or event text, wrap it in a template with standard boiler plate, and email it to the mailing list.  I tried a few things, like simplenews, which turned out to be anything but.  I wasn’t at the point where I yet want to build a module from scratch (though I’ll probably get there at some point), so I did the next best thing, and hacked the crap out of it do what I want.  The module I hacked up with the print module.

Print provides printer friendly pages, as well as send by email functionality.  I just exposed the print function to the users, but send by email I kept as admin only, and gutted it so that for meeting node types it built me exactly the kind of boiler plate I wanted.  Using mimemail it sends it in both html and text, and looks pretty good.  It is a hack, but one I’m willing to live with for now.

Making Editing Pleasant

The last thing to mention is the fckeditor integration.  I added this early on just to make editing a lot easier on folks.  Drupal doesn’t use any special markup, it uses html under the covers.  Fckeditor is a quite good wysiwyg editor.  The only gotcha here is to make sure that you exclude certain administrative fields, because fckeditor will fix up your text into proper html (like wrapping it in P tags) on submit.  Some times you really don’t want that.  It’s actually quite amazing how good visual editors in javascript have gotten now a days, and what will come over in a copy and paste.

Pulling it all together

With the modules and configuration I’ve layed out here, I now have a quite good community site that supports events, calendars, users that edit pages, users able to manage their mailing lists subscriptions, and a front page that is always going to show the next meeting.  And beyond that, it’s just pretty.  I’ve also started playing with things like twitter integration, and sending email to the list on new news stories (which I’m manually doing now).

I learned a couple of lessons on Drupal in the process.  First, don’t be afraid of modules.  Drupal modules, especially the good ones, do a small amount of specific function.  To have a robust site that does what you want is going to take adding quite a few.  Building this up from scratch is what Drupal often gets dinged for compared to Joomla, but I actually like it better this way, as it provides more flexibility.

Second, sometimes there is no module to do what you want.  In that case you have 2 options, see if you can do it simpler (like I did on the map field), or see if you can hack up the function you need on a related module (like I did with the email announcement).  Both work, depending on what you are going for.

Lastly, the keystone to any site is really Views, CCK, and how you create your custom node types.  Think about this one the most.  What is a meeting?  What is an event?  You can always modify them later, but consider figuring out what your custom types are to be your first and most important mission when building a site.

Mediawiki vs Drupal for a community site

After our android hack-a-thon, Frank put a page in the MHVLUG wiki to start to stub out Android information that we’re all finding useful.  It’s small, and largely a stub, but it’s a start.  When I went back to shift a couple things around, I started to really wish this whole things was Drupal instead.

Drupal gets a lot of hate in the tech world, so I’m sure that I’ll get complaints or at least scoffs.  I’ll get into why I wish this was drupal in a bit, but first, some theories on why people hate drupal.

It’s popular.  There are over a million drupal sites out there, and you’d loose your tech street cred if you didn’t scoff at that which is popular.  I suspect that our high school experiences dealing with being unpopular probably shape that point of view.

It’s a big php application.  PHP has a low barrier of entry.  This is a double edge sword, because your average php developer is probably less skilled than your average developer in another language (Visual Basic has this even worse, as some time with the daily wtf will show.)  Being both php and big means that drupal security issues come up relatively regularly.  Like with wordpress, you need to keep on top of updates.  A 2 year old drupal install, like a 2 year old wordpress install, is basically a rusty door waiting to fall off.

People learn one tool, then try to make the world out of it.  I’ll brush off the first 2 complaints, as I think they are both addressable.  When talking to O’Connor recently, I found a much more reasonable complaint, which he was in the middle of.  Drupal is a content management system, and if you use it for semi structured content storage and display (i.e. a ghetto database), it’s great.  But because it has a module structure, a lot of people turn Drupal into a web application development framework…. which it really is not.  If you want that, check out rails, django, cake-php or something that actually gives you that level of control.  This is a standard issue – if you have a hammer, everything looks like nails.

If I was doing mhvlug.org over from scratch, it would be with Drupal.

What I’ve found over the last almost 7 years with MHVLUG is that being a wiki was better than being a static website, but that the number of edittors for the site is still in the single digits.  We’re a group of 150 people on the mailing list, 20 – 30 at a monthly meeting, 10 – 12 at our monthly lunch.  It’s a solid community, but it’s one with a pretty well defined reach, that’s been more or less constant now for at least the last 4 years.  People are fading in at basically exactly the same rate as people are moving away or fading out.

The bulk of what gets updated is information around the meetings.  If that was stored in a semi-structured way, it would be a whole lot easier to update in one place, and have it viewed different ways in different places.  If I didn’t have a project list queued out the door, around the block, down the hill, and… well you get the point, I’d seriously think about this one.

I’m pretty happy with how things are coming together in exactly this way when it comes to the farm website build on drupal.  Now that I understand what the right “types” structure is for the farm, the rest of it is just falling into place.

So for people looking to put up a website for a community in this day and age, I’d highly suggest that drupal is a good place to start.  Yes, you’ll catch grief from your other tech friends, but such is life.  Long term, I think it’s a pretty good call.

Update: I ended up redoing the MHVLUG website as a drupal site, with an extensive writeup on how.

Programming with Constraints and my adventures with Drupal

I’ve finally wrapped my head around the blocks, views, and content construction kit model for Drupal, which we’re going to be using for the upcoming relaunch of the Poughkeepsie Farm Project website (you can see the work in progress here).  It took a number of days to make a mental breakthrough that let me understand what it was that Drupal wanted me to do to get results.  The big break through was getting that custom content types is really what I wanted and then how to use views to display them correctly.

The exercise was one of programming with constraints.  As a veteran software developer, I’m used to getting a blank page where I can happily build up features from scratch. That has the advantage of the end product doing exactly what I want, but has the disadvantage of having to do everything from scratch.  While from scratch is often satisfying, it’s also often really tedious.  By the time you’ve written your 3rd password reset system for a web application, you’ll feel that way as well.  Life is too short to keep repeating yourself.

The alternative is something like the drupal approach.  Start with a lot of the application done, and just complete the pieces you need for the project.  As a veteran software developer, I’ve been there too.  I’ve come into many a project late in the development cycle, and had to work inside the constraints that are already there because of decisions made in the past.  These decisions might have been based on money, expertise, timing, politics, or any number of other reasons.  At the end of the day it doesn’t really matter, you just have to accept what is, and figure out how to work with it.

Building on top of a framework like Drupal is like being brought in late on a project, where the rest of your team is the drupal community.  They made decisions on how things will work, and you just need to figure out what those were, and if you can work constructively inside those contraints.  It’s a different skill than the “from scratch” skill, but is just a valuable.  The real world has a lot more half finished projects out there than blank slates.

People love to complain about frameworks, especially tech folks, because they feel the constraints are hampering their productivity.  As any tech person knows, when you switch up what you are working on substantially, you go through the “wow I’m stupid” phase again.  Your output suffers, and you feel like you are never making any forward progress.  It takes weeks to months to get familiar enough with the new skill set, and actually start creating anything of value.  No one likes to feel stupid, so a very standard reaction is to lash out at the tools, say they are the issue, and go back to your comfort zone.  You get a lot of hating in tech on exactly that.  But not all frameworks or tools are bad, and writing things off as evil because you never invested the time to understand how they work are as silly in web frameworks as they are with compound miter saws and belt sanders.

At some point in the next couple of months I’m going to write up my own “getting your head around drupal” blog post, because I have to admit I did it by brute force.  Eventually I got enough insites to start getting productive.  It took a while.  I’ve been actively working on the farm site since september, and did a whole drupal layout back in the sprint just to kick the tires.  But overall, this was definitely worth it, and I’m quite happy with the output levels I’m getting now.