Tag Archives: fastmail

Syncing Sieve Rules in Fastmail, the hard way

I’ve been hosting my email over at Fastmail for years, and for the most part the service is great. The company understands privacy, contributes back to open source, and is incredibly reliable. One of the main reasons I moved off of gmail was their mail filtering system was not fine grained enough to deal with my email stream (especially open source project emails). Fastmail supports sieve, which lets you write quite complex filtering rules. There was only one problem, syncing those rules.

My sieve rules are currently just north of 700 lines. Anything that complex is something that I like to manage in git, so that if I mess something up, it’s easy to revert to known good state.

No API for Sieve

Fastmail does not support any kind of API for syncing Sieve rules. There is an official standard for this, called¬†MANAGESIEVE, but the technology stack Fastmail uses doesn’t support it. I’ve filed tickets over the years that mostly got filed away as future features.

When I first joined Fastmail, their website was entirely classic html forms. Being no slouch, I had a python mechanize script that would log in as me, then navigate to the upload form, and submit it. This worked well for years. I had a workflow where I’d make a sieve change, sync via script, see that it generated no errors, then commit. I have 77 commits to my sieve rules repository going back to 2013.

But, a couple of years ago the Fastmail team refreshed their user interface to a Javascript based UI (called Overture). It’s a much nicer UI, but it means it only works with a javascript enabled browser. Getting to the form box where I can upload my sieve rules is about 6 clicks. I stopped really tweaking the rules regularly because of the friction of updating them through clear / copy / paste.

Using Selenium for unintended purposes

Selenium is pretty amazing web test tool. It gives you an API to drive a web browser remotely. With recent versions of Chrome, there is even a headless chrome driver, so you can do this without popping up a graphics window. You can drive this all from python (or your language of choice).

An off hand comment by Nibz about using Selenium for something no one intended got me thinking: could I manage to get this to do my synchronization?

Answer, yes. Also, this is one of the goofiest bits of code that I’ve ever written.

#!/usr/bin/env python3

import configparser
import os
import sys

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

config = configparser.ConfigParser()
config.read("config.ini")

chrome_options = Options()
chrome_options.add_argument("--headless")
driver = webdriver.Chrome(executable_path=os.path.abspath("/usr/local/bin/chromedriver"),
                          chrome_options=chrome_options)

driver.get("https://fastmail.fm")

timeout = 120
try:
    element_present = EC.presence_of_element_located((By.NAME, 'username'))
    WebDriverWait(driver, timeout).until(element_present)

    # Send login information

    user = driver.find_element_by_name("username")
    passwd = driver.find_element_by_name("password")
    user.send_keys(config["default"]["user"])
    passwd.send_keys(config["default"]["pass"])
    driver.find_element_by_class_name("v-Button").click()

    print("Logged in")

    # wait for login to complete
    element_present = EC.presence_of_element_located((By.CLASS_NAME, 'v-MainNavToolbar'))
    WebDriverWait(driver, timeout).until(element_present)

    # click settings menu to make elements visible
    driver.find_element_by_class_name("v-MainNavToolbar").click()

    # And follow to settings page
    driver.find_element_by_link_text("Settings").click()

    # Wait for settings page to render, oh Javascript
    element_present = EC.presence_of_element_located((By.LINK_TEXT, 'Rules'))
    WebDriverWait(driver, timeout).until(element_present)

    # Click on Rules link
    driver.find_element_by_link_text("Rules").click()

    # Click on edit custom sieve code
    element_present = EC.presence_of_element_located((By.LINK_TEXT, 'Edit custom sieve code'))
    WebDriverWait(driver, timeout).until(element_present)
    driver.find_element_by_link_text("Edit custom sieve code").click()

    print("Editing")

    # This is super unstable, I hate that we have to go by webid
    element_present = EC.presence_of_element_located((By.CLASS_NAME, 'v-EditSieve-rules'))
    WebDriverWait(driver, timeout).until(element_present)

    print("Find form")
    elements = driver.find_elements_by_css_selector("textarea.v-Text-input")
    element = elements[-1]

    # Find the submit button
    elements = driver.find_elements_by_css_selector("button")
    for e in elements:
        if "Save" in e.text:
            submit = e

    print("Found form")
    # And replace the contents
    element.clear()

    with open("rules.txt") as f:
        element.send_keys(f.read())

    # This is the Save button
    print("Submitted!")
    submit.click()

except TimeoutException as e:
    print(e)
    print("Timed out waiting for page to load")
    sys.exit(0)

print("Done!")

Basic Flow

I won’t do a line by line explanation, but there are a few concepts that make the whole thing fall in line.

The first is the use of WebDriverWait. This is an OvertureJS application, which means that clicking parts of the screen trigger an ajax interaction, and it may be some time before the screen “repaints”. This could be a new page, a change to the existing page, an element becoming visible. Find a thing, click a thing, wait for the next thing. There is a 5 click interaction before I get to the sieve edit form, then a save button click to finish it off.

Finding things is important, and sometimes hard. Being an OvertureJS application, div ids are pretty much useless. So I stared a lot in Chrome inspector at what looked like stable classes to find the right things to click on. All of those could change with new versions of the UI, so this is fragile at best. Some times you just have to count, like finding the last textarea on the Rules page. Some times you have to inspect elements, like looking through all the buttons on a page to find the one that says “Save”.

Filling out forms is done with sendKeys, which approximates typing by sending 1 character every few milliseconds. If you run non headless it makes for amusing animation. My sieve file is close to 20,000 characters, so this takes more than a full minute to put that content in one character at a time. But at least it’s a machine, so no typos.

The Good and the Bad

The good thing is this all seems to work, pretty reliably. I’ve been running it for the last week and all my changes are getting saved correctly.

The bad things are you can’t have 2 factor enabled and use this, because unlike things like IMAP where you can provision an App password for Fastmail, this is really logging in and pretending to be you clicking through the website and typing. There are no limited users for that.

It’s also slow. A full run takes

It’s definitely fragile, I’m sure an update to their site is going to break it. And then I’ll be in Chrome inspector again to figure out how to make this work.

But, on the upside, this let me learn a more general purpose set of tools for crawling and automating the modern web (which requires javascript). I’ve used this technique for a few sites now, and it’s a good technique to add to your bag of tricks.

The Future

Right now this script is in the same repo as my rules. This also requires setting up the selenium environment and headless chrome, which I’ve not really documented. I will take some time to split this out on github so others could use it.

I would love it if Fastmail would support MANAGESIEVE, or have an HTTP API to fetch / store sieve rules. Anything where I could use a limited app user instead of my full user. I really want to delete this code and never speak of it again, but a couple of years and closed support tickets later, and this is the best I’ve got.

If you know someone in Fastmail engineering and can ask them about having a supported path to programatically update sieve rules, that would be wonderful. I know a number of software developers that have considered the switch to Fastmail, but stopped when the discovered that updating sieve can only be done in the webui.

Updated (12/15/2017): via Twitter the Fastmail team corrected me that it’s not Angular, but their own JS toolkit called OvertureJS. The article has been corrected to reflect that.

 

Moving off GMail

In early December I finally decided it was time to move my primary email out of google. There were a few reasons to do it, though the practical (reaching the limits on their filtering) largely outweighed the ideological.

Movable Email

If email is important to you, you should really register your own domain name, so you have a permanent address. I got dague.net back in 1999 to create a permanent home for my identity. This has meant over they years the backend for dague.net has changed at least 5 times, including me hosting it myself for a large number of years.

My Requirements

  • Can host email on my own domain – as I’d be moving dague.net
  • Web UI – because sometimes I want to access my email via Chromebook
  • Good Search – because there are times I fall back to full text search to find things
  • IMAP – because most of the time I’ll be accessing via Thunderbird or Kaiten Mail
  • Good spam filtering
  • Good generic filtering on the server side – My daily mail volume is north of 1000 messages (40% spam), I need good filtering otherwise I drown

Fastmail.fm

I eventually landed on Fastmail.fm, who I’ve been watching for a lot of years. They are fairly priced ($40/yr), their company contributes back to the open source software they run their business on, and because they are actually an Australian company, you’ll get disclosure if some agency is accessing your accounts. They also give you a 60 day free trial, so you can do a slow migration over, and see if it will meet your needs.

Migration

One I was sure I was going to do this, I created my fastmail.fm account, and then pulled and configured imapsync to sync my existing gmail content over. I have a couple of GB of email, which means an imap sync takes a good 24 hours at this point. Imapsync is rerunable, so run it once, wait until it finishes, then run it a second time, and pick up the changes. Once it seems like you’ve basically closed the gaps between the two accounts, you can change MX records, and start getting email at the new service provider.

For safety the first thing I do once this has happened is build a forward rule from the new provider to the old one. Then if something goes horribly wrong, all my email remains in both locations for a while. A month later I’m still running that forward, though will be disconnecting it soon.

So far so good

The webmail for fastmail is really solid, honestly I like it better than gmail’s web ui, which has become incredibly cluttered over the years. This is just email, which is good. It also has a search facility which is on par with google’s. It’s also available as part of the IMAP protocol, which means real searching from Kaiten mail on Android.¬†Switching from GMail App to Kaiten Mail on my phone was about 10 minutes. And it means I can actually customize things I get alerted to, which gmail broken at some point. Thunderbird transition was simple.

I had gotten used to Raportive on gmail that would give me people’s pictures on their email. I found the Ldap Info Show extension on Thunderbird, which looks people up on various social networks, and gives you pictures they have public.

Lacking APIs

The one complaint I have with fastmail, is that it’s lacking APIs to handle your data. For instance, my filtering rules are complex. 342 lines of sieve and counting at this point. This is managed via a web form, but copy / paste on every change is something I’m not really into. I solved this by writing a python mechanize sync script so I can manage the rules locally, version controlled, then sync them up afterwards.

Address book has some issues, and I’ve not built a work around. The sieve rules they give you whitelist your address book as spam sources, so it’s something I’d like to keep in sync. However, without an API it’s not really worth it.

Overall: Very Good

Overall I’m very happy with the move. My biggest complaints are around the API issue, which I hope they correct in the future.