Fork me on GitHub

A Plan for 2017

Posted by Bjarni Rúnar on January 30, 2017

Happy Belated New Year!

Thank you for your ongoing support and encouragement in 2016. Although there was no "big news" from Mailpile in 2016, development continued and I was happy to see a whole bunch of new contributors submitting patches, reporting bugs and chatting with us on IRC. It's wonderful to have a community.

As any reader of this blog knows, my track record when it comes to predictions and planning, is absolutely abysmal. I'm sorry.

However, I still want 2017 to be the year Mailpile hits 1.0. The project will be 4 years old this summer and we need to make a release. Here is how I hope to accomplish that:

  1. Over the next few months, I will (with help from the community) chip away at the list of 1.0-blocking issues.

  2. Once (most of) the blocking issues are resolved, a release candidate will be tagged, packaged and made conveniently (apt-get install ...) available for Debian and Ubuntu Linux users.

  3. At this point, I will use some of the remaining Mailpile funds to hire specialized help to work on the Mac OS X and Windows integration and packaging efforts. With a couple of exceptions, I have been self-funding my Mailpile work since the beginning of 2015, so in spite of our failures to raise grant money, we do still have some cash in the bank (and our bitcoin wallets).

  4. Once a release candidate has been published, we will ask our community of international translators to finish their work and try and get some "QA" work done to ensure the translations are accurate.

  5. At this point, we will also encourage other Linux distributions to contribute build recipes so we can build packages for them as well.

  6. Feedback from early Linux adoptors will be used to squash bugs, polish the code and clean up the translations, eventually leading to a 1.0 release. Whether the 1.0 release will be simultaneous for all three platforms is undecided.

I think all, or most, of these things can happen within the next 11 months.

Especially with your help...

Too Cool for PGP

Posted by Bjarni Rúnar on 12 December, 2016

Some kids are just too cool for school.

And some security experts are too cool for OpenPGP.

It's almost become a rite of passage for security folks: work in the trenches, build a reputation, climb the ivory tower, write a detailed epiphany about why you've given up on PGP. Suggest we all buy an iPhone and use Signal, start giving people phone numbers instead of e-mail addresses...

Wait, what?

Please take a moment to go ask any young woman if she thinks giving random strangers her phone number will improve her security. I'll wait.


Of course, the experts are right about many things. OpenPGP is old and more recent tools with more modern designs have a lot going for them. But I still think they're mostly wrong.

The experts, by and large, have yet to offer any credible replacements for PGP. And when they suggest abandoning PGP, what they're really saying is we should give up on secure e-mail and just use something else. That doesn't fly. Many people have to use e-mail. E-mail is everywhere. Not improving the security of e-mail and instead expecting people to just use other tools (or go without), is the security elite proclaiming from their ivory tower: "Let them eat cake!"

Furthermore, if that "something else" also requires people use their phone number for everything... well, that's the messaging world's equivalent of the widely despised Facebook Real Name Policy. If you ever needed a clear example of why the lack of diversity (and empathy) in tech is a problem, there it is!

Compartmentalization, presenting different identities in different contexts, is a fundamental, necessary part of human behaviour. It's one of the basics. If you think taking that away and offering fancy crypto, forward secrecy, deniability instead is a win... well, I think your threat models need some work! You have failed and people will just keep on using insecure e-mail for their accounting, their work, their hobbies, their doctor visits and their interaction with local government. Because people know their needs better than you do.

But I digress.

The ridiculous phone number thing aside, I also take issue with the fact that when our opinionated experts do suggest replacements, the things they recommend are proprietary, centralized and controlled by for-profit companies. Some of them (mostly the underdogs) may be open source, but even the best of those use a centralized design and are hostile to federation. In pursuit of security and convenience (and, let's be honest, control, power and money), openness has been hung out to dry.

This is short-sighted at best.

These cool new apps may be secure today. But what about tomorrow? Odds are, they will be compromised by government mandate, blocked or shut down. Or just dead because messaging is a cut-throat business and the money runs out. Anyone remember ICQ? MSN? GChat? Sprinkling these new messaging apps in security pixie dust doesn't make them qualified to replace e-mail.

But what if I'm wrong? What if one of these businesses succeeds, e-mail dies and all our comms become dependent on proprietary protocols mediated by for-profit monopolies? Is that a problem?

Here, let me google that for you.

I really hope it doesn't happen.


Please, if you are at risk, if you have powerful adversaries, follow the advice of the cool kids. The experts are absolutely right when they say PGP is too confusing and messy today for most people to use safely. It takes training, practice and diligence.

So sure, get an iPhone if you can afford it. Use Signal or iMessage. Use Tor, carefully. For e-mail, create as many GMail accounts as you need to blend in with the crowd and not draw attention to yourself; their security team is the best in the world, let them protect you! Enable two-factor auth, use HTTPS.

But most importantly; if you can avoid digitizing incriminating information, do that. Rubber hose cryptanalysis is real and it's much easier to avoid creating data in the first place, than it is to keep it secure after the fact.

Mental Models and Deniability

A rule of thumb for creating usable software, is don't make me think.

What this means in practice, is software should match the mental models of its users as closely as possible. If it doesn't, users will inevitably make mistakes. If your tool is a security tool, those mistakes may compromise their security.

PGP in e-mail has failed this on many fronts. The lack of protection for message headers (the subject line) is one, as is pretty much anything to do with encryption keys (too much math). But it's not all bad! OpenPGP gets other things right, and actually corrects some of the things insecure e-mail gets wrong.

One of the most vexing things about e-mail, is people actually think e-mail is already secure. They just assume e-mail is like regular mail, in an opaque envelope that will prevent tampering and keep postal workers from reading it. Encryption and signatures bring e-mail closer to user expectations, which means if we can get it working smoothly, users won't have to think as much to make good security choices.

One thing people don't expect from e-mail, is deniability. Deniability means after a message has been delivered, it can no longer be strongly linked to the sender. It's like an anti-signature... which most sane people would consider a horrible misfeature in any communication system. Explicitly designing a system so people can disavow their statements and go back on their word? What is this, a system for assholes??

And yet, all the cool kids in the security world seem to want exactly that. They keep bringing up the lack of deniability (and forward secrecy) in PGP as if it were some sort of fatal flaw.

Why? Are security people all assholes? I don't think that's it.

I think they're quite enamoured with the elegant math, and really, really pissed off with certain Three Letter Agencies. There is good reason to believe major governments plan to, or already have been recording all our encrypted communications in the hope of being able to decrypt them later. Forward secrecy (deniability's more attractive twin sister) prevents that sort of thing. But OpenPGP doesn't need to provide forward secrecy to thwart mass surveillance. If we just use TLS (with the right ciphers) for SMTP, IMAP and web-mail then that does the job just fine.

So I agree forward secrecy in transit is a good thing. Let's do that!

Let's put our mail in secure envelopes, and let's also drive it from place to place in nice, secure vehicles. Users don't expect the cops to routinely stop the mailman and photocopy all the mail, so let's make sure that doesn't happen to e-mail either. Let the mental models be our guide.

But we don't need or want deniability. Deniability for individual messages is, quite simply, a horrible misfeature to be avoided. People already assume e-mail is on the record; trying to change that means going against their mental models and setting them up for failure in new and exciting ways. The fact that OpenPGP wasn't designed to empower assholes is a feature, not a bug.

(Yes, there are other arguments for forward secrecy and deniability. They are in my oh-so-humble opinion, mostly bunk. And this post is already too long...)

Making Progress

Anyway, like it or not, e-mail is important.

E-mail is the most successful open messaging standard we've got and OpenPGP is the best tech we have to secure our mail. OpenPGP may be dated and a bit clunky, but it's a hell of a lot better than nothing.

Folks like myself, implementors who are not cryptographers, have long been admonished to not invent our own crypto. Over and over again, we are told to use tried and tested solutions. OpenPGP is that. It may have baggage, it may not be perfect, but it is mature and it solves certain problems. Most of the flaws can be avoided and worked around. If the security community really wants us to use something else, you're going to have to step up and provide something a bit more tangible than rants on the Internet.

OpenPGP is also not standing still, OpenPGP is still developing. The community is well aware that the technology is flawed and needs work. An update to the standard is in the works and there are multiple projects working on improving both the security and usability side of things.

Mailpile is one such project, but we're in good company: PEP, LEAP, OpenKeychain for Android, Mailvelope, and more. Even Google and Yahoo are developing solutions based on OpenPGP. There's actually quite a lot going on!

As an industry, we should be supporting these efforts, not writing and promoting self indulgent posts on how we've given up and moved on.

Oh, and stay in school kids! It's worth it!

Protecting Your Local Data

Posted by Bjarni on November 23, 2016

Everyone understands physical security.

Starting in play-school, we learn that if we don't want the other kids to play with our favourite toys, then steps must be taken. Hopefully most of us also learned to share and play nice, but that's another matter...

Keeping physical objects safe is such a basic skill that it requires no explanation. We all know the importance of keeping our wallets or our phones safe, our jewellry, our keys. We may not do a great job all the time, some of us are absent minded and maybe we live in a safe place and don't worry too much - but we all understand the concept.

One of Mailpile's main goals is to bring this kind of intuitive security to our digital lives, to e-mail. If a physical device in your possession stores your mail, then you already know the basics of keeping your e-mail secure, private and safe.

So far, so good!

But what happens when things go wrong? What happens when your Mailpile machine gets lost or stolen? Or dropped on the floor?

The Risks

When something goes wrong with your Mailpile device, there are two bad things that are likely to happen:

  1. You lose access to your Mailpile data
  2. Someone who shouldn't gains access to your Mailpile data

The first case is more common; it can happen when a machine gets lost, gets stolen, gets damaged in fire or flood, or malfunctions of its own accord. Given enough time, it's almost inevitable.

The second case is much less common, but when it happens the consequences can be dire. If your e-mail contains very sensitive material (confidential documents, nude photos, incriminating evidence) your job, your relationships or even your life may be at risk.

So those are the problems. How about some solutions?

Avoiding Data Loss: Backups

Protecting you from losing access to your Mailpile is conceptually simple. You just need backups. That's all.

Famous last words...

Making backups available and easy is actually quite hard, because it is ultimately a physical problem. The data needs to be copied to another device that won't be lost or damaged by the calamity that made you need backups in the first place.

Any local physical solution ultimately relies on user education and user action; which means our less skilled or more absent-minded users will not end up with working backups. This is unacceptable.

A much better solution would be to use the network, automatically store Mailpile users' backups somewhere else. But where? If Mailpile were to directly provide an online backup solution, that would make us stewards of your data (a role we would like avoid) and would also cost us a bunch of money to do properly - money which the project simply does not have.

We also need to take care that the backups themselves don't become a security risk - we don't want to create a backup solution that accidentally grants unauthorized parties access to your mail!

Our current strategy is simple: we want to encrypt all your data and then upload it right back to one of the IMAP servers you already have access to.

Simple right? A mere matter of programming (this will take some time).

Avoiding Data Loss part II: Keys, Keys, Keys

We have a plan for backups! But what about the encryption keys?

If those are lost, all is lost.

If those are stolen, all is stolen.

Currently, Mailpile requires users log in to their Mailpile with a password (or passphrase). Internally, this passphrase unlocks a file encrypted using GnuPG, a file which contains the actual encryption key used for everything else.

If we trust GnuPG's security (which we do), it should be perfectly fine to store this along with all the other data. Right?

Unfortunately, no. The problem is, our users may not choose strong passwords. No matter how much we try to educate, some will end up using the same password for their Mailpile as they used for the IMAP server we're uploading the backups to. Since the operator of the IMAP server can access that password with relative ease, that would leave the backups exposed.

So this part of the plan requires more thought; the current strategy we are leaning towards, is to create an unguessable "recovery key", made up of 9-12 dictionary words. This recovery key will be used to encrypt the keys to the kingdom, before they too are uploaded to the cloud.

Once backups are enabled, the app will do the following:

  1. Ask the user to print out or write down the recovery key, and keep it safe.
  2. Ask the user to nominate a few trusted friends or family members, and e-mail each of them a part of the recovery key.

The first option is for people who don't have any friends. The second option is for everyone else!

When a user needs to restore their Mailpile from backups, they will either fetch the printout from the shoebox under the bed (or from the bank vault), or they will contact their friends asking for help. Either way, it's a relatively simple procedure.

We think... your feedback is most welcome!

There are of course lots of little details; the length of the recovery key, the number of trusted friends who receive key details versus the number of friends required to respond for successful recovery, the wording of the key fragment e-mails. And so on! But I think the basic idea is sound and the implementation should be able to adapt to the needs of both casual users and users with strong security requirements.

Incidentally, this procedure can also serve as a decentralized "password reset" feature for people who forget their Mailpile password. They just have to remember who their trusted friends were...

Keeping Your Data Private

The final piece of this puzzle, is keeping data private.

This is also conceptually simple: we encrypt everything!

Considering that strong cryptography is available "off the shelf" from multiple mature libraries these days, our job is to choose one of the industry-standard algorithms from one of the standard libraries, and make use of it.

The main constraints are performance and reliability. We would like the encryption to not slow things down too much, and we would like the encryption to not increase the odds of data loss.

Given those two constraints, our current preference is to use AES in CTR (Counter) mode, with a SHA256-based MAC to detect data corruption and/or tampering.

Since CTR mode localizes any damage (unlike AES-CBC which we were mistakenly using before), it can be argued that the encryption doesn't reduce reliability. Of course in practice this will depend on the what the app does when corruption is detected and how easy (and safe) it is to recover - but as long as the data is mostly intact, our recovery strategies can evolve and improve over time.

An alternative mode, GCM (Galois Counter Mode) would also be a good candidate, but it's currently not as widely available as CTR mode. We may switch to it in the future.

Another item for the long-term wishlist is use of error correction codes to automatically recover from minor corruption; potentially making Mailpile Encrypted Storage more resiliant than normal unencrypted files.

Current State

This is all well and good. But we're not there yet!

Here is a summary of the current state of things in Mailpile and what we expect for version 1.0:

  1. Automated backups are not implemented yet and probably will not be ready by version 1.0. We have however laid the groundwork by defining an on-disk storage format which can be easily uploaded to IMAP servers (since the format looks like e-mail).

  2. Key backups are not implemented, but are a goal for 1.0 due to their importance and the extreme risk of data loss if they are compromised. Join the discussion here.

  3. Local data encryption is implemented. We are in the process of switching encryption algorithms to AES-CTR.

Thanks for reading!

Search as a Core Feature

Posted by Bjarni Rúnar on October 26, 2016

One of the main differences between Mailpile and most other Free Software e-mail clients has to do with the approach we take to handling e-mail.

The first generation of e-mail clients focused on the e-mail itself and provided mailboxes or folders as places to store it. Organizing your e-mail meant moving it around, from one mailbox to another. This is how most desktop e-mail clients work today.

Mailpile's approach is different. Inspired by GMail, we decided to make search the central metaphor. Organizing mail in Mailpile then became a matter of labeling messages in such ways that they could easily be searched for; this is how Mailpile's tags work.

Tags are much more flexible and powerful than mailboxes; once the search engine has indexed all your mail it no longer matters in which mailbox or folder the mail is stored since you can access and organize any combination of messages using a mixture of tags and search terms. Searching a well designed index is actually faster (both for the human and the computer) than finding and opening the right mailbox.

So Mailpile is a search engine first and foremost. Most of the other features it has are built on top of that foundation.

What About Mailboxes?

This is all well and good.

But the fact remains that sometimes we need to open a mailbox and look inside; after all, that is where the mail is!

Mailpile has struggled with this from the beginning. Being built around a single search engine meant Mailpile couldn't really do much with the contents of a mailbox until it had all been processed and added to the search index.

This led to usability problems. If Mailpile was given a large number of mailboxes to process it could take quite some time before it got to the one the user was interested in. If the background indexing process had a problem, mail would just never appear. Users coming from traditional mail clients had expectations which Mailpile could not meet. And last but not least, it made troubleshooting very difficult because there were so many layers of code, each introducing potential bugs or delays, that an e-mail had to travel through before it appeared (or failed to appear) in the user interface.

Sometimes you just want to open up a mailbox and look inside, without having to add all the mail to your Mailpile.

Embracing Search

The solution I have found to this problem wasn't to stop treating search as a core feature. It was to embrace it and take it a step further: who says there should only be one way to search?

Many IMAP servers offer search features. We should be able to make use of them. Similarly, searching a raw mailbox is relatively straightforward - it may not be elegant or as fast as Mailpile's native index, but running "grep" or the equivalent very often gives useful results. It turns out there are many ways to search mailboxes that haven't been fully processed and added to the main Mailpile index.

So Mailpile now internally supports multiple search indexes. At the moment it only searches one at a time, and some search indexes are not very good at searching yet... but the code is elegant and clean, works well and has interesting potential for the future.

Maybe someday we'll have hybrid search, which searches both remote IMAP servers and the local index. Maybe someday we'll be able to pull in results from notmuch or some alternate index.

But for now, at least this approach makes it easy and quick for Mailpile to look inside raw mailboxes. That alone clears a major roadblock out of the way for a 1.0 release.

I am still mulling over how best to expose this in the user interface. At the very least, it will be the default behaviour when accessing mailboxes from the browsing tool. I may also make it accessible via the per-mailbox tags in the sidebar; doing so is likely to match user expectations better than using Mailpile's internal index. But I'm not entirely sure.

I'll be pushing this up for review once I've finished a few more test.

Older stuff

Some Tweets

The crypto in Mailpile depends on @GnuPG - please support their work if you can!
@MailpileTeam, Tue, 13 Jun 2017 17:12

Ever wondered what sort of work goes into writing a secure e-mail client? We discuss most things in detail on #mailpile on Freenode (IRC).
@MailpileTeam, Wed, 29 Mar 2017 11:03

Looking for a weekend project? Like e-mail attachments? We could use help with these issues:
@MailpileTeam, Fri, 24 Mar 2017 12:41

Our @HerraBRE is in Valencia until Wednesday to work on AutoCrypt and hopefully meet some #InternetFF attendees. Get in touch, say hi!
@MailpileTeam, Sat, 04 Mar 2017 11:15

Need a weekend project? Mailpile setup fails if the machine is low on disk space; it should be easy to fix:
@MailpileTeam, Fri, 03 Mar 2017 17:50 finally mirrors our tweets! We only display static tweet content (no widgets) so Twitter cannot track visits.
@MailpileTeam, Thu, 23 Feb 2017 11:20

Should we start tweeting links to Mailpile issues our community could help us fix? Or would that be spammy? What do you think?
@MailpileTeam, Wed, 22 Feb 2017 15:51

Mailpile 1.0 will implement Memory Hole, a standard for encrypting or signing e-mail headers. See thread:
@MailpileTeam, Wed, 15 Feb 2017 12:46

Many of the things Mailpile will do to protect your privacy depend on Tor. Support them if you can!
@MailpileTeam, Mon, 05 Dec 2016 21:18

Our goal: Make Mailpile good enough enough to be mentioned in guides like this. We'll get there eventually!
@MailpileTeam, Thu, 01 Dec 2016 17:34

Nice to see folks are hard at work preparing cool products based on Mailpile, even though we're not at 1.0 yet.
@MailpileTeam, Mon, 28 Nov 2016 17:43

Things may have been quiet lately, but our monthly contributor count is growing! It's great to have a community.
@MailpileTeam, Fri, 18 Nov 2016 18:42

Developing a secure e-mail client isn't without risks... ;-)
@MailpileTeam, Wed, 16 Nov 2016 20:19

We don't tweet as much as we should. But there are other ways to check out pulse. For example this one:
@MailpileTeam, Mon, 31 Oct 2016 18:28