Delegate, Automate, Collaborate, Pirate
Posted by Bjarni Rúnar on April 18, 2016
Avast! Be welcome to this latest irregular Mailpile status update!
In this episode, I will discuss:
- A Strategic Spin-Off Project
- Deletion and Tag Automation
- GnuPG Collaboration
- Piracy in Iceland
Progress towards a release has been very slow. This is entirely due to me being busy with other work - things that pay the bills, looking after my lovely baby daughter, buying an apartment and moving to Iceland. I'm swamped!
As I am often exhausted and pressed for time, I have had a hard time sticking to anything resembling a schedule and have basically indulged any vaguely productive impulses, rather than worry about roadmaps.
So if this doesn't look like progress towards a release, you're probably right. I've been very distracted. But it's progress all the same!
A Strategic Spin-Off Project
As mentioned before, Mailpile's desktop integration on Mac OS X and Windows is currently unacceptable and needs a lot of work.
We do have some code, however, and a rough design. In order to encourage people to help out (and maximize the utility of the code we've already written), I spun off the existing GUI code into a separate project: GUI-o-Matic.
This should both lower the barrier to entry and encourage contributions; You no longer need to check out all of Mailpile to hack on the GUI-o-Matic. And because it's a stand-alone utility, it's more likely that other projects will want to make use of it. We hope!
If you've ever wanted an easy way to add a cross-platform desktop graphical user-interface to your code (not just Python!), take a look: GUI-o-Matic is a bit like "dialog" for modern desktop environments.
Old farts will understand.
Deletion and Tag Automation
Did you know the current incarnation of Mailpile cannot actually delete e-mail? It's true. This was actually a deliberate, conservative choice to avoid losing valuable data during development. It was never meant to be permanent, but temporary hacks do tend to outstay their welcome...
In the context of shipping 1.0... well, a mail client isn't really a mail client if it cannot delete mail, is it? More pressingly, a tool which aims to safeguard user privacy has to support the most basic privacy feature of all: deleting unwanted data.
So I decided to (yet again) break the feature freeze and implement message deletion.
This had a knock-on effect. Mailpile's deletion strategy was supposed to be similar to that of other webmail: once things have sat in the Trash for a while, they get deleted automatically. Similarly, messages should automatically move from Spam to Trash after a while and blank drafts should get purged and deleted.
So Mailpile needed a way to a) detect messages had been untouched for a period of time and b) a way to trigger actions once a) was satisfied for a message carrying a particular tag.
So now Mailpile has exactly that!
The search-engine is used to keep a record of when a message tags were last modified, and each tag now has an automation section which specifies a number of days and an action to perform. A few times a day, Mailpile will search for idle messages in tags with automation enabled and either retag or delete the matching messages.
While I was implementing the configuration interface for this, I also added an option to enable statistical auto-tagging for any tag, as described in A Plan for Spam ... and Bacon!, and exposed a few more of the technical tag settings in the Tag settings editor. All features that already existed, but weren't really accessible.
So there we go, tags now have automation and you don't need any command-line black magic to create your own statistical tagging or time-based workflows.
These capability are now available to all tags, including user-created ones. Some of the potential use-cases include:
- Deleting Trash after a while
- Moving Spam to Trash
- Moving untouched blank drafts to Trash
- Creating statistical categories for promotions or paperwork
- Creating a "Postponed" tag which hides mail from view for a few days
Now we just need an auto-responder and Mailpile will be able to automatically recognize and reply to tech support requests that have been unanswered for more than a week...
(In the process I also fixed bugs in the bayesian auto-trainer, the periodic scheduler that triggers it and the tag editing tools the UI - proper release work after all!)
I write this, sitting on a train back home from London.
I was in London today to meet with Neal of the GnuPG project. We discussed how the projects could collaborate more closely in the future and some of the difficulties Mailpile has had integrating with GnuPG.
It was an excellent meeting and I'm optimistic that once GnuPG 2.1 (or 2.2) becomes widespread, Mailpile will be able to make full use of it without any horrible hacks.
Conversations will continue!
Piracy in Iceland
Finally, some bad news.
Iceland's government is broken and I feel an obligation to help fix it. I will be dedicating some time this summer to helping the local Pirate Party prepare for our next elections. Mostly I'll be working behind the scenes on internal party tools, but this inevitably means I will continue being distracted from Mailpile work. But don't worry, I'm not running for a seat in parliament. ;-)
If you can help out in some way to help pick up the slack, please get in touch on #mailpile on Freenode.
That's it for now, thanks for reading.
Time to pack some boxes and move to Iceland!
Python SSL Woes
Posted by Bjarni Rúnar on March 24, 2016
There's not much to say about progress this week; things have been slow, but most of my hacking time has been spent getting frustrated by SSL and Python.
So following in our Grand Tradition of biting the hand that feeds us and publicly complaining about libraries and tools we rely upon, I am going to write a few words about the state of SSL/TLS support in Python.
This rant was (sort of) requested by Christian, one of the maintainers of the Python SSL module. Hi Christian, thanks for all your hard work! ;-)
Why We Care
SSL (or TLS) is the technology used to secure how Mailpile communicates with online mail servers, and is also used when downloading data from the web, whether for key discovery or things like user avatars and icons.
The e-mail world is a bit different from the web, in that the use of encryption is somewhat less mature. Self-signed certificates for mail servers are more common than on the web and use of encryption for e-mail in transit tends to be more opportunistic, so the "standard" SSL key management strategies employed by web-servers and web-browsers don't really apply.
The current iteration of the Mailpile code-base doesn't validate SSL certificate at all when connecting to SMTP or IMAP servers; this means any motivated attacker can easily perform a man-in-the-middle attack to listen in on, or modify, sent or downloaded e-mail.
Fixing this requires validating the server certificates somehow; for those that have a certificate issued by one of the known certificate authorities, this is largely the same process as on the web. However, a large number of e-mail servers use self-signed certificates, particularly within the enthusiast community which is a significant part of the people we expect to adopt Mailpile. So we need to handle those certificates gracefully as well.
While working on this, I yet again encountered limitations and bugs in Python's standard ssl code.
Issue 1: Python 2.6, 2.7.3, ...
It is incredibly frustrating that Python 2.7.9 was the first release of Python that had semi-functional SSL support in the standard library. Previous versions were almost completely insecure and did not expose the API hooks required to improve the situation.
Since I'm pretty sure nobody in the Python community has a time machine, there's not much the Python community can do about this. All we can do is thank the people that are working to fix things today - and get those fixes used!
On that note: distributions that are shipping Python 2.7 but refuse to issue a security fix to upgrade to the latest point release are failing to protect their users. At least back-port the affected modules, for crying out loud!
In a moment of drama, I tweeted the following:
The more I use Python's ssl module, the more obvious it is that nobody ever used it for anything serious. Even the 2.7/3.x code just fails.
Those are fighting words and I'm sorry if they offended anyone. Christian responded with a very level-headed request for details, which in turn prompted this post.
The frustration I expressed there, was to a large degree related to Issue 1 above, but the comment on 2.7 and 3.x needs explaining. I'll talk about the big picture first, with critiques of individual methods further down.
This should by no means be considered an exhaustive review of the Python ssl module; but I hope at least some of my suggestions are actionable and constructive.
One caveat: I haven't checked the Python 2.7.11 code, this critique is based on reading the sources from 2.7.10 as installed by my distro's package manager. Some of these issues may have been fixed, which would bring me back to Issue 1.
Issue 2: import ssl, poplib, smtplib, imaplib
I am going to charitably assume that the ssl backport in 2.7.9 took care to preserve compatibility and that is why imaplib, poplib and smtplib (all used by Mailpile) still make insecure SSLv3 connections.
Unless I am reading the code completely wrong, the more secure new "default SSL contexts" are not actually used (see below for more details). This is fixed in Python 3.x.
Fair enough, backwards compatibility is important! When forced to choose between fixing a security hole and suddenly breaking apps that used to work, it's not obvious that security should always come first.
Unless this was a mistake?
Either way, the backported ssl module provides no mechanism for changing this behaviour, short of monkey-patching. That seems like a pretty serious oversight, and one easily corrected with a global module variable or two.
More generally, the ssl module would benefit greatly from an API that would allow an app to specify default behaviours and callbacks or hooks for "interesting events"; allow more control and more introspection without forcing all the other libraries to complicate their APIs.
Issue 3: Certificate Management
An e-mail client needs to be able to manage SSL certificates, including self-signed ones.
Showing my ignorance here, I couldn't figure out how to do this with the Python ssl module until I got some helpful replies to my angry tweet. Even then, the APIs I will need to work with if I stick with the standard python ssl module are very awkward.
I think it would be time very well spent to improve both the documentation and the APIs (see below) for this use-case.
Googling for Python ssl examples will in the vast majority of cases show people disabling certificate verification entirely. Surely that is not what we want people to do! The official API documentation is the most effective place to combat these bad examples.
These are random warts in the ssl API. Some major, some minor.
A typo: the method "fetch_server_certificate" does not exist.
If asked to validate, this method may throw an exception instead of returning the certificate. This is unhelpful.
The data returned by this method is a binary blob; the library does not provide any utilities for decoding the contents.
This exception is raised when a certificate fails to validate. However, it doesn't tell you anything about the certificate itself.
Including the certificate as an attribute of the exception would allow for more nuanced and meaningful error handling.
Details as to why the validation failed would be nice too.
As far as I can tell, this is the only method provided which parses an SSL certificate, making any certificate handling code reliant on the network. This is bad, but isn't the only problem here.
Aside from crypto geeks, few consumers of this API are likely to care about the details of a validated certificate. It's trusted, let's get on with things! It's the certificate that fails to validate that you need to know more about, so you can tell the user something useful.
Unfortunately, this method completely refuses to return structured information about a certificate unless it has been validated first. And of course, if validation is enabled then an exception will be thrown before this method can be called.
It appears the workaround is to first use
ssl.get_server_certificate and then provide the certificate as
will return actual information.
This is both non-obvious and quite convoluted.
This method should always return structured data when requested, but backwards compatibility concerns will probably preclude that. :-(
Either way, an standalone method that just parses certificates would be very nice to have.
This method confused me.
At first I thought this method would be used by various Python libraries that support the SSL protocol... and wanted to figure out how to customize the output of this so my app would have control over how SSL was done.
Turns out I was completely wrong on all counts.
I read the ssl library source and found that the stdlib modules are
expected to use the undocumented variable
maps to the undocumented method named
Except for http.client, which should use
which maps to
ssl.create_default_context after all...
Feeling somewhat confused, I read the Python 2.7.10 sources for imaplib,
smtplib and poplib. Turns out none of them use SSL contexts at all! They
all just call
ssl.wrap_socket, which in turn uses none of the above
Argh. Okay. :-(
It looks like those comments were just back-ported from 3.x, without the libraries themselves getting updated. It's confusing, but the main take-away is that the backport appears to have failed to secure any of the standard libraries except for httplib (and ftplib).
I raise this issue to illustrate how hard it is for an app developer to truly understand how SSL/TLS is being used within their app, let alone have control over it. It's frustrating, to say the least.
My original wish, to be able to control the default context returned by this method, would be a great feature request if this method were actually used! It's not. However, I think the feature request still makes sense - it's just more work!
What is Mailpile Doing About it?
- Talking to the Python devs (complaining)
- Planning to publish a TOFU-for-TLS module
- Planning to publish our monkey-patching magic
Step 1 is pretty much done; I submitted an early draft of this blog post to the current maintainers of the Python ssl module. In spite of the fact that it contained an extended and distracting rant about the SSL community in general (which I in the end decided not to post), they responded well to the technical points and gave constructive feedback.
The main take-away was that apps like Mailpile, that need control over how SSL connections are made and want to implement non-standard (non-web) certificate verification models, are not well served by the standard Python modules. So we'll be using pyOpenSSL instead and patching or subclassing the protocol libraries.
The TOFU code and policy management logic will be released separately from Mailpile and under a liberal license, so the community can benefit or point and laugh, whichever seems more appropriate. I don't know whether this will happen before or after 1.0.
Stay tuned. :-)
Status Update and PyCon SK report
Posted by Bjarni on March 16, 2016
Apologies for how quiet this blog has been lately; the last couple of months I have been very busy with fatherhood, preparing to move countries and making sure my other job gets done.
I do expect development on Mailpile to pick up again over the next few weeks and this blog will start getting regular updates again as well. There will be a lull again in the latter half of April, when my little family and I move back to Iceland.
Release Candidate Progress
During the first few weeks of 2016, quite a lot of progress was made on the release. My focus has been to first complete the tasks most likely to generate new text in the interface, so that our translation teams would have enough time to do their jobs.
- We are now very close to having working Debian packages that comply with offcial Debian policies, thanks to the heroic efforts of Alexandre Viau. This means Mailpile 1.0 should be in Debian-testing almost from day one!
- The most important privacy and security settings are now explained and can be configured by the user. This includes Tor support and expressing preferences on what is encrypted and whether external services like Gravatar are used or not.
- Improvements were made to the Event Log and a network activity viewer was exposed to facilitate troubleshooting.
- Jack Dodds contributed support for parsing stand-alone PGP encrypted and signed attachments.
- Many bugs got fixed.
Some of this work may have slightly circumvented the feature-freeze, but these are important enough features that I'm not too fussed about it.
I think I am almost done with the strings work and aim to declare a string freeze this week or next. The next task will then be bug hunting, putting the last touches on our Debian package and set up a build-bot to autogenerate packages.
Last weekend I was happy to attend Slovakia's first PyCon.
I had been invited to speak about some of the challenges involved in making Free Software development a full time job; one of the stories I told was of course the story of Mailpile so far. The talk was well received and I had many excellent conversations afterwards.
I was very impressed with how professional the conference was, considering it was their first ever PyCon. The organizers should be proud of themselves!
As usual, I made sure to perform some usability tests, answer questions, demonstrate Mailpile and help people set up the app. Our getting-started-on-linux wiki page now mentions Fedora! And of course I gave away lots of stickers.
Fixes suggested by the usability tests are already in Github.
Happy Frozen New Year!
Posted by Bjarni on January 1, 2016
Happy New Year everyone, and thank you for all the support and good times in 2015!
This fall, we announced that we would try and get Mailpile 1.0 for Linux released in 2015. We're not quite there, but we're so close we can smell it...
Mailpile is now in a state of Feature Freeze, which means the only development expected until 1.0 is bugfixes, "sweeping things under the rug" (hiding features if they have bugs that need more than 1-2 hours of work to fix), packaging work, translations and documentation.
How long we stay in this state depends on how many showstopper bugs we find, which is impossible to predict. But the work I know needs to be done should be doable in January.
Progress is being made!
New Year News
Some random news and updates unrelated to the Big Freeze:
Hackathon - thanks!
We'd like to thank everyone who took part in our hackathon at the 32C3 conference in Hamburg last week!
In particular, thanks to Daniel for organizing, and the Chaos Computer Club for organizing on such an amazing event - my wife and I followed along from home using the live streams.
If you would like to organize a Mailpile hackathon at your local LUG, get in touch - we're happy to mail out swag kits (T-shirts and stickers) to folks who are willing to help expand our community.
Many people have asked for an ETA for Mac support. The short answer is: we don't know. We don't know because a) we're terrible at plans and deadlines (sorry!) and b) nobody has volunteered to help yet.
Unless someone offers to help, Mac work waits until I have time after the 1.0 for Linux. This means work will begin in February at the earliest.
CloudFleet on IndieGoGo
We have a vested interest: CloudFleet have been one of the most active supporters of Mailpile so far, both contributing cash and code; the Apache integration I announced in the last post was to a large degree based on their work. CloudFleet are also likely to be the first consumer-focused product to bring Mailpile to a general audience. So give them your support if you can!
- Merry Christmas, Hello 32c3!
- Why so quiet?
- UI Updates, OTF news
- Fall FAQs
- PyCon UK and Mailpile Mel
- Goals and Releases
- Quiet days; new people
- Thank you!
- On Crowdfunding and Burnout
- Github Issue Cleanup Frenzy
- Vacation over; Back to work!
- Mailpile Beta III: WYSIWYG
- Beta III preparations
- License: The Python's Tongue
- Fast Startup and Key Discovery
- The DCSS Conference in Cardiff
- Licensing: Your Feedback So Far
- Multiple PGP Key Support!
- Roadmap Reality Check
- An AMA with the Localization Lab
- Digging for Data
- Site updates and community roadmap launched
- Choosing a License for Mailpile 1.0
- Last week: code pushed, Cloudfleet, key discovery
- A Roadmap to Version 1.0
- Last week: Roadmap, Memory Hole, refactoring
- The First OpenPGP E-mail Summit
- Back to Work!
- Beta Rejected!
- More thoughts on working with GnuPG
- Mailpile Beta II - the 4096 bit release
- To PGP/MIME or not to PGP/MIME
- Some thoughts on working with GnuPG
- One Year Later: Mailpile Beta
- Our Upcoming Beta Release: Part II
- Our Upcoming Beta Release
- Mailpile Alpha II - The Dogfood Edition
- Where is the Community Site?
- Mailpile Workshop in London
- Development, Perks, and Alpha (IGG Update #8)
- Alpha Release: Shipping Bits and Atoms
- A Plan For Spam ... and Bacon!
- A Pound of Security
- Speaking Your Language
- Perks? What perks? (IGG Update #7)
- DarkMail and Secure Protocols
- The Month of Dog Fooding
- Thank you! (IGG Update #6)
- A Rough Budget and Alpha Roadmap
- Surveillance And Centralization (video)
- The Home Stretch (IGG Update #5)
- PayPal News (IGG Update #4)
- PayPal Freezes Campaign Funds
- PayPal News (IGG Update #3)
- Fonts and Copyright Licenses
- We are funded! (IGG update #2)
- Turning Money Into Code
- Digging Through the Details
- Our first week (IGG update #1)
- Designing Security (video)
- Mailpile Launched