Motivation

Freedom of speech and privacy are more and more threatened by governments and hostile corporations working against the public interest. In this season of death of liberty the minimum we can do to respond is making surveillance more difficult, by providing the general public with easy tools to use for private communication.

That is the p≡p Project’s mission. Current p≡p software is based on email, which the younger generation no longer use as its primary communication medium — to the chagrin of greybeards like me who still like to take the time to compose a well-crafted message.

The p≡p project is also working on something new and more chat-like which will make make casual users feel at home; the new software (also, of course, free software) will reuse most of the current existing p≡p code base; email was always intended to be just a “transport” among others¹, to carry our messages with automatic encryption and verification.
However, the new application will be some other day’s topic for me or for somebody else.

Today I am writing about email, embracing my grey-bearded grumpy self with its futile taste for writing with care and cultivating an impractically sentimental attachment to language. Email is my favourite communication mean, maybe after Usenet. As a hacker I must not be the only one.

p≡p for email before p≡p-mail-tool: p≡p for Thunderbird

Before p≡p-mail-tool the p≡p software for email working on GNU/Linux systems consisted in a Thunderbird addon. While the software is very easy to use (the usual Thunderbird, plus essentially only the coloured “privacy bar” showing the privacy status of each incoming or outgoing message), the addon hides some complexity: inserting encryption and verification everywhere interferes with the application logic in a deep way, to a degree that was not anticipated in the Thunderbird extension API. The addon itself is a set of JavaScript files communicating over HTTP with a local server holding the local persistent state and performing calling the cryptographic primitives.

Even with a few minor warts one cannot really complain of p≡p for Thunderbird: it works and is in fact easy to use.

One uses Thunderbird with the p≡p addon exactly like an ordinary Thunderbird, without losing compatibility with ordinary unencrypted email. If we receive a message from a new communication partner using p≡p, the system automatically imports the partner’s attached public key (TOFU: Trust On First Use). A communication partner not using p≡p and receiving a message from us will get an unencrypted email, with a sender-key attachment which is perfectly safe to ignore.

When a public key for a communication partner is known, it is automatically used² to encrypt outgoing messages: this is everything a user needs in normal circumstances.

p≡p for Thunderbird, composing a message to a communication partner to whom we can send a protected (encrypted and authenticated) message

p≡p for Thunderbird, displaying an incoming message from the same communication partner: again the message is yellow: encrypted and authenticated.

The coloured “privacy bar” shows the privacy status of an incoming or outgoing message: red for “under attack”, yellow for “protected”, green for “trusted”; and without any colour if the message is unprotected.

An unprotected incoming message with no colour, from a non-p≡p communication partner or the first exchange with a p≡p communication partner.

TOFU is vulnerable to a Man-in-the-Middle attack only inside a very narrow window: as long as the very first message between them is not compromised two communication partners can rely on protected (encrypted and authenticated) exchanges. This level of protection is already reasonable for most circumstances, but it is possible to do even better and close the initial attack window as well, as an optional measure:

Optionally one can contact the communication partner out-of-band, and check that her trustwords (a function of the fingerprints of both communication partners) match: if the users accept that the trustwords match, messages between them will be shown with a green bar.

For the unusual cases in which one needs to replace a lost or compromised key (typically after a disk crash or device destruction, loss or theft) there are mechanisms to manually reset keys; these are trivial to use, and key management is otherwise completely automatic.

Now, by adding that p≡p is also compatible with standard OpenPGP I should have mentioned most of its features.

Assessment

Jörg Knobloch, who wrote p≡p for Thunderbird and large parts of Thunderbird itself, is an expert whose work should not be overlooked; p≡p for Thunderbird is an achievement in itself.

I do not dislike Thunderbird: in fact I used it myself (actually first the old “Mozilla suite”, then Thunderbird) for about a decade, between 2002 or 2003 and early 2012. Its message search capabilities are excellent, and its “message filters” my second favourite client automatic classification system after Gnus’s “fancy mail splitting”.

For me the real problem with Thunderbird is that it is not a piece of software I love. It is not a set of functionalities running on top of my favourite editor, keyboard-controllable, extensible to no end, written in Lisp which is itself a thing of beauty. Since I love Gnus and to me email and editing are very important I will not accept a compromise.
Other hackers have preferences different from mine, but usually just as strong. We are demanding, particular: we want power and control.

Even with p≡p for Thunderbird being a sensible choice for the general public many hackers will not want to adopt it. I use p≡p for Thunderbird almost every day for testing the p≡p Engine, but Thunderbird it is not my production MUA.

My proposed solution: p≡p-mail-tool

I want users to be able to switch to p≡p while keeping their MUA, adapted as little as possible; ideally with no changes at all.

A good solution to this problem is to provide a standard interface to an MUA: the new software must be able to use existing standard protocols or formats, for example IMAP and SMTP: the user’s MUA will connect to the local host, possibly on non-standard ports, to send and receive ordinary unencrypted messages. All of the encryption, decryption and verification will happen automatically, out of the MUA’s view.

This turns out to be almost entirely feasible; for the single exception see Displaying the outgoing message rating below.

p≡p-mail-tool downloads incoming messages from the user’s “incoming nodes”: usually remote IMAP or POP servers, but possibly also mbox files, Maildir spools or something similar, decrypting messages as necessary and storing the decrypted versions into a local store. A good candidate for a local store is an IMAP server but a Maildir directory or an mbox file will also do, as long as the user’s MUA can handle the formats³.

When the MUA submits a message it will connect to the submitter side of p≡p-mail-tool over SMTP (or over a sendmail-compatible command-line interface); p≡p-mail-tool will automatically encrypt the message and then deliver it to the appropriate outgoing node: usually a remote SMTP server, but possibly another sendmail, or some spool file.

An example using a POP or IMAP incoming nodes, a Dovecot IMAP server as a local source, and SMTP servers as outgoing nodes. The connections inside the black box happen within the local host.

It is quite easy to have p≡p-mail-tool supporting multiple mail accounts, by receiving from multiple incoming nodes and submitting to multiple outgoing nodes: the specific outgoing node for a message can be chosen from the From address, via a regular expression. This account configuration is stored in p≡p-mail-tool’s configuration file; the actual MUA is simpler to handle, since it only needs to know about stores and submitters.

The following excerpt comes from p≡p-mail-tool’s sample configuration file. This imaginary user has three accounts quirky@the-fun-project.org, johndoe@imaginary.pep-project.org and john@johndoe.org.

incoming_rules = [
    # (INCOMING-NODE,
    #  (own-username, own-email-address),
    #  LOCAL-STORE),
    (('pop',   'pop3.the-fun-project.org',    None, 'quirky',  'PASSWORD'),
     ('Mr Quirky', 'quirky@the-fun-project.org'),
     ('dovecot-subprocess', '~/.pEp-dovecot', 1143, 'john',    'PASSWORD')),
    (('imaps', 'imaginary.pep-project.org',   None, 'johndoe', 'PASSWORD'),
     ('John Doe',  'johndoe@imaginary.pep-project.org'),
     ('dovecot-subprocess', '~/.pEp-dovecot', 1143, 'john',    'PASSWORD')),
    (('imaps', 'mail.johndoe.org',            None, 'john',    'PASSWORD'),
     ('John Doe',  'john@johndoe.org'),
     ('dovecot-subprocess', '~/.pEp-dovecot', 1143, 'john',    'PASSWORD')),
]

outgoing_rules = [
    # (regexp,
    #  OUTGOING-NODE),
    ('(johndoe|john|doe|quirky)@(the-fun-project\.org)',
     ('smtps', 'smtp.the-fun-project.org',    465,  'johndoe', 'PASSWORD')),
    ('(johndoe|john|doe)@imaginary\\.(pep\\.foundation|pep-project\\.org)',
     ('smtp', 'imaginary.pep-project.org',    587,  'johndoe', 'PASSWORD')),
    # The last server always matches, so the regular expression is
    # irrelevant and in fact ignored.
    ('.*',
     ('smtp', 'johndoe.org', 465, 'johndoe', 'PASSWORD')),
]

Early on I noticed the need for a fast and reliable local IMAP server, able to hold a high number of messages. Being a happy long-time user I chose Dovecot for my own tests; but soon I had the interesting realisation that I could have p≡p-mail-tool machine-generate the entire Dovecot configuration from p≡p-mail-tool’s configuration and entirely handle Dovecot as a subprocess, its complexity hidden from the user. This feature is used in the example above and I expect it to be frequently used in practice.

The configuration file has more details than shown here (see https://codeberg.org/pEp/pEp-mail-tool/src/branch/master/configuration-examples/dot-pEp-mail-tool.py.example) but p≡p-mail-tool ends up being easy to configure for its power. For example it is trivially possible to start multiple Dovecot instances, all started up automatically by p≡p-mail-tool, as separate sources, presenting as separate mailboxes to the MUA.

Privacy bar and colours with p≡p-mail-tool

If the MUA is unmodified, how can it show the privacy bar for an incoming message?

This turns out to be easy as long as the MUA can display arbitrary headers, and the kind of flexible MUAs we hackers like of course will let us do that. The solution, therefore, is for p≡p-mail-tool to add a non-standard header after decryption, displaying the message rating. The header looks like:

X-pEp-Rating: yellow (reliable 6)

Seen from Gnus this would be:

While this privacy display does not look as friendly as the one in Thunderbird above, it does contain the relevant information.

There remains another major unsolved problem: how can we display the colour of an outgoing message while it is being composed, since the message has not passed through p≡p-mail-tool yet?

Displaying the outgoing message rating

First of all, it is necessary to modify the MUA for this. In the case of Gnus we can add some message-mode hook to be called at the right time every time the message is edited, or periodically with a timer⁴.

The hook to call is an Emacs Lisp function, computing the rating. But where does the function get the information from?

The solution is having a command-line interface in p≡p-mail-tool.

p≡p-mail-tool can be invoked in the command line to compute, among the rest, the rating a message would have if sent; the message is passed through p≡p-mail-tool’s standard input; from Emacs’s point of view the message-mode hook invokes p≡p-mail-tool as a subprocess using the Lisp equivalent of the system libc function.

This help message documents the current command-line interface of p≡p-mail-tool, including the option --outgoing-rating-stdin I am discussing now.

[luca@moore ~/pep-src/pEp-mail-tool]$ ./pEp-mail-tool --help
usage: pEp-mail-tool [-h] [-v] [-s] [--conf FILE] [--no-op] [--wake-up]
                     [--exit] [--ping EMAIL EMAIL]
                     [--incoming-stdin EMAIL] [--outgoing-stdin]
                     [--outgoing-rating-stdin] [--trustwords-language LL]
                     [--trustwords-full EMAIL EMAIL]
                     [--trustwords-partial EMAIL EMAIL] [--trust EMAIL]
                     [--mistrust EMAIL] [--trust-reset EMAIL]
                     [--key-reset EMAIL] [--key-reset-all-own]

A command-line tool for using p≡p for mail without a special client.

options:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit
  -s, --server          run as (foreground) server [default: run as
                        client]
  --conf FILE           use FILE as a configuration file (only useful
                        when running as server); this can be given
                        multiple times. If never given use the hardwired
                        path ~/.pEp-mail-tool
  --no-op               do nothing, exiting with sucess as long as the
                        server is responding. This is intended to check
                        whether the server is up
  --wake-up             wake up every server thread that is currenly
                        waiting. This is useful to force the server to
                        fetch mail immediately, skipping the random wait
                        time for one iteration of the infinite loop
  --exit                kill the server
  --ping EMAIL EMAIL    send a Distribution.Ping message from the own
                        identity having the given first email address to
                        another identity having the second email address
  --incoming-stdin EMAIL
                        handle an incoming message from stdin, to be
                        received by an own identity with the given
                        address
  --outgoing-stdin      handle an outgoing message from stdin
  --outgoing-rating-stdin
                        print the rating of the outgoing message from
                        stdin (without encrypting or sending the message)
  --trustwords-language LL
                        use the given two-letter language code for
                        trustwords, instead of the server default. This
                        only makes sense combined with --trustwords-full
                        or --trustwords-partial
  --trustwords-full EMAIL EMAIL
                        print full trustwords (10 words or 160 bits) for
                        the identities having the two given email
                        addresses.
  --trustwords-partial EMAIL EMAIL
                        exactly like --trustwords-full , except that this
                        only prints partial trustwords (5 words or 80
                        bits)
  --trust EMAIL         make the default key of the given identity
                        trusted
  --mistrust EMAIL      make the default key of the given identity
                        mistrusted, and no longer use it as a default key
                        for any identity
  --trust-reset EMAIL   clear the trust bits for the default key of the
                        given identity
  --key-reset EMAIL     reset the default key for the identity with the
                        given address, be it own or non-own
  --key-reset-all-own   reset all own keys

Invoke with a basename containing the substring "sendmail" for an
alternative command-line syntax compatible with (a subset of) sendmail's.

Please report bugs by opening a ticket on
<https://codeberg.org/pEp/pEp-mail-tool/issues>
or by contacting the author Luca Saiu <https://ageinghacker.net>.

The rapid frequency of p≡p-mail-tool invocations, potentially once per key press, makes this use case performance-critical. In order to make this important feature efficient p≡p-mail-tool is itself divided into a client part and a server part, communicating with each other over a Unix-domain socket.

p≡p-mail-tool --outgoing-rating-stdin will just start up as a client performing a request to the server: the server, already initialised with p≡p-mail-tool’s configuration and with a thread pool ready, will reply quickly, on my machine within 40~50 ms.

As the output of p≡p-mail-tool --help above shows, one an use this command-line interface for other p≡p operations such as checking trustwords, or resetting compromised keys.

Thanks to p≡p-mail-tool’s command-line interface and p≡p-mail-tool’s Emacs Lisp hooks for Gnus we can now have a rating header automatically updated as the user composes a message:

The X-pEp-Rating header appears automatically and is updated as the user edits the message sender and recipients.

This would already suffice to cover the basic functionality but it took me relatively little effort to do better and use an Emacs overlay to display the header value in its intended colour:

Now the X-pEp-Rating header value is covered by an overlay making it look nicer — and in actual colour.

Gnus uses message-mode for messages being composed, and article-mode for messages (incoming or outgoing) being read. I reused the overlay code to decorate article-mode buffers as well:

This makes Gnus quite pleasant to use with p≡p. It is also possible to customize (in the GNU Emacs sense) some parameters in pEp-mail-tool.el.

Implementation

When I talked to Volker Birk maybe six months ago about my still very rough design he convinced me to write my tool in Python, because of the availability of Python wrappers for the p≡p primitives: we call that component the p≡p Python Adapter (https://codeberg.org/pEp/pEpPythonAdapter). Later when I actually started the plan turned out to be correct and using the p≡p Python Adapter, Python’s library and aiosmtpd it turned out to be indeed relatively easy to put together a first working version of p≡p-mail-tool. aiosmtpd remains the one component I do not master completely.

I have never hidden my personal dislike of Python. I used the language mostly guided by my preferences, by avoiding or wrapping the language features I wanted hidden. For example I have always loathed the threading.Thread class, obviously influenced by the terrible Java API without even the excuse of language constraints; and so my threads use a thin function wrapper on top of that class. I do not use inheritance at all, anywhere in the project. The code remains quite simple and understandable. p≡p-mail-tool is after all a very high-level application, quite different from my usual p≡p Engine work, and switching to it for a while has been a refreshing diversion.

After seeing the code already in an advanced state Volker remarked, correctly, that my way of using Python displayed my Lisper’s aesthetics.

More about p≡p-mail-tool

I gave a presentation about p≡p-mail-tool to my p≡p Foundation colleagues in late June, “p≡p-mail-tool: Using p≡p for email with ordinary mail clients — A status report with some personal opinions”. I am publishing a recording with permission.

Notice that this presentation has not been updated to cover the recent p≡p-mail-tool developments, and the software has already become considerably more general and powerful since the time of the presentation. The video should still be suitable as an introduction.

Please also notice that some of the naming conventions in p≡p-mail-tool changed since the presentation.

• Slides https://ageinghacker.net/talks/pEp-mail-tool-slides--saiu--2023-06-27.pdf (PDF, 1.4MiB)
• Video https://ageinghacker.net/talks/pEp-mail-tool--saiu--2023-06-27.webm (WebM, 322MiB, 1h18m)
• Abstract: I have written a new tool for using pEp for email, particularly with traditional clients different from Thunderbird, on Unix systems; it is not complete but already at a point where a random reasonably technical person could use it production. I normally use Gnus as my mail client – Gnus being one of the mail applications running on top of GNU Emacs.
  I wrote a small extension to Gnus for taking advantage of pEp-mail-tool, making the interface quite easy and friendly.
  Other mail programs can be adapted; but even without any adaptation the tool is useful.

I am willing to give an updated presentation over Jitsi, possibly in the evening, if sufficient interest arises.

About the software

p≡p-mail-tool is free software, released under the GNU Affero General Public Licence; written by me, Luca Saiu, for the p≡p Project.

The official p≡p-mail-tool repository is https://codeberg.org/pEp/pEp-mail-tool.

Trying p≡p-mail-tool

Building the p≡p stack from its source, while not difficult, is a long process because of the multiple repositories involved.

The scripts in https://codeberg.org/pEp/pEp-mail-tool/src/branch/master/build make it easier to build the entire pEp stack up to p≡p-mail-tool from its sources, as an ordinary non-root user; the script has been tested on several GNU/Linux distributions. At the time of writing:

• Debian 11;
• Fedora 38, “basic functionality”;
• Ubuntu 22.04.02-deskop, “minimal installation”;
• Mint 21.2 XFCE.

We are working to also have p≡p-mail-tool officially packaged by distributions such as Debian.

Acknowledgements

Volker Birk, the founder of the p≡p Project, is a Mutt user who like me had always delayed a personal switch to p≡p in order to keep his favourite MUA. Volker’s feedback has always been particularly encouraging.
Volker provided me with a lot of feedback about p≡p-mail-tool; in order to support his own use cases I understood ways of generalising the utility and making it remarkably more powerful. Volker spent quite some time teaching me and giving me suggestion about notation and syntax, many of which I had to begrudgingly concede to be pragmatically correct, no matter how much I disliked them: p≡p-mail-tool’s configuration file syntax will at some point allow some more idiomatically Pythonic structures as well, as alternatives to my positional nested-sequence Lisp-style notation.
Most of all, I am grateful to Volker for the freedom of design he ultimately left me.

Thanks to Nkls, now working on Mixnet and Onion-Routing support for p≡p, who was the first p≡p-mail-tool user and who helped me to discover and solve many problems.

Thanks to my other p≡p Foundation colleagues: Ariodante, Heck and Sva for their interest and feedback listening to my rehersal presentation; and all the others who gave me positive and encouraging feedback.

[2023-10-19 update: the public repositories and issue trackers have been moved to Codeberg. I updated every affected link in this article without marking each change.]

Tags: aiosmtpd, bash, command-line, dovecot, emacs, english, free-software, gnu, gnu-linux, gnus, hacking, imap, lisp, minimalism, myself, parentheses, pop, p≡p, privacy, python, script, smtp, software, surveillance, thunderbird, unix