The Simple Mail Transfer Protocol

(This was written one day after @1606…)

…I should note that SSH isn’t quite a protocol you can talk to. Even though SSH greets us with text, the rest of the key exchange happens in binary (I think… I am not familiar with the SSH protocol itself at all.) But you know what is plain text? Besides Telnet because I don’t have a good telnet server in mind (except of course towel.blinkenlights.nl). And HTTP… they’re gonna teach that and whatever they teach is now boring :) FTP nobody likes them anymore (well not some people screaming to get rid of “unsecure” protocols from the internet). Oh but how can I forget SMTP, the Simple Mail Transfer Protocol. The protocol that hearkens back to the days of ARPANET, that co-existed with USENET (that’s NNTP), yet while that fell into obscurity, SMTP by some miraculous means survives… that which we call email today, still runs on one of the oldest protocols of the Internet. Its amicability still eager to shake your hands, despite incubated in a time where no fancy graphics existed; just lines and lines of text on an 80-column screen (somehow, uh. I’m not old, so naturally I don’t know why. I do find it somewhat ironic that programs developed from such a time are more human than those today, despite people having much less to express humanity with.)

But hey! You gotta admit this stylesheet is pretty dope. Thanks Pandoc and Apostrophe. :)

* * *

Actually, I am feeling particularly generous today. Since you found telnet interesting, let’s look at how to send an email! (Something I think every CS student should know, but I’m too impatient to wait for the 5XX course to come. :)

Fun fact, the mail relay protocol fondly known as SMTP since the days of ARPANET can be found as the sixth most frequently visited TCP port, i.e., just below ssh:

$ curl -sfLo - https://raw.githubusercontent.com/nmap/nmap/refs/heads/master/nmap-services | grep -v '^#' | sed 's/ *#.*$//' | sort -srnk 3 | grep \/tcp | head -5 | column -t
http    80/tcp   0.484143
telnet  23/tcp   0.221265
https   443/tcp  0.208669
ftp     21/tcp   0.197667
ssh     22/tcp   0.182286
smtp    25/tcp   0.131314

Sadly, most (if not all) internet-facing servers today ban port 25 because of spam. But… not on the internal network of CSL machines. If you try to connect to localhost:25 like I did accidentally one day, you will find…

$ telnet localhost 25
Trying 127.0.0.1...
Connected to localhost.cs.wisc.edu.
Escape character is '^]'.
220 snares-06.cs.wisc.edu ESMTP Postfix (Ubuntu)

… that it listens. So, as I promised. Here is how to send an email.

Sending email in CSL (telnet)

For the example, I will represent the CSL username/$USER as bbadger (that seems appropriate I guess); replace it with your own.

Type telnet localhost 25. (you can use nc(1) too, but shhh… I’ll pretend I don’t know that ;)
Greet it with a Extended HeLlO (EHLO)!
```
EHLO localhost
> 250-snares-06.cs.wisc.edu
> 250-PIPELINING
> 250-SIZE 24576000
> 250-VRFY
> 250-ETRN
> 250-ENHANCEDSTATUSCODES
> 250-8BITMIME
> 250-DSN
> 250 CHUNKING
```
This seems optional, but some mail servers expect you to greet it at least once. Plus, I think it’s more polite this way.

Notation: > indicates the SMTP server’s output. You won’t see them appear literally in your terminal, but you will know because you didn’t type them.
Tell it who you are and which mailboxes you want to send to. We do this with the MAIL and RCPT verbs. (Normally, you’d want to authenticate with SASL first, but in this case we don’t because in a sense being logged in already authenticates you (?) which I think is pretty nice.)
```
MAIL FROM:<bbadger@cs.wisc.edu> BODY=8BITMIME
> 250 2.1.0 Ok
RCPT TO:<alice@example.org>
> 250 2.1.5 Ok
RCPT TO:<bob@example.org>
> 250 2.1.5 Ok
```
Of course, replace those example mailboxes with real email addresses; and only include as many recipients (RCPT) as you want to send to.

Note: This won’t put the email you are about to write into your mail server. If you want to keep a “receipt” of this sent email, a quick and dirty trick I found is to mail to yourself:
```
RCPT TO:<bbadger@cs.wisc.edu>
> 250 2.1.5 Ok
```
A mail client (MUA) would typically directly store the message with POP/IMAP, but I won’t be covering that. (FWIW, I know IMAP is a lot more painful to type by hand than SMTP. :)
Here’s my favorite part: we get to write the actual email message, using the DATA verb. Here’s the syntax:
```
DATA
<line 1>
<line 2>
...
<line n>
.
```
It’s a lot like writing a heredoc, except you must always terminate with a period (or full-stop). And once you are gone, your message is queued… and hopefully within seconds your email is delivered to the mailboxes you specified.

That is, you can still quit now and nothing will send! But once you end the DATA command with that final period… we can only hope you entered your recipients correctly.
Assuming you have entered DATA and the server responds with “Go ahead; tell me when it ends with a period on its own line.” Now, there are a couple more things we would otherwise need to dutifully prepare (namely the Date and Message-ID headers, which are both required headers), but I’ll cut us some slack. Here is the full thing, with a bare minimum header:
```
DATA
> 354 End data with <CR><LF>.<CR><LF>
From: "Bucky Badger" <bbadger@cs.wisc.edu>
To: alice@example.org, bob@example.org
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 8bit
Subject: Come up with your own!

Write your own message here!  Something like:
"I am writing this email with my bare hands!
No Outlook, no Gmail, not even a proper MUA...
And no HTML!  This is plain text email talking!"
Unicode, non-Latin characters like 嗨 or 😈 OK.
And once you are done, end with a period...
.
> 250 2.0.0 Ok: queued as 0490060180
```
Change the From and To header however you like; it literally doesn’t matter. Do try to match the ones you entered in Step 3 though. (This is how email spoofing happens, by the way; there is nothing “wrong” with forging these From/To headers. You’d really have to count on the sender being decent enough to not lie about who they are.)

The three lines in the middle (from MIME to 8bit) are uh… magic. Don’t touch them for now. :) Essentially they’re there to tell the recipient on the other end
1. the following is text, not binary garbage (and I hope your terminal encodes characters in UTF-8, so I tell SMTP that too…) and (2) the following is not encoded a second time for transfer; treat everything I write exactly the way I type them. (The historical reason is, simply speaking, that SMTP ran on very limited technology. We can do this thanks to the 8BITMIME capability we saw earlier when greeting the server; in the past, SMTP simply cannot handle non-ASCII characters, and CTE was invented to precisely get around that.)
The blank line between the headers and body is crucial. Just… don’t forget it, and you’ll be fine for the most part.
Don’t forget to thank the server and say goodbye to it:
```
THANK YOU
> 502 5.5.2 Error: command not recognized
QUIT
> 221 2.0.0 Bye
```
Okay… that might not be a command, so don’t really type that. Still, we can show some gratitude internally though. After all, it sent a whole email for you!

Sending email elsewhere (openssl s_client)

If you are not near a CSL machine, don’t worry — you can still send emails through an Internet-facing SMTP server. Though there will be two additional steps:

Instead of telnet localhost 25, we’d have to connect over TLS. Luckily, openssl-s_client(1) is just the right tool for us!

$ openssl s_client -nocommands -quiet -crlf -starttls smtp -connect smtp-auth.cs.wisc.edu:587
> Connecting to 128.105.6.3
> depth=2 C=US, ST=New Jersey, L=Jersey City, O=The USERTRUST Network, CN=USERTrust RSA Certification Authority
> verify return:1
> depth=1 C=US, O=Internet2, CN=InCommon RSA Server CA 2
> verify return:1
> depth=0 C=US, ST=Wisconsin, O=University of Wisconsin-Madison, CN=cs.wisc.edu
> verify return:1
> 250 CHUNKING
EHLO localhost
> 250-smtp-auth-01.cs.wisc.edu
> 250-PIPELINING
> 250-SIZE 24576000
> 250-VRFY
> 250-ETRN
> 250-AUTH PLAIN LOGIN
> 250-AUTH=PLAIN LOGIN
> 250-ENHANCEDSTATUSCODES
> 250-8BITMIME
> 250-DSN
> 250 CHUNKING

And now we are connected just like before.

Though… we still have a bit of work to do. We need to authenticate with SASL. Note that the mail server responded back with “AUTH PLAIN LOGIN”. We’ll use the PLAIN mechanism. Now is a good time to open another terminal, or temporarily suspend this process using Ctrl-Z. Because we need to manually encode the credentials.

We’ll need two things: the username and password, as you would expect. According to this guide, the username is your bare CSL username (without at-sign or the domain!) and the password is the email token you must request here.

Let’s assume your username is bbadger and your token is 123456789ABCDEFGHIJ. Then this would be your credentials:
```
$ printf '\0%s\0%s' bbadger 123456789ABCDEFGHIJ | base64
AGJiYWRnZXIAMTIzNDU2Nzg5QUJDREVGR0hJSg==
```
which you should be able to use to authenticate with AUTH PLAIN:
```
AUTH PLAIN
> 334
AGJiYWRnZXIAMTIzNDU2Nzg5QUJDREVGR0hJSg==
> 235 2.7.0 Authentication successful
```
I am guessing that AUTH LOGIN would work the same way; I’m not too sure what the difference is but I usually stick to AUTH PLAIN.

Now you can continue with the same thing!

Bonus: Net::SMTP!

If you are wondering how I send emails, well do I have a story to tell you. You see, there was a time when openssl s_client was the only thing I could reliably send emails in a “raw” way. It’s the exact same process I showed you above, except I would copy my pre-written email with pbpaste at the DATA step.

Python’s smtplib wasn’t satisfactory to me (for reasons I don’t fully understand and am probably not justified to say), and even though I had re-written the algorithm for encoding quoted-printable CTE in Python… the lack of an SMTP client was still jarringly problematic.

And of course: as tempting as it sounds to write a Tcl expect script to interact with SMTP, there are way too many states to handle. Like what if the connection times out? What if my password didn’t work? What if the server freaks out, and returns some code other than OK? There was no way I was going to write an SMTP client from scratch just to automate this.

(Actually, there was something else I forgot to mention: Martin Lambers’ msmtp. I have trouble remembering the arguments, but that wasn’t a real problem. I guess I just enjoy getting to the protocol level more; msmtp(1) is pretty great, actually! (For those of you who don’t know, think of msmtp(1) as a drop-in replacement for sendmail(8). For those of you who don’t know sendmail(8), uh…. don’t worry about it. Just treat msmtp as its own thing I guess :))

But I was willing to revisit an old language known for its I/O capabilities… and ooh boy did I fall in love with it.

Here’s what my current directory structure looks like.

.
├── append_index.py -> ../../../2024/10/24/append_index.py
├── compose.pl      -> ../../../2025/03/23/compose.pl
├── create_index.py -> ../../../2024/10/24/create_index.py
├── queue.pl        -> ../../../2025/03/23/queue.pl
├── sendmail.sh     -> ../../../2025/03/23/sendmail.sh
├── test
├── test.conf
└── test.d
    ├── date
    ├── mid
    └── text

The culmination is this sendmail.sh script:

$ cat sendmail.sh
#!/bin/sh
set -e
if [ $# -ne 1 ]
then
    echo "usage: $0 name" >&2
    exit 1
fi
name=$1
shift
./compose.pl "$name"
for cp in "$name.d"/compose/*; do
    if [ -x "$cp" ]; then
        "$cp" "$name"
    fi
done
compose.py -y "$name.conf" -o "$name"
b="$(wc -c "$name" | awk '{print $1}')"
echo "Composed $b byte$([ "$b" -eq 1 ] || echo s) in '$name'"
printf "Would you to preview? (y/[n]) " ; read -r ans
if [ "$ans" = y ] || [ "$ans" = Y ]; then
    if vim +'set filetype=mail' -R "$name"; then
        :
    else
        ret=$?
        echo "Abort! Vim exited with code $ret" >&2
        exit $ret
    fi
fi
exec ./queue.pl "$name"

… which you can see is very short, but calls a lot of other things. It starts at compose.pl, which generates time-specific RFC headers (namely Date and Message-ID):

$ cat compose.pl
#!/usr/bin/env perl
use v5.36;
use autodie qw(:all);

use POSIX qw(locale_h);
use File::Spec;

my $name = shift;
my $time = time;

defined $name   or die "usage: $0 name\n";
-d "$name.d"    or die "missing $name.d; stopped";
-r "$name.conf" or die "missing $name.conf; stopped";

sub the {
    my ($file) = @_;
    File::Spec->catfile("$name.d", $file);
}

sub gets {
    my ($file) = @_;
    open my $fh, '<', $file;
    my $text = do { local $/; <$fh>; };
    close $fh;
    $text;
}

sub puts {
    my ($text, $file) = @_;
    open my $fh, '>', $file;
    print $fh $text;
    close $fh;
    print "Wrote '$text' to '$file'\n";
    $text;
}

sub now {
    setlocale(LC_TIME, "C");
    POSIX::strftime("%a, %d %b %Y %H:%M:%S %z", localtime $time);
}

sub mid {
    my @CHARS = ('A'..'F', 'a'..'f', '0'..'9');
    my $noise = join '', map {; $CHARS[rand @CHARS] } (0 .. 12);
    my $epoch = POSIX::strftime("%s", localtime $time);
    my $procid = $$;

    my $hostname = -r the ('host')
        ? gets the ('host')
        : 'e.rapidcow.org';
    my $lcpart = join '.', $epoch, $procid, $noise;
    "<${lcpart}\@${hostname}>";
}

puts now, the 'date';
puts mid, the 'mid';

# (Don't worry about these for now; these are my attempt to
# cooperate with a "special" nonstandard header that Microsoft
# Outlook invented called Thread-Index.  Don't be like Microsoft.
# Just use References/In-Reply-To like normal mail clients do.)
if (-x 'append_index.py' && -f (the 'base') && -f (the 'since')) {
    my $base = gets the 'base';
    my $since = gets the 'since';
    my $delta = $time - $since;
    $delta >= 0 or die "Time traveler";
    my $rando = int rand 0x100;
    print STDERR "./append_index.py $base $delta $rando\n";
    puts `./append_index.py "$base" "$delta" "$rando"` => the 'index';
} elsif (-x 'create_index.py') {
    print STDERR "./create_index.py $time\n";
    puts `./create_index.py "$time"` => the 'index';
}

(The code for generating Date and Message-ID are actually taken from two places: here and here (which in turn was derived from Email::MessageID. Crazy!))

Then it continues to compose.py, which is a very large Python file (~1000 lines) that uses the built-in email library and my QP encoder to generate an email message from a config file. The config file… looks a bit deranged:

$ cat test.conf
headers
  Date: < test.d/date
  Message-ID: < test.d/mid
  From: test <emeng@cs.wisc.edu>
  To: <victim@rapidcow.org>
  Subject: Hello :)
content
  text/plain < test.d/text
    with charset us-ascii
    of encoding 7bit
      where treol = please!

(I am not even kidding, “please!” is a boolean.)

But you can see that it reads from the date/MID files generated by compose.pl (they cooperate very well, that is!) And the syntax looks eeriely similar to the actual output:

$ cat test
Date: Thu, 27 Mar 2025 23:35:35 -0500
Message-ID: <1743136535.66565.9D90e0De9bEfA@e.rapidcow.org>
From: test <emeng@cs.wisc.edu>
To: <victim@rapidcow.org>
Subject: Hello :)
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit

hi i am good and this is TesT

Mewheheheee.... >:))))
-evil

Of course, with the last three lines being the same “magic” MIME lines I had us include at the beginning.

Next would be a confirmation followed by a preview; I use Vim because it has this nice “mail” syntax highlighting. Also, when you quit Vim with :cq, it sets the error status; so if I’m in the middle of this and I realized I made a typo in my email, I can use that to abort the script and fix my typo.

But if I’m fine with that Vim exits normally, the script hands off to queue.pl… and this is the part I want to show you: because it is precisely what we just did – but in Perl.

Now, okay, admittedly there is a little more going on in this queue.pl: namely it extracts the addresses of the recipient (To, Cc, Bcc) and deletes the Bcc header. But the heart of queue.pl is still the sending part: and that is all thanks to the Net::SMTP package.

You can translate what we did with word by word!

When we typed telnet localhost 25, this translates to:
```
my $smtp = Net::SMTP->new('localhost', Port => 25);
```
When we typed openssl s_client -nocommands -quiet -crlf -starttls smtp -connect smtp-auth.cs.wisc.edu:587, that translates to:
```
my $smtp = Net::SMTP->new('smtp-auth.cs.wisc.edu', Port => 587);
$smtp->starttls();
```
It is quite hard to overstate how simple the interface Net::SMTP provides is. (Installing can be quite the challenge, but I haven’t found too much trouble installing IO::Socket::SSL and Net::SSLesy. There will be a complete, working script at the end that you can try.)
EHLO would have been done by Net::SMTP at the new call/starttls(), I think? I haven’t been able to figure this part out much. I guess this is where the author(s) of Net::SMTP decided that they would abstract away details of the protocol.

You authenticate by creating an Authen::SASL ref:

my $sasl = Authen::SASL->new (
    mechanism => 'PLAIN',
    callback => {
        user => 'bbadger',
        pass => '123456789ABCDEFGHIJ',
    },
)
$smtp->auth($sasl);

There is also a shortcut? Though I’m not sure what mechanism it would use in this case exactly. (It might try all of them one at a time.)

$smtp->auth('bbadger', '123456789ABCDEFGHIJ');

Then to do the MAIL and RCPT verbs you use mail() and recipient() methods:

$smtp->mail('bbadger@cs.wisc.edu', Bits => 8);
$smtp->recipient('alice@example.org', 'bob@example.org');

Now for the best part(s): sending the actual email. Probably my favorite part of Net::SMTP’s interface (second only to its support for the 8BITMIME which I just discovered while re-writing queue.pl for this) is how it lets you send the message line by line: just like how it was done on the terminal (and – dare I say – the old days of SMTP…)
```
$smtp->data();
$smtp->datasend("<line 1>\n");
$smtp->datasend("<line 2>\n");
...;
$smtp->datasend("<line n>\n");
$smtp->dataend();
```
This corresponds directly to:
```
DATA
<line 1>
<line 2>
...
<line n>
.
```
except with the benefit that if any of these lines happen to also be a lone period, datasend() adds a period in front to sort of “escape” it I guess. (I’m not sure how that’s supposed to work, but you can read the docs/code of Net::Cmd, which is the part that implements this data-transmission protocol.)
Now to close the connection we just call quit:
```
$smtp->quit();
```
Actually, the mail already gets sent by the time you hit dataend(). I just thought we’d be polite and tell the server that we’re done here before hanging up abruptly.

And now you know how to send email with Net::SMTP as well :) Or it can feel like that… the actual details are a bit gritty.

Lucky you, I’ve been through the gritty details, and as I said, I rewrote the actual queue.pl I have here so that you can use it too! Here. Read the comments. Go crazy! (though not crazy in the sense of spamming, of course.) Send me a mail!

And when people are comfortable, we can look at XOAUTH2 (so we can also send through Microsoft), and perhaps Mutt next… But this should be enough for now. :)