I’ve been generating email on my own services for quite some time. 20 years ago sending email was easy. I could fire up sendmail and do something like:
sendmail -f [email protected] [email protected] <<EOF
Subject: Hello my friend
From: Steve Jobs <[email protected]>
I love you
EOFEasy as pie. The problem with this ease is I could easily set that -f argument to any email address I like. On the other end this email would look legitimate so long as I didn’t abuse the receiving email provider with my IP. We know where this story goes. Spammer pretends to be CEO, asks finance to write a check to ABC company and finance made the blunder of actually trusting what they saw in an incoming email.
The 2000s are filled with poorly thought out solutions to these kinds of problems. Most of them either revolved around scanning messages for certain characteristics, phrases or expressions and then there were the IP reputation services. PGP as an attempt to secure message authorship, but required active participation from all receivers. While these ideas met limited success there were many casualties along the way. There are also the “double knock” type email systems where your intended party’s service will send an automated message back to you and ask you to verify you are human before your message will be released from a queue. Solutions like these cause all sorts of unintended side effects, especially if the email can never be seen by the intended party by perusing their spam folder.
Enter Sender Policy Framework, or SPF, is a way to use something authoritative like DNS to tell mail providers where they can expect mail to come from for a domain. An email from a domain without SPF available might seem sketchy to some email providers. Add an SPF policy and you get a gold star from the likes of Google, Outlook and other free e-mail providers.
v=spf1 ip4:127.0.0.1 ip4:127.0.0.2 include:mx.example.com include:mx2.example.com -allWhat does this mean? v=spf1 is the version of SPF we are utilizing. As of 2021 this will always be v=spf1. Next in importance is the end of this line. -all tells other providers to FAIL any email that doesn’t conform to this policy. If you domain does not serve email at bare minimum you should set this record to:
v=spf1 -allThe above basically says, reject all email claiming to be from my domain. Optional elements of the SPF record are “a” “ip4” “ip6” “mx” “ptr” “exists” and “include”.
- A – Tells providers that mail coming from the same source as the main A record for a domain should be accepted.
- IP4 (followed by a dotted IP4 address) tells providers what IPs or Networks can send email under our domain
- IP6 – is the IPv6 equivalent of the above.
- MX – tells providers that mail coming from servers in our MX record are safe.
- PTR – tells the provider that rDNS for a client is enough to allow email (this should be avoided)
- EXISTS – uses SPF macros to match address
- INCLUDE – includes the referenced domain’s SPF policy.
- ALL – matches everything, this is usually why ALL is defined at the end of a record.
Next are the qualifiers:
- + : Tells other providers that the rule is passing, lack of a qualifier defaults to this qualifier
- – : Tells other providers to fail on this rule.
- ~: Soft fail, typically quarantine the message
- ?: Neutral, is typically interpreted like no policy even exists.
Now that we know all of this we can interpret this line:
v=spf1 ip4:127.0.0.1 ip4:127.0.0.2 include:mx.example.com include:mx2.example.com -allVersion 1 Pass IPs 127.0.0.1 and 127.0.0.2 and include the SPF records from the domains mx.example.com and mx2.example.com, Reject everything else.
Now that we know SPF, problem solved right? We can now all implement SPF and never look at this problem again. Well, no, we need to apply a cryptographic signature to emails because how do we know that the email that we sent hasn’t been altered to look legitimate? Once it leaves my outbox, I trust many people with my email and any one of them could make an alteration to the email. I am sure you are familiar with the concept of a Man in the Middle attack. This is what Domain Keys were meant to solve.
Back in the mid aughts Yahoo was in the constantly frustrating position of being a spam target as such they saw a growing problem of manipulated emails. They brought out their own standard to resolve these issues. The problem was that not many people were aware of Domain Keys and other providers like Cisco were also trying to solve the issue of manipulated emails. They called their system Internet Identified Mail. Instead of competing to solve this problem they opted to join forces and create *drum roll* Domain Keys Identified Mail. We all know this today as DKIM, but this would solve the issue of mail authorship identity.
DKIM solves this by issuing a public key that is also placed on the authoritative DNS of our domain. Are you detecting a theme here? We also need to cryptographically hash our email at the source. Many use a tool called opendkim which is a filter placed in a mail server like postfix. A private key is used to generate a hash that is added to the header of our message. The header is labeled DKIM Signature. Inside this signature we define which header fields we want hashed, the “From” field is the only required header field. Along with the cryptographic algorithm in use. From Wikipedia:
DKIM-Signature: v=1; a=rsa-sha256; d=example.net; s=brisbane;
c=relaxed/simple; q=dns/txt; [email protected];
t=1117574938; x=1118006938; l=200;
h=from:to:subject:date:keywords:keywords;
z=From:[email protected]|To:[email protected]|
Subject:demo=20run|Date:July=205,=202005=203:44:08=20PM=20-0700;
bh=MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=;
b=dzdVyOfAKCdLXdJOc9G2q8LoXSlEniSbav+yuU4zGeeruD00lszZVoG4ZHRNiYzRThe important elements of the above are version (in this case 1) , algorithm (in this case rsa-sha256) domain (example.net) selector (The DNS record holding the public key) bh and b are our body header and body hashes. The mail provider will then use the public key along with elements of the message to hash and compare the result. This TXT DNS entry will look something like this:
Name: brisbane._domainkey.example.net.
Value: v=DKIM1; k=rsa; t=s; p=<public key>If the hashes match, we have a winner and it passes DKIM, if they are mismatched then it fails. The absence of DKIM is considered, as of 2021, neutral because not everyone has caught on to the importance of DKIM.
However, a new problem is coming. We have these great tools to help fight fraudulent email but setting all these things up is confusing and every mail provider could weight certain aspects of these tools more than others. On top of this, once our email messages are delivered to another mail provider we really have no idea what happened to that message. Was it delivered? Did our broken DKIM implementation block it? What if Kevin in support decided to setup a newsletter service for his customers and neglected to tell IT that they need to add a new provider to the SPF record. He is happily toiling away building these newsletters that are never seen because they all go straight to spam or are outright blocked and IT is unaware because who would ever report this issue.
Enter Domain-based Message Authentication, Reporting and Conformance, boy that was a mouth full. DMARC was created to solve this issue. We setup a new, oh you guessed it, DNS record to tell email providers what to do with our broken email systems. On top of this we also define a reporting email to send helpful reports about the activity of email carrying our domain. This is an important point because any number of email systems could be claiming to be an SMTP provider of our domain. Some of these systems are legitimate some are not. Using a DMARC policy we can tell other email providers what to do with email that doesn’t pass DKIM or SPF. We also can get daily reports on what is happening with emails claiming to be from our domain.
v=DMARC1;p=quarantine;sp=reject;pct=100;rua=mailto:[email protected];v Defines the version of DMARC we are using, p is the policy, sp is the subdomain policy pct is the amount of emails to apply the policy to and finally rua is the address to send reports to.
Generally you want to be fairly permissive with DMARC while you figure things out. I recommend utilizing the quarantine policy until you have things working as they should, then move to reject. You can then examine your DMARC reports and determine if things are working as they should. Once you are certain that the systems that should work do, move to a reject policy.
What about these DMARC reports, what do they say exactly? Well let’s look…
<policy_published>
<domain>example.com</domain>
<adkim>r</adkim>
<aspf>r</aspf>
<p>quarantine</p>
<sp>quarantine</sp>
<pct>100</pct>
</policy_published>This block tells us what the mail provider detected as our current policy. This policy is defining a relaxed posture towards dkim and spf issues. Additionally when a problem is detected we are asking them to only quarantine messages which means they typically go to the spam folder.
adkim and aspf both default to a relaxed posture “r”. You can define a strict one by setting your DMARC option adkim=s;
Next you will find a record…
<record>
<row>
<source_ip>127.0.0.1</source_ip>
<count>36</count>
<policy_evaluated>
<disposition>none</disposition>
<dkim>pass</dkim>
<spf>pass</spf>
</policy_evaluated>
</row>
<identifiers>
<header_from>example.com</header_from>
</identifiers>
<auth_results>
<dkim>
<domain>example.com</domain>
<result>pass</result>
<selector>mail</selector>
</dkim>
<spf>
<domain>example.com</domain>
<result>pass</result>
</spf>
</auth_results>
</record>Seeing this record tells us that 127.0.0.1 is a good source for our domain and has successfully sent 36 emails to this specific provider. DKIM and SPF pass and all mail has made it to their intended parties. Note: that emails could still be filing as spam for not meeting other heuristics of the provider or being marked as spam by individuals. So what happens with problem sources?
<record>
<row>
<source_ip>192.168.1.1</source_ip>
<count>5</count>
<policy_evaluated>
<disposition>quarantine</disposition>
<dkim>fail</dkim>
<spf>fail</spf>
</policy_evaluated>
</row>
<identifiers>
<header_from>example.com</header_from>
</identifiers>
<auth_results>
<spf>
<domain>example.com</domain>
<result>softfail</result>
</spf>
</auth_results>
</record>This tells us that a server located at 192.168.1.1 tried to send 5 messages as us to this provider. Since they were not in our SPF and likely could not have provided a valid DKIM hashed email they failed and according to our DMARC policy were properly quarantined.
We setup all three and now we can continue sending automated email and ride off into the sunset. Right? Well no, automated email can easily break DKIM. Even if you setup everything perfectly, if you do not know how to format a message to conform to SMTP limits a message can be altered in transit, breaking DKIM signatures. See, still broken. SMTP has limits and unfortunately a lot of tools have been placed over the top of this unruly protocol. First and foremost you must ensure that your systems are using the correct sender. Use -f to set the appropriate email address that contains our real domain, don’t just let the system set whatever it likes because it will just use the current user @ hostname as the author. Even if you set a “From” field in the body of your email, the “-f” flag is critically important in identifying who the real author should be. I know this sound counter-intuitive but “From” “-f” “Return-Path” “Reply-To” all do different things.
Additionally SMTP has line length limits. It’s easy for the body of a message to contain log lines or html content that could hit these limits. The general rule I follow is no line longer than 900 characters. As a general rule you can opt to Encode the message to protect it from these limits. Encode protocols that I quickly reach for are:
Content-Transfer-Encode: base64
# or
Content-Transfer-Encode: quoted-printableWait, why does this matter, you probably thought I was talking about troublesome concepts like DKIM or DMARC. It matters because if you fail to limit line length some SMTP server along the chain are going to force the change for you. When that enforcement occurs it will break the DKIM hash because now some other SMTP server is going to enforce a line length restriction which will alter the original message. In the old days this alteration would have gone unnoticed but with cryptographic integrity on the line, it will be noticed.
Base64 encoding is probably the easiest to understand, all we need to do is encode the material and split it.
cat file.log | base64 | fold -w 76Now you just need to set a Content-Transfer-Encode header in the email along with the original Content-Type.
Another option is to use Quoted Printable, this is typically used for HTML emails so I won’t go into examples but I can tell you how it works. You will want to fold all lines to be no longer than 76 chars and you will need equal (=) encode lines that have been wrapped. Any = characters found in the source document will also need to be encoded to =3D
An example:
Content-Transfer-Encoding: quoted-printable
<html><head></head><body style=3D"font-family:Arial;"><table align=3D"cen=
ter" border=3D"0" cellspacing=3D"0" cellpadding=3D"0" width=3D"60%" bgcolor=
3D"#FFFFFF" style=3D"background-color:#FFFFFF;table-layout:fixed;-webkit-te=
xt-size-adjust:100%;mso-table-rspace:0pt;mso-tablelspace:0pt;-ms-text-size-=
adjust:100%;min-width:500px"><tr><td><table align=3D"center" border=3D"0" c=
ellspacing=3D"0" cellpadding=3D"0" width=3D"100%" bgcolor=3D"#EDF0F3" style
...Basically = allows you to define any character but you must use this escaping policy for any non-printable ASCII characters. You can read more in the wiki article linked below.
Hopefully these hints will help someone who runs into these issues in the future. Some resources that will help you in your journey.