The anatomy of the SMTP subsystem

Overview

UCB Compatibility
   SENDMAIL.EXE
       ³
       ³
SMTP Submission     Direct submission        SMTP Relay
   SMTPSD.EXE           MAILTO.EXE           SMTPRD.EXE
       ³                    ³                    ³
       ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÂÁÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ
                           ³
                     SMTP Maildrop area
                           ³
                           ³
                 SMTP Post Office Sorting
                       SMTPPD.EXE
                           ³
         ÚÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÁÄÄÄÄÄÄÄ¿
         ³                         ³
  SMTP Local delivery      SMTP Remote delivery
      SMTPLD.EXE              SMTPRCD.EXE
         ³                         ³
         ³                         ³
    Mail folders			 SMTP Relay client
         Ý                     SMTPRC.EXE
         Ý
     IMAP server

The flow of mail in the SMTP subsystem of the Internet Utilities divides into two areas: local or remote entry of messages into the message system, and sorting of messages and their delivery to or towards their destinations.

There is a "straight paper path" for a message that arrives via SMTP Relay, passes through the SMTP Post Office Sorting dæmon, and leaves via the SMTP Relay Client dæmon. Everything along that path operates on the message envelope alone. The content of the message (headers and body) is not inspected, used, or modified in any way (except for the single "Received:" header line that the SMTP Relay dæmon is required to prepend to the entire message).

SMTP Input

Messages are built as temporary files by the SMTP Relay server, the SMTP Submission server, or the direct submission utility. Once the program that builds the message has collected the entire message, and performed whatever extra processing on it that is appropriate, the file containing the message is transferred to the SMTP Maildrop area.

The SMTP Submission dæmon supports message submission from User Agents via the SMTP Submission protocol. It listens for connections on the standard SMTP Submission port, and performs a standard SMTP dialogue with clients. This is the preferred mechanism for User Agents to submit messages. (The SMTPSD command can be used directly to perform "Batch SMTP" submission of messages, as well.)

The Direct Submission (MAILTO) command allows messages to be submitted from the command line, using a simple syntax similar to that of UNIX "mail" commands. It submits messages directly, without requiring an SMTP Submission dæmon.

Both the SMTP Submission dæmon and the MAILTO command validate the user-supplied message headers, before submitting the message to the maildrop, to ensure that they are present and correct, and rewrite them according to various optional local policy considerations (such as site masquerading).

The SMTP Relay dæmon, receives messages from Mail Transfer Agents on other hosts via the SMTP Relay protocol. It listens for connections on the standard SMTP Relay port, and performs a standard SMTP Relay dialogue with clients.

Because such messages are "in transit" and must not be altered, and must already be in "canonical" RFC 822 form, the SMTP Relay dæmon is able to place them immediately into the SMTP Maildrop without further processing. The only processing that the SMTP Relay dæmon does is to prepend a "Received:" header.

The UCB compatibility utility is in fact an SMTP Submission client. It provides an interface similar to that of the UCB "sendmail" program, and submits the message by performing an SMTP Submission dialogue with a local SMTP Submission server.

The SMTP Maildrop is where inbound messages are stored once they are in "canonical" form. Messages in the SMTP Maildrop comprise two parts, the message envelope and the message contents.

Because placing a message into the SMTP Maildrop has to be interlocked with the operation of the SMTP Post Office Sorting dæmon, which decides how to route messages to local or remote delivery, the SMTP Maildrop should not be accessed directly.

SMTP Output

The SMTP Post Office Sorting dæmon decides whether messages are to be delivered locally or remotely, reading the envelope and routing the message at the same time. A message is routed by routing each of its individual recipients.

The SMTP Local Delivery dæmon handles local delivery to mail folders. The SMTP Remote Delivery dæmon handles message transport to other MTAs via their SMTP Relay servers.

Both dæmons take responsibility for all message+recipient pairs that are handed to them by the SMTP Post Office Sorting dæmon, in that they will either deliver the message successfully to that recipient, or generate a bounce message back to the sender. Bounce messages are generated by a permanent failure to deliver a message to a particular recipient, or when the number of temporary failures exceeds a given limit.

The SMTP Remote delivery dæmon spawns one or more instances of the SMTP Relay Client. It is this that actually performs the grunt work of delivery. It attempts to establish an SMTP connection to an SMTP Relay server on a remote host, and transport messages over that connection. The SMTP Remote delivery dæmon handles scheduling of re-tried delivery attempts upon failure.

SMTP Submission

SMTP Submission is designed for MUAs to submit "outbound" messages from them to the rest of the world (or, at least, to the MTS at large) using SMTP (as per RFC 2476). All messages submitted via SMTP Submission are converted to canonical form before they are moved to the SMTP Maildrop area. Messages entered by SMTP Submission have their headers parsed and possibly altered, normalising mailbox names and adding default values for missing parts. MUAs are thus allowed to omit most headers, since they will be filled in automatically.

No restrictions are placed upon the envelope sender and envelope recipients of messages sent by clients that are allowed SMTP Submission service, as this would be improper for a submission service. It is important, therefore, that the submission service IP address not be reachable by Internet at large and that proper access controls be set using TCPSRUN if further restrictions are necessary.

SMTP Relay

SMTP Relay is designed for MTAs to relay "inbound" messages to the local MTS using SMTP (as per RFC 821). All messages submitted via SMTP Relay are assumed to be in canonical form already, and are placed directly into the SMTP Maildrop area without any modifications apart from having a single "Received:" header prepended to them. No header rewriting or canonicalisation is performed. MTAs on other machines are trusted to put messages into the proper RFC 822 format, and it is their fault if they are not. SMTP Relay does not modify the contents of in-transit messages in any other way.

Clients that are granted normal SMTP Relay service may only specify envelope recipient mailboxes whose domains are listed as configuration parameters to the SMTP Relay dæmon service. All other envelope recipient mailboxes are rejected. This ensures that all provision of SMTP Relay service be by prior agreement. If, unilaterally and without one's agreement, the owner of a domain lists in the public DNS database one's SMTP Relay service as one of the mail exchangers for their domain, the domain name will not be in the list of acceptable envelope recipient domains, and all mail to the domain will be rejected by the SMTP Relay service. It also ensures that the malicious cannot use the SMTP Relay service to send mail to arbitrary third party destinations, making one bear the expense of queueing and transporting that mail.

SMTPRD also optionally implements Teergrubing (a.k.a. "tarpitting"). When the list of envelope recipients exceeds a particular size, SMTPRD begins to insert deliberate delays into its responses to "RCPT" commands. As the list grows further, the delays are increased. The purpose of this is to keep clients that are sending messages addressed to hundreds of recipients connected longer, on the principle that this penalises those who send unsolicited bulk mail by increasing their Internet connection charges.

Null SMTP

Null SMTP is designed to reject undesirable MTAs/MUAs. The clients are greeted with an initial 554 greeting instead of the standard 220 greeting. All further commands apart from QUIT will result in a 503 error result code.

DNS-based blacklisting and whitelisting

Both the SMTP Relay dæmon and the SMTP Submission dæmon are capable of DNS-based blacklisting and whitelisting. The look up the IP addresses of clients in the specified SMTP client access lists, by querying the local proxy DNS server. The lists are searched in the order specified to the server program, and searching stops at the first list containing the IP address.

Clients listed in a black list may only specify envelope recipient mailboxes whose local parts specify the postmaster. Blacklisting denies access to all other local parts, and applies to both normal and privileged access. Clients listed in a white list are not restricted in this manner. The purpose of white lists is to allow one to configure a local white list, with (say) DNSTSD, in order to override third party blacklisting services for clients to whome one specifically wishes to allow unrestricted access.


The Internet Utilities are © Copyright Jonathan de Boyne Pollard. "Moral" rights are asserted.