I Built My Own Mail Server: Postfix, Dovecot, rspamd, and the Calm Path to Deliverability (with IP Warm‑Up)

So there I was, staring at a flood of logs on a late Tuesday evening, realizing I had built myself a perfectly functioning email server that nobody trusted yet. The messages were queued neatly, signed with DKIM, the SPF was crisp, DMARC was ready for inspection. But those first few emails still tiptoed into promotion tabs and sometimes even the void. If you’ve ever tried to self‑host email, you know the feeling. It’s like building a cozy café and then wondering why the neighborhood hasn’t discovered your perfect cappuccino.

In this guide, I’ll show you how I set up a calm, reliable email stack—Postfix for sending, Dovecot for IMAP/POP3, and rspamd for filtering—and the deliverability work that actually moves the needle. We’ll also walk through a practical IP warm‑up plan (the part many skip), because reputation is everything. I’ll share the exact steps, the missteps I wish I’d avoided, and the small tweaks that made a big difference. By the end, you’ll have a working stack, a sensible deliverability workflow, and the confidence to send without the drama.

Why Self‑Host Email at All?

I get this question a lot. When a SaaS sender can get you up and running fast, why bother? For me, it’s about control and craft. Self‑hosting email lets you understand what’s under the hood, tune it to your needs, and keep your data where you want it. It also teaches you a ton about DNS, TLS, authentication, and how inbox providers actually judge your messages.

Here’s the thing: email is not just software. It’s a reputation engine. And reputation moves slowly. When you run your own MTA (Mail Transfer Agent), you’re responsible for how that reputation is built, nurtured, and protected. That can be incredibly empowering when you do it right. But it rewards patience and good habits.

The Groundwork: DNS, Identity, and Reputation Before a Single Byte

Before you install anything, set your foundation. Think of DNS and identity as your café’s signage and permits. Skip them and you’ll wonder why nobody walks through the door.

Start with a clean domain. If the domain has a history of spam complaints, consider using a new subdomain like mail.example.com or mg.example.com for outbound traffic. Make sure your VPS provider gives you control over rDNS (PTR). Your reverse DNS must resolve to your server’s hostname, and that hostname should resolve back to the IP. Forward and reverse resolution should match; mailbox providers love that neat loop. If your provider doesn’t allow rDNS, pick one that does—this detail matters.

Then line up your email authentication trio: SPF, DKIM, and DMARC. SPF says who’s allowed to send for your domain. DKIM signs the message so it can’t be tampered with. DMARC ties SPF and DKIM to your domain’s policy and reporting. If you want to lean into brand trust later, BIMI can help, but only after your core reputation is healthy. When you’re ready for the deeper end of the pool, I wrote a friendly playbook for advanced DMARC, RUA/RUF analysis, and BIMI that pairs nicely with this guide.

Two more signals worth your time: MTA‑STS and TLS‑RPT. MTA‑STS lets you declare that you expect TLS for SMTP, and TLS‑RPT collects reports when that doesn’t happen. They don’t unlock instant deliverability, but they round out a story of a careful sender who values secure transport.

VPS Prep: Hostname, Time, Firewall, and Updates

I’ll assume a Debian or Ubuntu VPS, but the concepts apply broadly. Pick a stable image. Set your hostname to match what you intend to present via SMTP—something like mail.example.com. Make sure DNS A/AAAA records and PTR/rDNS are in place first. Then sync time with NTP; mismatched clocks cause weird TLS and DKIM signatures to fail. Update your packages, enable a firewall (UFW or nftables), and open only what you need: 25 (SMTP), 587 (submission), 465 (smtps, if you prefer implicit TLS), 993 (IMAPS), and maybe 995 if you still want POP3S. I tend to skip plaintext 110/143 entirely.

One more thing that quietly matters: IPv6. If you have a clean IPv6 allocation and working reverse DNS, great—enable it. If your provider’s IPv6 rDNS is broken or non‑configurable, stick to IPv4 for outbound at first. A broken IPv6 path can throttle your deliverability without obvious errors. I learned that one the hard way after three quiet days and a lot of head scratching.

Postfix: Your Calm, Reliable MTA

Postfix is the heart of this stack. I love it because it’s predictable and well‑documented. The defaults are sensible, but we’ll turn on the things that matter for a modern, authenticated sender.

Install and basic layout

apt update && apt install -y postfix postfix-pcre libsasl2-modules

Choose “Internet Site” during installation and set your system mail name to your primary mail hostname. You can always change it later in /etc/postfix/main.cf.

Core Postfix config

Here’s a practical starting point. Adjust domains, hostnames, and certificates as needed.

# /etc/postfix/main.cf
myhostname = mail.example.com
myorigin = /etc/mailname
mydestination = localhost
relay_domains =
internet_protocols = ipv4

# Network
inet_interfaces = all
inet_protocols = all
smtpd_banner = $myhostname ESMTP

# TLS for inbound
smtpd_tls_cert_file = /etc/letsencrypt/live/mail.example.com/fullchain.pem
smtpd_tls_key_file  = /etc/letsencrypt/live/mail.example.com/privkey.pem
smtpd_tls_security_level = may
smtpd_tls_protocols = !SSLv2, !SSLv3
smtpd_tls_mandatory_protocols = TLSv1.2 TLSv1.3
smtpd_tls_auth_only = yes

# TLS for outbound
smtp_tls_security_level = may
smtp_tls_loglevel = 1
smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt

# SASL auth via Dovecot
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
smtpd_sasl_auth_enable = yes
smtpd_sasl_security_options = noanonymous

# Restrict who can use submission
smtpd_recipient_restrictions =
    permit_mynetworks,
    permit_sasl_authenticated,
    reject_unauth_destination

# HELO hygiene
smtpd_helo_required = yes
smtpd_helo_restrictions =
    reject_invalid_helo_hostname,
    reject_non_fqdn_helo_hostname

# Virtual domains
virtual_mailbox_domains = example.com
virtual_transport = lmtp:unix:private/dovecot-lmtp
virtual_mailbox_maps = ldap:/etc/postfix/ldap-users.cf # or hash/sql if you prefer
virtual_alias_maps = hash:/etc/postfix/virtual

# Performance / rate controls (tune for warm-up)
maximal_queue_lifetime = 3d
bounce_queue_lifetime  = 3d
smtp_destination_concurrency_limit = 3
smtp_destination_rate_delay = 1s

That last bit on concurrency and rate delay is a gentle handbrake that helps when you’re warming up your IP. Don’t go overboard; start conservative and observe how large providers respond.

Submission and smtps services

Enable authenticated submission on 587 (and 465 if you want implicit TLS) so your users or apps can send securely.

# /etc/postfix/master.cf
submission inet n       -       y       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject

smtps      inet n       -       y       -       -       smtpd
  -o syslog_name=postfix/smtps
  -o smtpd_tls_wrappermode=yes
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject

Reload Postfix when you’re ready:

systemctl restart postfix

If you want to go deeper (and you should), the Postfix official docs are a goldmine. I’ve lost count of how many tiny “aha” moments I’ve had there.

Dovecot: IMAP That Feels Effortless

Dovecot handles mailbox storage and authentication. I like to think of it as the quiet concierge who keeps everything tidy and fast. It also offers LMTP delivery and Sieve filtering—two features that make the whole experience feel polished.

Install and basic config

apt install -y dovecot-imapd dovecot-lmtpd dovecot-sieve dovecot-managesieved

Set TLS, mail location, and auth. For virtual users, a common pattern is to create a vmail user and use Maildir storage.

# /etc/dovecot/dovecot.conf
protocols = imap lmtp
listen = *,::

# TLS (use the same cert as Postfix)
ssl = required
ssl_cert = 
ssl_key  = 
ssl_min_protocol = TLSv1.2

# Mail storage
mail_home = /var/vmail/%d/%n
mail_location = maildir:~/Maildir
first_valid_uid = 150
first_valid_gid = 8

# Auth
auth_mechanisms = plain login scram-sha-256
passdb {
  driver = passwd-file
  args = scheme=SHA512-CRYPT username_format=%u /etc/dovecot/passwd
}
userdb {
  driver = static
  args = uid=vmail gid=vmail home=/var/vmail/%d/%n
}

# LMTP for local delivery from Postfix
service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    mode = 0600
    user = postfix
    group = postfix
  }
}

# SASL socket for Postfix submission
service auth {
  unix_listener /var/spool/postfix/private/auth {
    mode = 0660
    user = postfix
    group = postfix
  }
}

Create the vmail user and directories, then add users to /etc/dovecot/passwd with hashed passwords. Dovecot’s doveadm pw helps generate secure hashes.

adduser --system --group --home /var/vmail vmail
mkdir -p /var/vmail
chown -R vmail:vmail /var/vmail

# generate a hash
doveadm pw -s SHA512-CRYPT

Restart and test IMAP connectivity. I still use openssl s_client -connect mail.example.com:993 just to peek at certificates and ciphers. Old habit, still useful.

When you’re ready to dive deeper, the Dovecot documentation is excellent and practical.

rspamd: Smart, Fast, and Surprisingly Fun

I remember flipping the switch on rspamd for the first time and watching spam scores roll in with laser‑like detail. It’s fast, modular, and integrates beautifully with Postfix via milter—so much so that it feels like it was designed for this exact trio.

Install and wire up milter

apt install -y rspamd redis-server

Enable Postfix milter integration so rspamd can scan inbound and (optionally) sign outbound via DKIM.

# /etc/postfix/main.cf (continued)
smtpd_milters = inet:127.0.0.1:11332
non_smtpd_milters = $smtpd_milters
milter_default_action = accept
milter_protocol = 6

Restart both:

systemctl restart rspamd postfix

DKIM keys and signing

rspamd makes DKIM easy. Generate a selector and publish the public key in DNS.

rspamadm dkim_keygen -b 2048 -s mail -d example.com -k /var/lib/rspamd/dkim/example.com.mail.key > /var/lib/rspamd/dkim/example.com.mail.pub
chown -R _rspamd:_rspamd /var/lib/rspamd/dkim

Then configure signing in /etc/rspamd/local.d/dkim_signing.conf:

domain { example.com { selector = "mail"; path = "/var/lib/rspamd/dkim/example.com.mail.key"; } }
allow_envfrom_empty = true
sign_authenticated = true
sign_local = true

While you’re there, consider enabling ARC and DMARC reporting modules. ARC can help in forwarders, and rspamd’s DMARC checks give you an extra layer of insight. The rspamd documentation has practical examples that translate well to production.

Greylisting, RBLs, and rate limiting

I tend to enable a modest greylist window and a few conservative DNSBLs. rspamd’s default rules are already strong, so resist the urge to pile on every blocklist under the sun. Less is more, as false positives can burn trust quickly. For outbound, rspamd’s rate limit module is a handy guardrail when warming up. You can also lean on Postfix destination concurrency and rate delays to keep things smooth.

TLS That Sticks: Certificates and Ciphers Without the Drama

Your mail stack should present consistent, trusted TLS. I usually use Let’s Encrypt for the server certificate and set it for Postfix and Dovecot. Make sure the certificate common name or SAN includes the exact hostname you advertise in SMTP (that myhostname in Postfix) and IMAP. Rotate and reload on renewal—cron or systemd timers make it hands‑off.

If you’re wondering about DV vs OV/EV for mail: in practice, DV is usually enough for SMTP and IMAP. If your compliance or customer trust story demands more, I walked through choosing the right certificate type for real‑world hosting. Apply the same thinking to mail: pick the certificate that matches your risk, brand, and budget.

DNS Records That Earn Trust

Let’s sketch the essentials you want published before sending anything at scale.

For SPF, keep it tight. If Postfix is your only sender, something like v=spf1 a:mail.example.com -all is perfect. For DKIM, publish the mail._domainkey.example.com TXT with the rspamd public key. For DMARC, start with p=none and reporting: v=DMARC1; p=none; rua=mailto:[email protected]; ruf=mailto:[email protected]; fo=1. Watch reports for a week or two, then move to quarantine and then reject once you’re confident. If you want a deeper, friendlier strategy to go beyond p=none, see my advanced DMARC and BIMI playbook.

Don’t forget PTR (reverse DNS) for your sending IP, MTA‑STS (a simple TXT and HTTPS‑served policy file), and TLS‑RPT (a TXT that announces where providers can send TLS failure reports). These small moves paint a picture of a careful sender.

Testing the Plumbing Before Hitting Send

Before inviting real humans into the story, I run a handful of tests. Send yourself a message from the server to a few seed accounts on major providers. Look closely at the headers: Authentication‑Results tells you whether SPF, DKIM, and DMARC aligned. If something looks off, it usually comes down to a DNS typo, clock skew, wrong DKIM selector in the header, or reverse DNS mismatch.

I also nudge the SMTP handshake by hand. A quick openssl s_client -starttls smtp -connect mail.example.com:25 is old‑school but still golden for verifying TLS. For outbound, queue management is your friend: postqueue -p to peek, postqueue -f to flush, and postsuper -d ALL deferred when you need a clean slate after a config mistake.

Deliverability: The Part That Actually Decides Your Fate

Now the honest part. Deliverability is a trust game, and that trust is measured by engagement and consistency. If the first messages you send are cold blasts to old lists, your IP will be scolded like a kid who barged into the wrong classroom. Start small, start warm.

In my experience, three things move the needle most early on. First, send to people who already know you: transactional emails, password resets, and real conversations. Second, segment by domain and watch responses—Gmail and Outlook have their own rhythms. Third, keep your content simple at first. Avoid heavy images and trackers. Real, clear text with a sensible link tends to land better while your reputation is young.

Handle bounces with care. Hard bounces should be removed fast. Soft bounces deserve a second chance, then a timeout. If you’re sending newsletters, use engagement pruning: if someone hasn’t opened in a while, slow down and then pause. I’ve seen reputation recover dramatically by removing the bottom 10–20% of inactive recipients.

IP Warm‑Up: A Calm, Realistic Plan

Think of IP warm‑up like breaking in a new pair of boots: short walks first, then longer ones. If you sprint out of the gate, blisters are guaranteed.

Here’s a simple pattern I’ve used repeatedly. For the first few days, send only transactional or high‑trust emails. If you run an app, this happens naturally. If not, pick a few internal accounts and partners, and have them reply. Replies are a strong positive signal. During the first week, keep daily volume in the hundreds, not thousands. Spread it across the day. Avoid spikes. By the second week, gently double the volume, but watch bounces and spam placement like a hawk. If you see a provider getting grumpy—deferred deliveries, soft bounces, or an uptick in spam foldering—pause, scale back for that domain, and resume slowly.

Two tricks that help a lot: throttle by domain and schedule. In Postfix, the smtp_destination_concurrency_limit and smtp_destination_rate_delay settings give you per‑destination pacing. You can also create transport maps for big providers to apply specific throttles. And don’t send large batches at the top of the hour—spread them out to look more human. One of my clients cut spam foldering in half simply by staggering sends and tightening their bounce handling.

And yes, keep the content helpful and personal. Few things beat a short, clear email that actually gets a reply. Warm‑up is about building a reputation for being wanted in the inbox. Value signals matter more than volume.

Security: Don’t Invite Trouble to the Party

Lock down your submission ports. Strong passwords or, even better, long randomly generated ones. Consider enabling fail2ban to watch for authentication abuse. For admin surfaces—like the rspamd web UI—put them behind your VPN or protect with client certificates. If that sounds interesting, I shared a step‑by‑step on protecting panels with mTLS on Nginx that fits perfectly here. The day I put mTLS in front of internal dashboards was the day I stopped worrying about brute‑force noise.

On the Postfix side, don’t be shy with smtpd_helo_restrictions and smtpd_client_restrictions. Require FQDN HELO/EHLO, reject non‑FQDN senders, and guard submission to authenticated clients only. For Dovecot, disable plaintext auth unless under TLS, and stick to modern ciphers. And always, always keep your OS patched. Email stacks age quickly in the eyes of scanners.

Logs, Metrics, and Sanity Checks

Deliverability lives in your logs. I keep a split view open during warm‑up: Postfix logs to watch queue behavior, rspamd logs for spam scores, and Dovecot logs for auth patterns. You’ll quickly spot misaligned DKIM, DMARC failures, or a rogue client battering your submission port. When stuff goes sideways, it usually shows up as a deferral trend before it becomes a full‑blown block.

Make a habit of reading headers of sample messages received at major providers. The Authentication‑Results section tells you the truth about SPF, DKIM, and DMARC alignment. Google’s and Microsoft’s postmaster tools are helpful, but even without them, you can steer with good logs and a handful of seed inboxes.

Automation and Reproducibility (So You Can Sleep)

The first time I hand‑built a mail server, I felt like a magician. The second time, I realized I needed a runbook. The third time, I wrote Ansible roles and never looked back. If you plan to maintain this, aim for reproducible builds. Automate user provisioning, DKIM key generation, and safe restarts. You don’t want to solve the same tiny puzzle three times in a row.

If you’re into clean infrastructure flows, here’s a story about how I use cloud‑init and Ansible for first‑boot hardening. And for the DNS side, I’ve leaned on automating DNS and VPS provisioning with Terraform and Cloudflare to keep SPF/DKIM/DMARC consistent across environments. Your future self will thank you when you rotate a key or move IPs.

A Few Real‑World Troubleshooting Stories

One time, Gmail decided my transactional receipts looked “promotional” for three days. Nothing changed in our stack—same IP, same DKIM, same template. What fixed it? We stopped sending receipts at exactly the minute after the hour, added a tiny text‑only intro line, and asked a handful of users to reply if something looked off. The pattern normalized. The lesson: technical correctness and human signals both matter.

Another time, a client’s DKIM kept failing only at one provider. Turned out the DNS host silently truncated the DKIM TXT record because it was too long for their UI. We switched to a split TXT with proper quoting and it was instantly green. When DKIM fails intermittently, suspect DNS formatting first.

And then there was the sneaky IPv6 rDNS with a stale hostname. Outbound connections preferred IPv6, but the PTR pointed to the old server. Everything looked fine on IPv4. The fix was either forcing IPv4 for a bit or getting rDNS updated. The moral: check both stacks, not just one.

Going Beyond the Basics: Sieve, Aliases, and Nice‑to‑Haves

Once your core is steady, sprinkle the quality‑of‑life upgrades. Dovecot’s Sieve filtering lets users sort mail into folders automatically—great for billing, notifications, and newsletters. Postfix aliases make role accounts easy—[email protected] routing to a shared mailbox is a common pattern. If you run webmail, keep it separate from your MTA/IMAP stack for simplicity and security, and consider rate limiting login attempts at the reverse proxy.

If you plan bulk sends, I like using separate subdomains and signing keys for bulk vs transactional traffic. It’s cleaner for reputation and makes DMARC reports easier to interpret. Later, if you pursue BIMI, that neatly separates the brand assets, too.

Documentation You’ll Thank Yourself For

Write down three things: how you create a new mailbox (end‑to‑end), how you rotate a DKIM key, and how you handle a surge in soft bounces at one provider. Those runbooks cut resolution time dramatically. Pair them with simple dashboards or log queries that show deferred vs delivered counts per domain. When the queue whispers, you’ll hear it sooner.

Wrap‑Up: A Calm Mailbox Is Built, Not Bought

If you remember nothing else from this guide, remember this: self‑hosting email is about telling a consistent, trustworthy story—technically and humanly. The software stack—Postfix, Dovecot, rspamd—gets you a sturdy chassis. DNS, TLS, SPF/DKIM/DMARC give you identity and guardrails. Deliverability and IP warm‑up build your reputation, one message at a time. None of it is magic. It’s habits.

Start with a clean domain, set rDNS right, keep your TLS steady, and sign every message. Send to people who want to hear from you, observe how providers respond, and adjust with empathy. If you automate your setup and document your playbook, you’ll sleep better and recover faster from the occasional wobble. And when you’re ready to go further with trust and brand signals, the advanced DMARC and BIMI guide is there for you.

Hope this was helpful! If you have a story about finally seeing your emails land where they belong, I’d love to hear it. Until then—steady sends, clean logs, and inboxes that smile back.

Quick Reference: Handy Links You’ll Actually Use

Official docs if you want to go deep without detours:

Postfix official docs — for transport maps, TLS tuning, and queue behavior
Dovecot documentation — for auth backends, Sieve, and LMTP
rspamd documentation — for DKIM/ARC, rate limits, and scoring

And for the broader hosting context around this stack, these pieces fit neatly alongside your mail server:

Automating DNS and VPS provisioning with Terraform and Cloudflare
From blank VPS to ready‑to‑serve with cloud‑init and Ansible
Protecting admin panels with mTLS on Nginx
Choosing the right certificate type without the drama