How to setup a free mail server on Windows Server – hMailServer, MailEnable and Apache James

Do you want to create a free mail server for your company? Here we will provide you with a step-by-step guide on how to install and configure a mail server on your Windows Server with various types of software: hMailServer, MailEnable, and, Apache James.

The solution will work no matter if you have purchased a Cloud Server or a Dedicated Server with Windows Server 2012, 2016, or 2019. 

Apart from a server, you will also need a domain and some basic DNS knowledge. 

Before installing an email server on Windows Server 

Before installing the mail server, we will need a .NET Framework and IIS for hMailServer and EnableMail. 

Installing .NET Framework 3.5 or .NET. 4.7 and IIS

If you don’t have a preliminary setup, you can install it from the Server Manager. Go to “Manage” (up on the right corner) and select “Add features and Roles”. There you just need to click on “Next” a few times and eventually you will see all the features. Search for the features you need and select them, then click on “Install”. 

For Apache James we need Java Runtime Environment. You can download it from Oracle’s page and install it.

What is hMailServer?

The hMailServer is a handy free (AGPLv3) software that serves as a mail server for Windows Server. Many people choose it, including governments, because it is relatively easy to use, has a rich set of features, and is of course, free!

hMailServer features: 

• SMTP, POP3, IMAP for sending and receiving mails. 
• Built-in backup is always welcome
• SSL encryption for better security
• Anti-spam for a cleaner inbox
• An Anti-virus that stops most of the malicious software
• Web administration for remote controlling

hMailServer installation

• Downloading hMailServer. 

Go to this address and proceed with the download. 

• Installation. 

Open the file you downloaded. Click “Next”, then select “I accept the agreement” and again “Next”. We recommend the full installation of the next step. It includes “Server” and “Administrative Tools”. 

Press “Next”. After that, you can select “build-in database” for this example, but you already have another one, you can select the second option. 

The last step is important too. It includes selecting a strong password for your mail server. 

Now you only need to choose the setup folder and then run the software. The next step is configuration. 

• Configuring the mail server. 

The first part is to click on “Connect” and use your password to enter. 

Now click “Add domain”. Type in your domain and press “Save”. 

From the left menu go to the folder “Accounts”. Add “Address”, for example, charley@YOURDOMAIN.COM. Here you can define the password, the size of the mailbox, and the maximum size of each file.  

Now go to “Settings”, “Protocols” and select “SMTP”. You will need to add the localhost of your server with a valid IP address. Later you need to add an A record that resolves the localhost to its IP and a PTR record for a reverse DNS lookup that resolves the IP to the hostname. The name of your server can be mail.YOURDOMAIN.COM.

Let’s also add an Auto-ban. Go to “Advanced” and select “Auto-ban”. Put the default domain and your password and then adjust the settings. Enable the setting. This will stop IP addresses if they continue to try to log in. 

Now let’s go to “IP ranges”. Select “TCP/IP ports” and configure the SMTP, IMAP, and POP3 ports for this mail server. SMTP – 25, 587, POP3 – 110 and IMAP – 143. When we open these ports, this will allow the mail server to receive mails. 

• Again some DNS work. You will need to add an SPF record to show that this mail server is authorized to send mails from your domain. You will need to add the domain and IP address in the SPF record.

Add an MX record too, for the incoming mails. The record will provide information, which enables accepting mails on behalf of the domain. 

What is MailEnable?

MailEnable is another excellent proposal if you have a Windows Server. It is a paid mail server, but there is a free version too (the Standard). It is one of the most popular mail servers for Windows.

There are 4 versions:

• Standard (Free)
• Professional (starting at 349 USD)
• Enterprise (starting at 699 USD)
• Premium (starting at 1098 USD)

It is popular of its integration with Outlook and its set of features similar to Microsoft Exchange.

MailEnable features

• Supports SMTP, IMAP, and POP3 (All versions)
• Webmail for easy access through the web (All versions)
• Contacts, Calendars, Schedule Tasks (All versions)
• Anti-Spam (All versions)
• Media players for videos and audio for the attached files (All versions)
• ActiveSync (cost extra) for iOS and Android. (Professional and up)
• Content filtering (Professional and up)
• Antivirus (Professional and up)
• Outlook MAPI Connector for MS Outlook (Enterprise, unlimited for the Premium)
• Collaboration and sharing (Enterprise, Premium)

How to install MailEnable?

• Downloading MailEnable.

Go to the MailEnable website and after that go to the download page. Choose the right version for you. The standard versión is free. You could try it out before deciding on an upgrade or a purchase of a more advanced version.

• Installation.

Open the file. The first window will be general information about the software, press “Next”. It will be followed by a License Agreement, click “Yes”. Write the owner’s details. The next step is to select what you want to install and where to install it– “WebMail Service (Server)”. The next screen will ask you about the Program Manager group, just leave it as it is and click “Next”. Here comes an important step, “Repository”. This is the location of the messages’ data. Choose and press “Next”. 

• Creating a post office.

The post office unites all of your domains under one hat. You will need a strong password for it. At this point, you will be asked your domain name, DNS host (the IP address/es for the domain), and port for the SMTP (25). Click “Next” and wait for the installation. 

• WebMail.

You need to select which website are you going to install it for. If you have one, leave it to “Default”.

• Configuration of the DNS.

To be sure that your mail server operates correctly, you will need a few DNS records. You will need to have a fixed IP address and create an A record with your domain and its IP address. You will need MX records for the incoming mail server pointing to the A record. 

Some more necessary DNS settings include the enablement of DNS Forwarding. You will also have to create a Reverse Lookup Zone. You can do it from the DNS Manager by selecting “New Zone”. Inside this zone, you will need a PTR record.

• DKIM enable.

DKIM is an encryption method that verifies the message. To start using it, you will need to go to the administration console – “Servers”, “Localhost”, and “Extensions”. Click with the right button on the “Domain Keys (DKIM). Select the “Enable DomainKeys Identified Mail (DKIM)” functionality on this server.  

To configure it go to the administration console and “MailEnable management > Messaging Manager > Post Offices > (postofficename) > Domains”. On your domain select properties and go to “DKIM”, “Configure”. There click on the “Sign outgoing messages” box. From there you can choose an encryption algorithm, the canonicalization one, impose body hash length limit, and include user identity. After that, you need to configure a selector. The selector is the combination of a public and private key. When you click on “new” you will need to give it a unique name and size of the key. The selector can work in “test mode, granularity (for specific users), notes“ and make this active selector (for all outgoing messages). 

The final step is to select a Selector and copy the text. You will need to add a new TXT DNS record with the text from it and put it in a subdomain _domainkey subdomain with the same name as the selector.     

• Mailbox configuration.

To create mailboxes, you need to use the administration console – “Messaging Manager > Post Offices > (postofficename) > Mailboxes”. Click on the “New Mailbox”, type in a name and password. Click ok and you are done.  

What is Apache James?

Apache James (Java Apache Mail Enterprise Server) is an open-source mail server. It is free under the Apache License 2.0 and completely written in Java

There is a small group of developers (currently 30) who work for the Apache Software Foundation, creating the Apache James mail server. The group is active and there are frequently new features added to the software. Just don’t expect the same level of complexity as MailEnable.

Apache James features 

• Supports SMTP, IMAP, and POP3
• jSPF, a java SPF
• jDKIM, a java DKIM
• FetchMail

Installing Apache James

• Installation.

It is a very different method of installation compared to the last examples. First, visit the download page and get the Apache James Server for your mail server. 

Then go to the installation folder, where you want it to be, and unpack it. That’s the installation process, now we need to configure it. This is the hard part.

You will need to install a database. 

Configure it. After unpacking, the configuration folder will be in ./conf/META-INF folder. There you need to change a few things: the postmaster address, open port 25 for SMTP, port 110 for POP3, and 143 for IMAP4. 

Create an A record

Add your domain, the default is localhost, you can change it.

Configure pop3 server by choosing the right port for it and you can add SSL too.

The Imap4 port should be set to 143. 

Configure the Smtp server to work on port 25. 

Run it. Go to the directory where you put the installation archive. Run the James Apache with the command $cd bin and then $ ./james start. 

Create a domain 

Use the command $ cd bin and $ james-cli –h localhost –p 9999 addomain mydomain.tld. Change the domain to your domain. 

Create users with the command $ james-cli –h localhost –p 9999 adduser myuser@mydomain.tld mypassword and respectively change the user and domain you want, followed by space and the password for the user. 

You can see additional components here. We recommend that you use at least jSPF and jDKIM. 

You can use it with a client like Mozilla Thunderbird. 

Firewall configuration

Let’s check if the ports that we have set in the software are open. If they are not, we can open then in Windows Firewall. Open it and go to “Inbound Rules”. There we will put “New Rule”. Choose “Port” and “Next”. Here write down the same port from before – 25, 110, 143, and 587 (or the one you have put). In the next window select “Allow The Connection” and “Next”. Select all of the available – “Domain”, “Private” and “Public”. In the last step, write down a name corresponding to the rule. For example, “Ports for the mail server”. 

Conclusion

Installing a fully working mail server is not that hard, as many will tell you. You can do it in a few easy steps, if you choose the right software. You can set it up and run it in a day.

Yes, you will need some basic knowledge about how servers work and what they need. But nothing is too hard to learn and there is enough information on the Internet with detailed instructions.

hMmailServer is a good solution for a mail server. It is free and relatively easy to use and there are enough professional features that can serve the needs of an average company.

MailEnable has a good enough Standard version, the free one. It is fine for those of you who need just an email server without additional features for sharing and cooperative work. It could get expensive in its Premium version, although not as much as most of the paid solutions.

Apache James is a smaller project. It is harder to set up and you will need more Java knowledge. The good part is it is free and there is a good chance it will remain that way for a long time.

If you want to check out a paid solution, Microsoft Exchange would be a solid choice.