Docker and Firewalls, who knew it could be this complicated ...

The Problem

This is by far one of the biggest issues I have had trying to scour the internet looking for a good way to deal with Docker and my hosts firewall. I have spent DAYS souring the internet trying to find someone who has solved this problem, because I know they have solved this problem. If there is one rule I have always lived by with the internet age, if you have come across a problem, it is almost guaranteed someone else had the same problem and came up with a solution.

I have tried using UFW, after all it is called Uncomplicated Fire Wall, you would think it would be uncomplicated. But while the solution I found for my firewall worked on some of my hosts, it didn’t work on all of them. For interest, this was the solution I came across for UFW.

Why didn’t the above solution work for me? Well it did work, there was just 1 massive flaw. When I enabled UFW, I was getting timeouts and extremely slow page loads with my Nginx webserver. I tried scouring the web looking for a reason as to why this was happening, and sadly no one had this problem before, or at least it was such an uncommon problem the answer is buried somewhere on the internet. I did say “almost guaranteed” someone else had the same problem.

The Solution

So how did I solve my problem? I did some more reading/searching and was able to find some information on DOCKER-USER within iptables whenever docker is started.

According to dockers help pages, which are a help, just not great at going in depth to solve the problem, it mentions the following:

Docker installs two custom iptables chains named DOCKER-USER and DOCKER, and it ensures that incoming packets are always checked by these two chains first.

All of Docker’s iptables rules are added to the DOCKER chain. Do not manipulate this chain manually. If you need to add rules which load before Docker’s rules, add them to the DOCKER-USER chain. These rules are applied before any rules Docker creates automatically.

Rules added to the FORWARD chain – either manually, or by another iptables-based firewall – are evaluated after these chains. This means that if you expose a port through Docker, this port gets exposed no matter what rules your firewall has configured. If you want those rules to apply even when a port gets exposed through Docker, you must add these rules to the DOCKER-USER chain.

What does this mean? It means that any rules you add to DOCKER-USER will be respected by docker, and can be modified by the user, ie you and me, to restrict/modify whatever we want to do to the rules inserted by docker when it first starts up or whenever a container gets added/removed.

So while I have modified what I have here, I have not come up with this solution. That all belongs to the many other people who had this problem and posted the solution for all the rest of us. Special thanks to unrouted for posting your solution.

Docker meet firewall - finally an answer

Integrating docker with your firewall has always been a pain. But those times are coming to an end!

unrouted

I highly recommend reading unrouted post to read up on why we are doing what we are doing.

Step 1: Create a iptables Config File to Store your Rules

Just as unrouted describes in his guide, lets create a new file located at /etc/iptables.conf

But just before we do that, there is one step that needs to occur, we need to find out what your Ethernet adapter is called for your server, where all your in/outbound traffic goes to reach the internet.

To do this type ifconfig your results may vary, but it should look something like this:

root@server:[~]> ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.18.206.89  netmask 255.255.240.0  broadcast 172.18.207.255
        inet6 fe80::215:5dff:fe90:8608  prefixlen 64  scopeid 0x20<link>
        ether 00:15:5d:90:86:08  txqueuelen 1000  (Ethernet)
        RX packets 681006  bytes 4284767481 (3.9 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 469723  bytes 30303159 (28.8 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 35760  bytes 168895398 (161.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 35760  bytes 168895398 (161.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

What you are looking for is your public IP on your server, in this example the public IP is 172.18.206.89, which means that my Ethernet adapter is called eth0. Now we can modify our /etc/iptables.conf.

*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:FILTERS - [0:0]
:DOCKER-USER - [0:0]

-F INPUT
-F DOCKER-USER
-F FILTERS

-A INPUT -i lo -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A INPUT -j FILTERS

-A DOCKER-USER -i eth0 -j FILTERS

-A FILTERS -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FILTERS -m conntrack --ctstate NEW -p tcp -m multiport --dports 22 -j ACCEPT
-A FILTERS -m conntrack --ctstate NEW -p tcp -m multiport --dports 80 -j ACCEPT
-A FILTERS -m conntrack --ctstate NEW -p tcp -m multiport --dports 443 -j ACCEPT
-A FILTERS -j REJECT --reject-with icmp-port-unreachable

COMMIT

This example is for a webserver which only has SSH and HTTP(s) running all through the TCP protocol. You will need to modify the ports by either adding or removing them to suit your needs.

Step 2: Test out the New Rules

Once you have them saved, we now need to test the new rules, just to make sure everything is okay. So lets give them a quick test. If something goes wrong you can either reboot the server or use a serial console to get direct access to your server to remove/fix the rules.

root@server:[~]> iptables-restore -n /etc/iptables.conf

If all went well, you should still be connected via SSH and and all your services should be running, it’s kind of a let down as nothing will actually happen if it works.

Step 3: Make the Rules Permanent

There is a small typo in unrouted’s guide, the location of where the service file you need to create is listed as /etc/system/system/iptables.service. The actual location of where the new file to create is /etc/systemd/system/iptables.service. Small typo but easily remedied.

Paste the following into your new file:

[Unit]
Description=Restore iptables firewall rules
Before=network-pre.target

[Service]
Type=oneshot
ExecStart=/sbin/iptables-restore -n /etc/iptables.conf

[Install]
WantedBy=multi-user.target

Enable the new service with the following commands:

root@server:[~]> systemctl enable iptables
root@server:[~]> systemctl start iptables

Alternatively, just learned this from unrouted’s post, you can do:

root@server:[~]> systemctl enable --now iptables

If you ever need to update your rules, just modify the file we created earlier /etc/iptables.conf and restart your iptables service:

root@server:[~]> systemctl restart iptables

Docker and Firewalls, who knew it could be this complicated …

The Problem

The Solution

Step 1: Create a iptables Config File to Store your Rules

Step 2: Test out the New Rules

Step 3: Make the Rules Permanent

Comments

Docker and Firewalls, who knew it could be this complicated …

The Problem #

The Solution #

Step 1: Create a iptables Config File to Store your Rules #

Step 2: Test out the New Rules #

Step 3: Make the Rules Permanent #

Comments

The Problem

The Solution

Step 1: Create a iptables Config File to Store your Rules

Step 2: Test out the New Rules

Step 3: Make the Rules Permanent