2013/06/12

Protect your Veeam backups from physical access to your repository

A feature that is not in Veeam is encrypted backups. The features is not one of the top requested features like tape but still, every now and then I get a mail asking on how you can store your backups in an encrypted way with Veeam. The short answer is, it is not possible. However with Linux repositories you can do some pretty neat stuff.

This blog article continues on my previous article "Veeam and Linux Repository" . What I will show you in this article is how you encrypt the home volume so that all your backups are stored in an encrypted way. If someone would steal your server, the data would be worthless without the key thus protecting you from physical access.

So lets continue. Just after you have configured the firewall, you can create the repo group
groupadd repos;
echo "%repos ALL=(root) NOPASSWD: ALL" >> /etc/sudoers.d/repos;
However just before you create the repo01 user, we will encrypt the home volume. To do this, you will need take the home volume offline. Also encrypting the volume will destroy all the data, so do this before you put the server in production or migrate the data first.

To put home volume offline, go to to the console and go to runlevel 1 so that all remote users and other users will be disconnected. This should clear all the file locks but will also disable networking so you really need to do this on the console and not via ssh. Afterwards we will switch back to runlevel 4
telinit 1
umount /home
telinit 4
Now check your /etc/fstab file and look for the logical volume that you want to encrypt. In my case it is  /dev/mapper/vg_repo-lv_repository


Then you can use shred to clear any existing data on the disk. If you are using thin provisioning in VMware this is not recommended
shred -v --iterations=1 /dev/mapper/vg_repo-lv_repository
Then you can encrypt the disk with cryptsetup and open it. This will create a new disk under /dev/mapper
cryptsetup --verbose --verify-passphrase luksFormat /dev/mapper/vg_repo-lv_repository;
cryptsetup luksOpen /dev/mapper/vg_repo-lv_repository encrypted_home;

You can check if the disk is properly mapped:
fdisk -l /dev/mapper/encrypted_home
Now that the disk is under /dev/mapper/encrypted_home, you can format the disk with ext4
mkfs.ext4 /dev/mapper/encrypted_home
Finally you will need to add some lines to crypttab and fstab so that the disk is mounted at boot
echo "encrypted_home /dev/mapper/vg_repo-lv_repository none" >> /etc/crypttab 
echo "/dev/mapper/encrypted_home /home                   ext4    defaults        1 2" >> /etc/fstab
You will also have to comment out or remove the line in /etc/fstab that is responsible for mounting the old unencrypted volume /dev/mapper/vg_repo-lv_repository


Now you can execute "mount -a" to mount the encrypted volume or just reboot the machine. During the boot, the machine will ask for a password to write and read from the encrypted volume:



Now that you have an encrypted home volume, you can create the user an add the repository to Veeam
useradd -m -G repos repo01;
echo "repo01:repo01" | chpasswd;

Now you are able to write backups to your encrypted volume

In my test, the repository was not the bottleneck, however I only have a limited lab environment so there might be some overhead when you try instant vm recovery or while running the backups. If in doubt add more CPU and Memory :)


2013/05/31

Veeam FLR and Linux searches

Veeam has an excellent framework for searching and restoring files for Windows. The one click file restore is a feature well appreciated by users. However the functionality is not available for Linux servers. I thought about this and came up with some possible solutions.

Using the Veeam FLR

The first possible solution is using the Veeam FLR appliance. In the end it is just a Linux appliance and guess what, you can just logon to it. You can find all the info you need on the KB (http://www.veeam.com/kb1447)

Once you are logged in you can use the "mount" commando to find out where Veeam mounts the partitions. This seems to be on the pretty standard location "/media". Then you can dive into those partitions and use "find" to locate your file. In the example below you can see I used the FLR to search for the file /tmp/processfollow


Once you know the path, you can go back to the Explorer and do the restore operation

Using the native mlocate method

Linux has a standard tool for doing Indexing. This tool called mlocate, can be easily installed on Redhat system by executing
$ yum install mlocate

To create your initial database, just run the updatedb commando. Then if you want to update the file you can use the updatedb command again
$ yum install updatedb

The great thing is that Centos for example automatically creates a daily cron job so that this updating is automatically. Just take a look at "/etc/cron.daily/mlocate.cron" . In the script you will also see that Centos uses renice and ionice so that the process of indexing does not take all the available resource.

Once you have a database, you can use "locate" to find a file. For example
$ locate findthisfile 


Multiple local versions of the index

The index or database is just a flat file you can find under "/var/lib/mlocate/mlocate.db" . In fact you could adjust your cron job so that it copies the index file and renames it using the current date. You can then use locate to find a file that might already be deleted from disk. You can see an example below. The copy statement is 

$ cp /var/lib/mlocate/mlocate.db  /var/lib/mlocate/mlocate-$(date +%y%m%d).db

Then you can use the "-d" parameter to find a file in an older index
$ locate -d  /var/lib/mlocate/mlocate-date findthisfile

In the screenshot below you will see a trial run, where you can see that I am unable to find a file in the current db but I am able to find the file in an old index


Multiple versions of the index on a remote server

The other great thing is that you seem to be able to copy those indexes to a central server and use locate to search for files that are or were on a specific server.

In my example I have a central server 192.168.149.55. I used the following statement to copy my index to this central server
$ scp /var/lib/mlocate/mlocate.db index@192.168.149.45:/home/index/$(date +%y%m%d)-$(hostname).db

Then I created a small script on this central server called lsearch
#!/bin/sh
INDEXDIR=/home/index
for curdb in $(ls -t /home/index/*.db)
do
        echo ">>>>> $curdb"
        locate -d $curdb $1
done

After chmoding it, I am able to search for a file using ./lsearch
$./lsearch findthisfile


Of course I am not sure about the performance with bigger machines and of course with more servers. The script itself is very very very basic but I do hope it might inspire some people to create nicer and better implementations.


2013/05/07

Veeam and Linux Repository

**** This blog article was tested with centos 6. However in centos 7, you might need to install an extra package via "yum install perl-Data-Dumper". You can follow the discussion on the Veeam forum. Credit goes to Tom Sightler for this update****

This post is a rather trivial post, but I just wanted to create a reference that you could use if you are in doubt on which distribution could work. Also I have some other blog post planned for which this would serve as an excellent start guide.

If you want to create a Linux repository and you look at the manual, it will only state
"Any major Linux distribution"
The FAQ on the forum gives a bit more detail stating
"Any storage directly attached to, or mounted on a Linux server (x86 and x64 of all major distributions are supported, must have SSH and Perl installed). The storage can be local disks, directly attached disk based storage (such as USB hard drive), NFS share, or iSCSI/FC SAN LUN in case the server is connected into the SAN fabric."

So I decide to test just a major distribution CentOS. The good thing is that they have a minimal version which only installs the bare essentials. In this post, I used "CentOS-6.4-x86_64-minimal". When you boot from the CD you can just go through the installer which is pretty self explanatory. Thus I'll only discuss some of the steps.


At boot I like to tab and add the kernel parameter resolution=1024x768. I've noticed that on a virtual console you are missing part of the screen if you do not do this.


I advice you to set the network settings via the gui. You can do it afterwards directly in the config file but it takes a bit more time. I also enable connect automatically, so that after the install, I can ssh directly to the new server.


For the storage option, select review and modify in the bottom left corner so that you have more control.


In this step you can see that I downsized the  root vol (/) to only 5GB. You will see that this is plenty. In fact I think even 3GB would suffice. I created another vol (/home) to store all the backups in. When your installation is configured it should only install a couple of packages and then you are ready to go.

When your machine is installed and rebooted, you should be able to SSH to the machine, if you configured the network. One thing I like to do is go to the config file of your network and change NM_CONTROLLED=yes to NM_CONTROLLED=no. You can edit the file via
vi /etc/sysconfig/network-scripts/ifcfg-eth0


Then when this is done you should install the software that is required. First do an update of the system
yum update -y
Now install client and server openssh. This should already be done but just in case
yum install -y openssh openssh-clients openssh-server
You will require sudo for elevating the rights. Notice this should also already be ok
yum install -y sudo
Finally install perl. This is not included in the base install
yum install -y perl
If you install it in a VM, for a test, this is the moment you can install VMware Tools. I have this oneliner for installing VMware tools.
mount /dev/cdrom /mnt;tar -xzf /mnt/VMware*.tar.gz -C /usr/src/;/usr/src/vmware-tools-distrib/vmware-install.pl

Now we will configure the firewall. Veeam requirements say that you need to open certain ports . Mainly this should be

  • 22 tcp (ssh)
  • 2500 - 5000 tcp

The easiest way at this point (so that you don't have to install extra components) is manipulating the iptables file directly. Just use vi to edit "/etc/sysconfig/iptables" and add the following line between "ssh (--dport 22) " and "-j Reject". The order is important!
-A INPUT -m state --state NEW -m tcp -p tcp --dport 2500:5000 -j ACCEPT

Then you can restart the system
service iptables restart
For this tutorial we will create a separate user for the repository. Of course since you allow the user to elevate to sudo it is not 100% secure but it offers a bit of seperation. If you want real seperation, I advise you to create multiple systems or multiple chroot environments.

In the following step, we will create a group repos. Then we create a user repo01 that is part of this repos group and change the users password. Finally we add the repos groups to a sudoers file so that you don't have to modify the original file nor do you need to allow Veeam to manipulate it.

groupadd repos;
useradd -m -G repos repo01;
echo "repo01:repo01" | chpasswd;
echo "%repos ALL=(root) NOPASSWD: ALL" >> /etc/sudoers.d/repos;


Now you should be able to add the repository to Veeam. By default you will see that we are not using too much space. In my example the lv_root was only used for 28% (1.3GB of 4.9GB)

Some interesting screen shots while adding the repository. You can see I don't use the root account because repo01 can elevate its right. You don't need to allow Veeam to alter the sudoers file as this is already been done

You can also see that we have opened up all the required ports


In the repository step, I just used the home folder of repo01 so that the backups are stored nicely in this separate container.


During backups, you will see that Veeam will automatically push and start the necessary agents



2013/04/04

Surebackup Network Deepdive

In this post I will deepdive into the Surebackup networking part. I can imagine that this might be to deep for some people but at least it should serve as a good reference how Surebackup vlab works.

But first things first. If you don't know what SNAT, DNAT or Netmap means, I advise you to read my previous post. Understanding this post is very crucial for understanding how Surebackup works internally. Even if you know those concepts, I advise you to read the last part of the post describing how to connect overlapping subnet ranges as this is the base for Surebackup.

The simple network

We will first start by creating a very simple Surebackup network. This particular Veeam customer has only one network, his production range.  So lets put the network parameter is in a clean table. This is very interesting information as it contains everything we need to set up the Virtual Lab

NamePortgroupVLANSubnetNetmaskDefault GatewayDNS
ProductionVM Network0192.168.149.x255.255.255.0192.168.149.2192.168.149.20

Now for every production network you want to use in Surebackup, you will need a subnet range that is not being used and has the same subnet size. How do you know which production networks you are going to use. Take a look at the VMs you want to test in Surebackup and write down in which network they are connected.  In this case it is only one network

Subnet MaskProduction SubnetSubnet not in use anywhere on the network
255.255.255.0192.168.149.x192.168.150.x

You could also use a different private range in a different private address space . This will allow you to make a clear distinction between production and your Surebackup network. For example:

Subnet MaskProduction SubnetSubnet not in use anywhere on the network
255.255.255.0192.168.149.x172.16.149.x
255.255.255.0192.168.129.x172.16.129.x

The Goal

Before I start explaining how to configure everything, let me first start by explaining what our final goal is. I think 99% of all problems with Virtual Labs occur because users don't know what the final result should be.

With Surebackup we will start up Virtual Machine directly from a backup in an isolated network. So what is an isolated network? An isolated network is a network that mimics a production network. It means that VMs will reuse the same network settings as listed up in our table earlier. This portgroup or network will be created automatically on a vSwitch without any uplinks by Veeam. You could say that Veeam creates a network sandbox.

Imagine a server SF0006 with IP 192.168.149.36 . It is running in the production network. When Surebackup is configured, a copy of this network is created: the isolated network. The backup copy we want to test will be started in this isolated network. The result is something like this. Notice that I added the IP of the Veeam Backup Server

VM Network (has uplinks on vSwitch)
  • SF0006 : 192.168.149.36
  • Default gateway : 192.168.149.2
  • Veeam Backup Server : 192.168.149.22

vLab VM Network (has no uplinks on vSwitch)
  • SF0006_Backup_Copy : 192.168.149.36
This is already a great start. The only thing is that we can not talk to our "SF0006_Backup_Copy" machine as it is isolated. This is where vLab networking comes in to play. Veeam will deploy a small Linux NAT Router. The router will sit between the Production Network and the Isolated vLab VM Network.

In the production network the vLab router will just need to get any available IP. You can use a DHCP address but I recommend using a fixed IP. In my case I choose 192.168.149.50. So let me update the VM Network

VM Network (has uplinks on vSwitch)
  • SF0006 : 192.168.149.36
  • Default gateway : 192.168.149.2
  • Veeam Backup Server : 192.168.149.22
  • vLab router interface 0 : 192.168.149.50
Then in the isolated network we will also have to choose and IP address. However the choiche is very easy. If a virtual machine wants to contact the outside world it will use its default gateway. The backup copy is not aware that it is started in an isolated environment. When it runs, it will also want to talk to the default gateway to send out traffic. So in the isolated environment our vLab router will mimic the default gateway that is running in production. Our vLab VM Network now looks like:

vLab VM Network (has no uplinks on vSwitch)
  • vLab router interface 1 : 192.168.149.2
  • SF0006_Backup_Copy : 192.168.149.36

Now if the Veeam server or the production machine wants to talk the backup copy it will just send packetsto vLab router interface 0. This vLab router can then forward or route the package to the isolated environment. What is more interesting is that you have overlapping subnets. So you will need to do some form of NAT to hide this isolated environment behind another subnet.

Lets see how it looks in VMware. First of all  you will see 3 running machines. The first one is the production machine SF0006. The second one is the backup copy. Notice that we won't use the name sf0006_backup_copy but rather sf0006_insertveryrandomhashhere. The last one is the vlab network

Now lets take a look at SF0006. Like stated before it is has the IP 192.168.149.36 and is connected to the VM Network

The backup copy of SF0006 also uses the IP 192.168.149.36. However it is connected to the vLab VM Network

 The vLab router itself has 2 IPs. One of those IP is 192.168.149.50 in the VM Network. You can also see that it is connected to the vLab VM Network

If we look at the ESX networking, you can see clearly that the isolated network has no way to talk with the outside world, except via the vLab router

The Configuration

So now we know what the result should be, lets see how you can configure it. You will need to go to the backup infrastructure and add a new virtual lab

Give the Virtual Lab a name. I like vLab because it is short

The vLab router is a Linux appliance. It is a virtual machine so you will need to select a vSphere host where you want to run it on. Important, all the networking is only created on this vSphere host. All the machines that will be test will be powered on on this host.

Select a datastore to store the Linux appliance on

Now configure the router so that it has an IP in your production network. Again this is the free IP we reserved for the virtual machine. 

Instead of using DHCP, we choose the static IP 192.168.149.50 . This IP will be set on interface 0 and will be entry point for all packets coming from production.


 Then in the next step choose manual configuration. If you made it this far, you know what you are doing.

Now lets create an isolated network. In our case we only need to mimic one portgroup called VM Network. However if you would have multiple networks (Production, DMZ, ...) and you need those to test your VM's, you will need to create a copy or isolated network of each production network . I will show you this in a later post.

For each isolated network you will have to define a vNIC or interface in that isolated environment. Remember that our Linux router will mimic the default gateway of production.

So in the settings for the isolated network vLab VM Network, configure the default gateway 192.168.149.2 of production. The vLab router will then mimic the gateway in this isolated network. Interesting enough you will now have to configure a "Mask". Basically you will configure a subnet that does not exists in your environment and that will hide away or mask your isolated environment. What you are actually doing is creating a NETMAP rule in the Virtual Lab router.

I choose 192.168.150.x because I have only one network. However I can not stress this enough, this mask should be unique in your network to avoid problems.

Just skip static mapping for now. I will cover it later.

And your are setup.

The Result

When you are running a Surebackup job the result will be something like this (our goal)


VM Network (has uplinks on vSwitch)

  • SF0006 : 192.168.149.36
  • Default gateway : 192.168.149.2
  • Veeam Backup Server : 192.168.149.22
  • vLab router interface 0 : 192.168.149.50
vLab VM Network (has no uplinks on vSwitch)
  • vLab router interface 1 : 192.168.149.2
  • SF0006_Backup_Copy : 192.168.149.36

Remember that we masked or did a NETMAP for the vLab network. So any production VM that wants to talk to SF0006_Backup_Copy from the VM Network will not use it regular IP 192.168.149.36 but will use his masked version 192.168.150.36. You can see this in the screenshot that I am able to ping the machine succesfully

There is only one problem. The Veeam Backup Server (VBS) its default gateway is set to 192.168.149.2. So if it wants to talk to 192.168.150.36, it would talk with that default gateway. The default gateway is not aware of the situation and just drops the packets.

So how do we fix this. Well it is automatically fixed. If you run a surebackup job or you run an U-AIR Wizard, Veeam will automatically add static routes on the Veeam Backup Server (VBS) or the machine running the U-AIR Wizard. You can see this in a console screen using the command "route print".

Veeam has added a static route saying that traffic for subnet 192.168.150.0/24 should be forwarded to 192.168.149.50 which is our vLab router interface 0. When the VBS wants to talk with 192.168.150.36, it will send a package to our vLab router and the traffic will be translated.

Deepdive

I've shown you how to configure the vLab router but lets see what happens underneath. If you have not read my previous post, please do so.

One thing we discussed in the NAT post is how you can manage overlapping subnet. This is exactly how the vLab router solves this same challenge. So lets take a look under the hood. If we run ifconfig we can see the IPs set on the interfaces. In this case

  • Interface 0 : eth0 : 192.168.149.50
  • Interface 1 : eth1 : 192.168.149.2




First lets look at the NAT rules. There are two important NAT rules. The Netmap rule and the Masquarade rule

The Netmap rule in Pre Routing stage
When?TypeComing from InterfaceExit InterfaceOriginal DestTranslation Dest
Pre RoutingNETMAP eth0 / Interface 0*192.168.150.x/24192.168.149.x/24

The Masquerade rule in Post Routing stage
When?TypeComing from InterfaceLeaving on InterfaceOriginal Source
Post RoutingMasquerade*eth1 Interface 10.0.0.0/0 = everything

So lets follow a packet coming from our backup server 192.168.149.22. It enters on interface 0 and matches the netmap rule. The destination is translated
IP Packet
Source192.168.149.22
Destination192.168.150.36 192.168.149.36
MessageHello from Backup Server

Again the routing will be solved by marking. I will show this later on. The packet is forwarded to interface1. There the Post Routing Masquerade rule kicks in. It replace the source IP with the the IP set on interface 1
IP Packet
Source192.168.149.22 192.168.149.2
Destination192.168.150.36 192.168.149.36
MessageHello from Backup Server


SF0006_backup_copy will be able to receive the message and respond back to the router. The router will reverse the whole sequence and the package is delivered

So how is the marking set up? Well this is a bit tricky and I hope I am explaining it right.

When a package enters on interface 0 (eth0) it will be marked using a value 0x6 with bit mask 0xffffffff (32-bit) if the destination is 192.168.150.x/24. Remember, this rule is applied before NAT rules are applied

Then if we look at the ip routing tables we will see that the all the traffic  should leave via interface 0 / eth0. As default gateway the production router is configured.

If we look at the ip rules, we will see that an fwmark is set. It say that all trafic matching 0x2 with bitmask 0x2 should use an alternative table 2. 

Now this is a bit tricky as you would expect that the data would be marked with mark 6. However fwmark is implying a bitmask 0x2. This bitmask works like a filter, only allowing bits to pass if the bitmask has that bit set to 1

So lets convert the last byte to binary
0x6                  = 0000 0110
bitmask 0x2     = 0000 0010
Result 0x6/0x2 = 0000 0010

0x2                  = 0000 0010
bitmask 0x2     = 0000 0010
Result 0x2/0x2 = 0000 0010


You can see that 0x6/0x2 will now match 0x2/0x2 and so the routing table 2 is chosen.

If we take a look at this alternative table 2, we will see that indeed it say to forward traffic for 192.168.149.0/24 via interface 1 / eth1 towards our isolated network

Static Mapping

Now that we know how the basic settings work we can look at static mapping. Static mapping is an alternative on top of the Netmap. Lets look for example at our server SF0006_backup_copy (192.168.149.36). If we want to reach it we will have to connect to 192.168.150.36. Our computer knows due to static routes that it must send packets for 192.168.150.36 to our vlab router 192.168.149.50 

If other clients in the same subnet want to talk with this server they will have to manually add the static routes. There is however another way. If for example in production you have a Free IP 192.168.149.136 you can map this IP to our server in our isolated environment. Other clients can just connect to 192.168.149.136 and the router will do the translation to our SF0006_backup_copy (192.168.149.36)

Static mapping is part of the virtual lab configuration. You can see I have enabled it

 I added a mapping in VM Network so that the production IP 192.168.149.136 will be mapped to the isolated IP 192.168.149.36

The result will be that SF0006_backup_copy is reachable on 2 addresses
  • 192.168.150.36 via the Netmap rule
  • 192.168.149.136 via Static mapping


If we look under the hood a couple of extra rules will be added


Static mapping adds 2 rules, one DNAT & one SNAT
The DNAT rule in Pre Routing stage
When?TypeComing from InterfaceExit InterfaceOriginal DestTranslation Dest
Pre RoutingDNAT eth0 / Interface 0*192.168.149.136192.168.149.36

The Masquerade rule in Post Routing stage
When?TypeEnter InterfaceExit InterfaceOriginal SourceTranslated SourceDestination
Post RoutingSNAT*eth1 Interface 1*192.168.149.2192.168.149.36

In 2 stages the source and destination will be rewritten
IP Packet
Source192.168.149.22 192.168.149.2
Destination192.168.149.136 192.168.149.36
MessageHello from Backup Server

An additional mark rule will be created so that not only traffic going to 192.168.150.x is marked with 0x6 but also traffic going to 192.168.149.x



Reference:

NAT 101

NAT

In this post I will try to explain as simple as possible how NAT works. This post is actually a prequel to my next post which will talk about Surebackup and Networking. But before I can go into detail in that post, you need to understand how NAT works on Linux (as this is the core). If you already know how SNAT, DNAT and NETMAP work please ignore the post.

So what is NAT or Network Address Translation? Well NAT is actually a technique which is used to modify the source or target IP address in a IP packet. NAT is mostly done on Routers when a packet transfers from one subnet to another subnet.

NAPT

One of the most common use cases of NAT is NAPT . When you have a broadband connection at home, your provider will probably only assign you one public IP. So when you want to use this connection with multiple devices you put a "router" in between. Basically this home router does NAPT between your local network and the internet.

Let us start with a simple example. You have a computer connected on your local LAN. Your router is the default gateway in this LAN but is also connected to the Internet. Your computer wants to get a page at 173.194.35.19. So the setup of IP's would look something like this:

  • Your computer 192.168.1.2 
  • Router LAN 192.168.1.1
  • Router WAN 109.132.94.20
  • Internet GW 109.132.94.1
  • Website 173.194.35.19


Remember that 192.168.x.x is a private range and has no meaning on the internet itself. So when your computer sends a package to request the website the following conversation will happens:

Your Computer send the following packet to the Router LAN
IP Packet
Source 192.168.1.2
Source Port 29831
Destination 173.194.35.19
Destinatin Port 80
Message Get index.html

Your Router will get this package. It only has one default gateway itself (the one given by your provider). It wants to forward this package to this default gateway but it knows that 192.168.1.x is a local subnet. So it translate the source to its own ip. While doing this it tracks that it send out this package for the internal server. With NAPT this tracking is done on port base. Basically it modifies the Source Port and keeps a table of the "original settings" linked to the source port and destination. It "tracks" the connection

Router WAN send a modified version of the package to Internet GW
IP Packet
Source 192.168.1.2 109.132.94.20
Source Port 29831 30001
Destination 173.194.35.19
Destinatin Port 80
Message Get index.html

In a "Translation table" it puts
Translated Source PortDestinationOriginal Source PortOriginal IP
30001173.194.35.1929831192.168.1.2

So when a package returns from the website via your Internet GW the following happens on the router. It will inspect the package and match it to his translation table based on the translated source port and destination ip (which is now the destination port and source ip). There it can see that the final destination is not itself but is an internal IP. It will translate the port and the destination back and put the package back on the lan

Internet GW > Router WAN
IP Packet
Source 173.194.35.19
Source Port 80
Destination 109.132.94.20 192.168.1.2
Destinatin Port 30001 29831
Message Random HTML

Router LAN > My computer
IP Packet
Source173.194.35.19
Source Port80
Destination192.168.1.2
Destinatin Port29831
MessageRandom HTML

This basically show how NAPT works. But of course there are multiple forms of NAT. There are a couple of interesting things to get from this example

The first interesting thing here is that the NAT rules are applied before the package leaves the ROUTER Wan port. However this is not always the case. Inside the router NAT can be done on two locations.
Lan interface or Entering interface (Pre Routing) > Routing (computer decides the package should go the internet default gateway) & Firewall  >  Wan Interface or Exiting interface (Post Routing)

Another interesting thing is that although the computer in the local network can talk to the outside world, the outside world can only talk back when the router has a way to translate the package back into the local LAN. In fact one of the reason why people could use one form of NAT is a form of security. Because the router intercepts all the traffic it hides away the local network specifics and can actually act as a firewall. In fact in Linux, NAT is configured via iptables, the same command that is used to configure the built-in firewall.


Linux and NAT

Now that you know what NAT does, lets look at some other forms of NAT in specific to Linux.

For the following example I will use the following networks. You can see that the Router has two interfaces: 0 and 1, one in each network.

Network A (192.168.1.x)
  • Computer X with IP 192.168.1.2 
  • Router interface 0 192.168.1.1

Network B (109.132.94.x)
  • Router interface 1 109.132.94.20
  • Server Y with IP 109.132.94.60

Source NAT (SNAT)

With Source NAT you will actually define a NAT rule that will translate a Source IP to another IP. Source NAT is defined as a Post Routing process.

So lets imagine the following rule
When?TypeComing from InterfaceLeaving on InterfaceOriginal SourceTranslation Source
Post RoutingSNAT*Interface 1192.168.1.0/24109.132.94.20

When  computer x sends a package to server y it will pass the router. Just before it leaves on interface 1, the source will be translated.
IP Packet
Source192.168.1.2  109.132.94.20
Destination109.132.94.60
MessageHello from Computer X

When a package comes back, the address is translated back
IP Packet
Source109.132.94.60
Destination109.132.94.20 192.168.1.2
MessageHello back from Server Y

Interesting to know is that source NAT can be done on a single IP as well, meaning that the original source address might be only be 192.168.1.2. In this case only 192.168.1.2 would be translated but no other hosts.

You can also see why SNAT should be a Post Routing process. If  it would be a Pre Routing / Firewall process, the source would be the router itself. It would be hard to make firewall rules for the original host as this information is already replaced.

Masquerading

Source NAT does look a lot like NAPT. This true but there is one problem. Your network interface at the WAN side might change IP. Most providers assign public ip addresses to home users via DHCP. So it is hard to predict what the outside ip will be. Masquerading solves this. It is a special form of Source NAT where instead of defining a fixed ip it will define only an interface.
When?TypeComing from InterfaceLeaving on InterfaceOriginal Source
Post RoutingMasquarding*Interface 1192.168.1.0/24

In this case the same thing will happen as in the previous example. The router will use the dynamic IP assigned to Interface 1 (in this case 109.132.94.20) and will rewrite the source. If the connection is reset and  interface 1 gets a new IP, the rule will dynamically use this new IP.

Destination NAT (DNAT)

DNAT is the inverse of source NAT. In fact it will rewrite the Destination address instead of the source address. A lot of companies use DNAT to hide their real server in a private range behind a Firewall / Router.

Consider the following example:

Network A (192.168.1.x)

  • Server X with IP 192.168.1.2 
  • Router interface 0 192.168.1.1
Network B (109.132.94.x)

  •  Router interface 1 109.132.94.20
  •  Client Y with IP 109.132.94.60

Notice that the server is now in the private subnet. There is one problem, the server is not reachable from the outside because of the use of private IPs. The ideal scenario would be that all traffic pointed to 109.132.94.20 is translated to 192.168.1.2 when it enters the router.

In this case we will do the translation in the Pre Routing stage. Since the connection is now initiated from the outside Pre Routing is now on Interface 1 (Pre Routing Interface 1 > Routing > Post Routing Interface 0) . If the translation would be done in the Post Routing process, the routing process would think that the packet with destination 109.132.94.20 would be for a service running on the router itself.

When?TypeComing from InterfaceLeaving on InterfaceOriginal DestTranslation Dest
Pre RoutingDNATInterface 1*109.132.94.20192.168.1.2

So how does the communication go?

Client Y sends a package to router interface 1. Router interface 1 will translate it before sending it to server X
IP Packet
Source109.132.94.60
Destination109.132.94.20 192.168.1.2
MessageHello from client Y

Before the answer is send back to Client Y, the address is translated back to the original IP
IP Packet
Source192.168.1.2 109.132.94.20
Destination109.132.94.60
MessageHello back from Server X

Of course if you have multiple server in DMZ and multiple public IP address you can map them on the same router. Most of the time what happens is that the router will claim multiple IP address on the same interface. In that case you will have to write multiple DNAT rules. Consider:

Network A (192.168.1.x)

  • Server X with IP 192.168.1.2 
  • Server Z with IP 192.168.1.3 
  • Router interface 0 192.168.1.1

Network B (109.132.94.x)

  • Router interface 1 109.132.94.20,109.132.94.21
  • Client Y with IP 109.132.94.60

When?TypeComing from InterfaceLeaving on InterfaceOriginal DestTranslation Dest
Pre RoutingDNATInterface 1*109.132.94.20192.168.1.2
Pre RoutingDNATInterface 1*109.132.94.21192.168.1.3


Redirection

This is a special form of DNAT comparable to Masquerading for SNAT. If you have a dynamic IP on the outside, you can use redirection to translate all the packages that are coming in to an internal destination without knowing the original destination (IP on Interface 1). The rule will be dynamically adapted. In this case the original destination 109.132.94.20 will be used. However if the connection is reset and the outside IP on interface 1 changes to 109.132.94.30, the rule will use 109.132.94.30 as the original destination.

Basically if you want to run a server in your home network, you will have to configure redirection on your home router.


When?TypeComing from InterfaceLeaving on InterfaceTranslation Dest
Pre RoutingRedirectionInterface 1 *192.168.1.2


SNAT vs DNAT

So when should you use SNAT or when DNAT. Well this depends on the network you want to hide and the server its position. If the server is in the network you want to hide, you should use DNAT. If the client is in the network you want to hide you need a form of SNAT.

One interesting this that is not discussed is that you can create NAT rules that are only valid for a certain port. Even more you can use NAT to not only translate the IP but also the Port part. Consider the DNAT rule. If you have a webserver running on port 8080 on server x and an smtp server on server y that you want to hide behind NAT rules, you create two rules:
  • Redirect source port 80 to port 8080 on server x
  • Redirect source port 25 to port 25 on server y
In this case you could use one Public IP that is redirected to an internal server based on which destination port was used.

What is also interesting is that I described all scenarios with a public network and a private network. However you could also use NAT between private networks. But again one network will be hidden from the other one.

NETMAP

Before I explain NETMAP know that an IP has a network part and host part uniquely defining a host in the network. For example if you have an IP 192.168.1.1 with subnet mask 255.255.255.0, the network part would be 192.168.1 and the host part would be 1.

In the previous cases you can see that for DNAT we need to define a fix Destination IP. For SNAT you need to define a fix Source IP. What if you wanted to translate a whole network at once. Let say that for all IPs coming in you want to translate the network part but not the host specific part. In that case you can use NETMAP.

NETMAP can be used as a substitute for DNAT and SNAT. If you use NETMAP in Pre Routing it will rewrite the destination so it will do DNAT. If you define it as a Post Routing rule, it will rewrite the source address thus acting like an SNAT rule

To see the use case consider the following DNAT example

Network A (192.168.1.x)

  •  Server A with IP 192.168.1.2 
  •  Server B with IP 192.168.1.3 
  •  Server C with IP 192.168.1.4 
  •  Router interface 0 192.168.1.1

Network B (109.132.94.x)

  •  Router interface 1 109.132.94.1, 109.132.94.2, 109.132.94.3, 109.132.94.4, 109.132.94.5 
  •  Client Y with IP 109.132.94.60

We want to map the outside IP to the corresponding inside IP address.

  • 109.132.94.2 > 192.168.1.2
  • 109.132.94.3 > 192.168.1.3
  • 109.132.94.4 > 192.168.1.4

You can do this using 3 DNAT rules. However each time a new server is deployed you will need to add another rule. A more lazy way of doing this is using NETMAP. To use NETMAP however the SUBNET mask must match -which is the case-.

When?TypeComing from InterfaceLeaving on InterfaceOriginal DestTranslation Dest
Pre RoutingNETMAPInterface 1*109.132.94.x/24192.168.1.x/24

The same things happens now as using standard  DNAT rules. Only the NAT process will only translate the host part.

Client Y sends a package to router interface 1. Router interface 1 will translate it before sending it to server X
IP Packet
Source109.132.94.60
Destination(109.132.94 192.168.1).2
MessageHello from client Y

Before the answer is send back to Client Y, the address is translated back to the original IP
IP Packet
Source(192.168.1 109.132.94).2
Destination109.132.94.60
MessageHello back from Server X

If a new server is deployed with ip 192.168.1.5 and the router has the matching public IP claimed to interface 1, it will automatically do the translation.


Overlapping SUBNETS

Another use case for NETMAP could be if you have duplicate subnets in different VLANs. For example if you have a VLAN 2 with subnet 192.168.1.x/24 and a VLAN 3 with subnet 192.168.1.x/24 there is no way of connecting this VLANs via normal routing. What you could do is use Netmap to translate one of the networks

Network A (192.168.1.x) VLAN A

  •  Client X with IP 192.168.1.2 
  •  Router interface 0 192.168.1.20


Network B (192.168.1.x) VLAN B

  •  Server A with IP 192.168.1.2 
  •  Router interface 1 192.168.1.1


When?TypeComing from InterfaceLeaving on InterfaceOriginal DestTranslation Dest
Pre RoutingNETMAPInterface 0*192.168.2.x/24192.168.1.x/24

If Client X wants to talk to Server A it will need to know that the router will translate the address from 192.168.2.x addresses to 192.168.1.x addresses. So Client X will not try to connect to Server A on address 192.168.1.2 but on 192.168.2.2.

Another interesting problem is that this will translate the Destination IP but not the Source IP. This can be confusing for the Server A as the Source IP is in the same range. To solve this problem we could add an SNAT rule or just masquerading.

When?TypeComing from InterfaceLeaving on InterfaceOriginal Source
Post RoutingMasquarding*Interface 1192.168.1.0/24

In this case the Router will replace the original source with its own IP forcing Server A to reply back to the router. So lets see the conversation

In Pre Routing the following package enters the Router on Interface 0. The NETMAP will do a translation
of the destination IP
IP Packet
Source192.168.1.2
Destination192.168.2.2 192.168.1.2
MessageHello from Client X

One if the interesting problems now is that the Routing will be confused because it has two networks that both have the Subnet 192.168.1.x. So how does the routing process know which is the correct Interface for the packages destination. This can be done using marks. Basically you mark a packet when it enters the router. The mark is only relevant during the stay in the router. For example you can mark all traffic that comes in on interface 0 with mark 2. In fact the Mangle / Mark rule is processed before the Netmap rule will apply so you can use the original target IP Address range. The router can then be set up to route correctly based on this mark

When?TypeMarkComing from InterfaceLeaving on InterfaceSourceTarget
Pre RoutingMangle / Mark0x2Interface 0*0.0.0.0/0192.168.2.0/24

IP Packet
Source192.168.1.2
Destination192.168.2.2 192.168.1.2
MessageHello from Client X
Mark2

Now all packages entering via interface 0 going to 192.168.2.0/24 are marked with a 2 and forwarded correctly if the routing tables are setup correctly. If you don't care how this is done, just skip the following indented paragraph.

When you mark packages and you want those package to select another route then the default routing table, you will need to create an IP rule
   ip rule add fwmark 0x2/0x2 lookup 2

Basically all the packets with mark 2 will be routed using another routing table as the default called "2" Then you will need to add routes to this alternate routing table
   ip route add 192.168.1.0/24 dev eth1 table 2

You can query the result using
   ip route list table 2

It should show you something like
   192.168.1.0/24 dev eth1 scope link

If the default routing table send all the traffic to eth0, your problem is solved. All the traffic generated on interface 0 that has to be NAT will be marked. During routing, based on this mark the router will use an alternate routing table. This will state that the traffic should be send via eth1. If the traffic is occurring from interface 1 it will not be marked. Hence the routing process will use the default table and forward the packet to interface 0.

Routing will be done and finally the package will enter the Post routing stage just before it is send out on Interface 1
IP Packet
Source192.168.1.2 192.168.1.1
Destination192.168.2.2 192.168.1.2
MessageHello from Client X

The Server will receive the package and will reply. It will send over the reply to the router; There again, the reverse process will rewrite the package in an inverse order
IP Packet
Source192.168.1.2
Destination192.168.1.1 192.168.1.2
MessageServer A at your service

IP Packet
Source192.168.1.2 192.168.2.2
Destination192.168.1.1 192.168.1.2
MessageServer A at your service

Interesting in this scenario is that although Client X can start the conversation to Server A, Server A can only talk back. Server A is thus in the isolated environment.