2013/12/13

Miniblog Series P001 : Where is the main menu in Veeam Backup & Replication?

Just to have some fun I will start some miniblogs. Just small questions I often get asked, maybe just to call it my personal "Screenshots FAQs". Instead of remaking the screenshots all the time, I will just put them online for everybody to enjoy! :)

In our manual, we will sometimes refer to the main menu. I had some cases where people where unable to understand what the main menu is. Well it is just the blue button in the top left corner of the UI :)


2013/12/09

Veeam One and Backup Servers : HRESULT: 0x80070005 (E_ACCESSDENIED)

One of the nicest thing Veeam One 6.5 added is the support to monitor and report on your backup infrastructure. In Veeam One 7 the possibilities are now expanded. For example a lot of reports for Backup & Replication has been added. For the monitoring part, you can now really monitor CPU/MEM/Network via One console. One will collect the data via WMI so it is a pretty standard approach.

It is really interesting as I often have customers telling me that B&R reporting is limited. Then I will show them Veeam One reporting and they will be blown away. Sometimes it even happens that the customer has a Veeam Essentials license and that he is unaware of the fact that he already has the licenses for One. Basically Veeam Essentials is the Veeam Backup Management Suite but limited to 6 sockets.

To start using it just go to the Data Protection View, right-click the Backup Infrastructure node in the inventory pane and choose Add Server from the shortcut menu. I would recommended adding the enterprise manager so that all your backup servers are automatically added.

When you add Veeam Backup & Replication to the Veeam One server you will use a certain user to connect to the backup server. Most cases it will be a separate service user. In my case I use "vlab\srvone".

Following the docs this "srvone" user should have local administrator rights on the backup server:
The account must have local Administrator permissions on the backup server and on all servers that run backup infrastructure components. For details on required permissions for Veeam backup servers
http://helpcenter.veeam.com/one/70/vsphere/backup_add_server_select.html
http://helpcenter.veeam.com/one/70/vsphere/backup_specify_server_creds.html

However I was getting an error on some of the proxies ( HRESULT: 0x80070005 (E_ACCESSDENIED) ). Basically One is unable to connect via WMI:

This make sense as Veeam One uses the same credentials you specified when adding the enterprise manager. You can check the credentials used by right clicking the enterprise manager and clicking "Change Connection Settings"


So if you are in this case, it probably means that srvone is not added to the local administrator group on your proxy servers. There are 2 things you can do. First of all you can right click the component and override the credentials for that certain proxy or repository. I had to do this for a remote repository / wan accelerator that is not in the domain.
http://helpcenter.veeam.com/one/70/vsphere/changing_connection_for_backup_servers.html

You could also of course add the user to the local administrator group. However I have some core servers in demo. So here is the oneliner you can use to add the user to the local admin group via cli
 net localgroup Administrators /add vlab\srvone


After this is done, you can resolve the alarm on the server. After 15m you should see WMI data flow in on the Network tab for example


For my lab I also have this oneliner I always use to disable the firewall. Might be irrelevant to this article but still in demo setups I always use it to make sure it is not a firewall issue I'm running into
 netsh advfirewall set allprofiles state off

2013/11/25

After the backup copy job, the auto import

For me personally the most interesting feature in Veeam v7 would be the backup copy job. Why? Well it solves one of the most important challenges Veeam users were having with backup policies.

First of all you can now do tiering of your backups. Start by creating fast backups on fast disks with a limited amount of restore points, then use the backup copy job to copy the data to slower disks for a longer retention. Before this was also possible ... with scripts!

Second of all, you can now apply GFS like retention policies. I like that GFS is only available on the backup copy job. It forces people to think about tiering in combination with GFS to slow disks, so that you still have a (limited) amount of fast restore points to do Instant VM Recovery and Surebackup. Before this was also possible ... with scripts!

Lastly WAN acceleration is now built into the product. People were trying to get there backups off-site with previous versions but maybe not always in the most successful way. People were using of course scripts to RSYNC & Robocopy to get the backups shipped to a second location but didn't always liked how much bandwidth this required or the manual actions they had to take. Now you can get your backups off-site via very small connections. Best of all, you no longer require somebody to manually export the tapes and take them home everyday so that your corporate data is safe.

But what about those off-site copies? Getting your restore points off-site is the easy part. You can just use the backup copy job. Actually there are 2 ways you can get your backups of site.

In the push strategy, you will install the Veeam B&R Management Server on the source location. Backups will be made by a source proxy to a source repository. Then the backup copy job will send the data to the remote location. This is mostly used when your IT staff is working on the source location. One disadvantage about this scenario is that you can not share WAN accelerators between management servers in v7. Since every location is running their own Management Server, you will have to install multiple Windows servers for each WAN accelerator instance. However, connection outages won't result in backups not running locally.

In the pull strategy, you will install the Veeam B&R Management Server on the remote location (or HQ in a ROBO design). You will have a source proxy and a source repository. Local backups will be made locally but scheduled by the remote location. The backup copy job will then copy the data to the remote location. This scenario is mostly used when you have centralized IT and you have very stable connections between HQ and your branch offices. In this case, because the Management Server is running centrally, you don't need to deploy a windows server for each pair of WAN accelerators. In fact, you can profit from the fact that if you set up a new WAN accelerator pair, the server will copy the cache from an existing WAN accelerator on HQ.

In the push strategy your local and remote backups will be visible as restore points at the source side. This is is good when you want to do local restores. However what if you want to do a file level recovery at the remote location? In this case you could have a clean install of the backup management server and import the backups there.

In the pull strategy restoring at the remote side or HQ is easy. However restoring locally is hard because you will need a clean install of the backup management server and import the backup locally just to do a file level recovery.

How to solve it? Well actually in both scenario I would start by installing a local empty management server. Please don't forget to install the Powershell SnapIn as well, as we need to do the autoimport. From a licensing perspective, you can reuse your license because this empty install won't be backing up any VM's and so you won't consume any sockets. Notice that both Veeam servers should be running the same version or at least the server importing should be newer or the same level as the one creating the backups.

Once you installed the empty server you can start by adding the repository directory as a new repository on the clean installed server. When you add the repository give it a name but start with the prefix "WANBCJ". Alternativally you can alter the script I am providing you later.


In the last step you can then click to automatically import existing backups


After this is done you should see your backups appear as "imported" and you can start FLR or Instant VM Recovery easily via the regular way.


One thing that won't happen however is that your repository will be automatically for new restore points. So if a couple of weeks later, you need to do a restore of a freshly copied restore point, you will have to manually rescan your repository. It's quite easy to do this. Just go to the repository in backup infrastructure, right click it and choose rescan


Well now here is the fun part. You can easily automate this. Just open up a Powershell window, by going to the main menu (it's the blue button in the top left corner of the GUI)


Ok so the basic script is actually one line of code:
Get-VBRBackupRepository | where { $_.name -match "^WANBCJ" } | ForEach-Object { Sync-VBRBackupRepository -Repository $_ | out-null}
You can see why WANBCJ is required as a prefix as the code will match any repository where the name start with WANBCJ. Then for each of these repository we will ask a resync. You can see the result poping up in the history tab


Now lets make this code automated. The easiest way on various platforms is just to create a ps1 file and add the following code to it:
Add-PSSnapin -Name "VeeamPSSnapin"
Get-VBRBackupRepository | where { $_.name -match "^WANBCJ" } | ForEach-Object { Sync-VBRBackupRepository -Repository $_ | out-null}
Notice that there is some added code that will load the VeeamPSSnapin. When you trigger powershell via the Veeam menu it is done automatically. However, the task scheduler of windows won't do it for you so you have to do it manually. In my case I have saved the script under c:\vbrscripts\syncrepo.ps1


Now in the Windows task scheduler you can schedule a new task to run this script on a daily basis (or more frequently if your prefer). You can see the program is "powershell" and the argument is the path to the script enclosed with quote signs.


If you want to test that it works, just hit the run button and see if you can see the event in the history tab.


The nice thing about this script is that if you add another repository which names starts with WANBCJ, you won't have to do anything as it will be automatically rescanned!

2013/09/24

Getting the most out of Windows 2012 Deduplication with Veeam

With the release of Windows 2012, Microsoft allows you to do deduplication in software. This feature can potentially save you a lot of storage space without having to buy specialized hardware.

For example if you have multiple Veeam backup jobs, storing the data on a common repository can give you global deduplication with Windows 2012. You can find a good blog article about this on the Veeam web site http://www.veeam.com/blog/how-to-get-unbelievable-deduplication-results-with-windows-server-2012-and-veeam-backup-replication.html . With the release of v7 I think you can build even more interesting scenario's where the primary repository is a volume without deduplication for fast backup and fast restore. Then you can use the backup copy job to copy the backups from the primary volume to a dedup volume in combination with GFS. Because GFS will create multiple full backups, this should lead to interesting dedup levels.

A couple of days ago I got an interesting question about in-guest deduplication and file level recovery with Veeam. I was pretty confident it would work because Veeam shows the disk to Windows via a propriety driver. I figured out that one of the requirement would be that the backup server is a Windows 2012 server.

However when I tried it I got the following error "Browsing deduplicated volumes requires that backup server is installed on Windows Server 2012"


I thought it was a bug because my B&R was running on 2012. After opening a case with support, it turns out that you just need to enable the deduplication role on the backup server (File and Storage Services > File and iSCSI Services > Data deduplication)


Once you do that, FLR will just work out of the box


So not only can you use deduplication on the backup server, you can also use it in guest, knowing that Veeam can successfully recover files from it.

2013/08/16

Veeam MultiHost SureReplica v7 - Demystified

Surereplica

After months of eagerly waiting to post about new features in Veeam Backup & Replication v7, I can finally go ahead. If you read through my blog post you will notice that I love to talk about Surebackup as I think we take a very interesting approach on how we separate the isolated network and the production network.

One of the new features in v7 is the Surereplica. I think it is a great features and has great benefits:

  • You will be able to test to Replica's at the other side and see if they work successfully. Again another checkbox that can be checked in your DR plan automatically.
  • More interesting is the fact that you will be able to use to resource at the other side as a test environment. The great thing is that the storage at the other side will probably be a copy or has similar storage performance specifics so that your lab runs at the same speed as the VMs in production. It will also allow you to create bigger sandboxes. In which case you could even use replica's just to create lab environments (replicating maybe only once a month or manually to refresh the latest data). Not specifically for DR scenarios.
One challenge with this setup is of course that not all replica's may land on the same ESXi host. In v6.5 this would have been a problem as a virtual lab (and then specifically the network part is created only one ESXi host)

In v7, specifically we have added the Multi host setup. Instead of creating the lab on vSwitch, you will need to have a dvSwitch in place (which you will be able to select during setup as shown below)


Now one of the tricky parts is that a dvSwitch has uplinks of course. This is good so that the VM's on different host will be able to talk to eachother. One tricky part, however is the vlan part now.


If you have a single host setup the vlan for the isolated network does not really matter as the switch has no uplinks so you won't connect it to production anyway. With multi host setup you need to watch out as you will have uplinks
  • For every isolated network, make sure that you use a VLAN ID that is not in use in production
  • Make sure your physical switch knows this VLAN and are forwarding the packets from one ESXi host to another.
Other then that, the setup is similar. Portgroups will be created automatically on the dvSwitch with correct VLAN ID.

Surebackup and Multi Host

So interesting question came in my mailbox this week. Can I use Multi host for surebackup as well. First thought was, yes of course you can. Then it hit me. You can not select the cluster for a virtual lab. So although the network is multi host, instant VM recovery will always be done to one host. It's the host you selected during vlab creation.

So I thought during testing, why not try to vMotion VMs as they are powered on. Well it turns out there are a few things you need to take into account.
  • Make sure that vPower is mounted on all your ESXi hosts. You can do this manually or initiate an Instant VM Recovery to every individual Host. To do this manually check out http://www.veeam.com/kb1055
  • When you backup a VM make sure all cdrom and floppies are disconnected. This avoid having local "cdroms" connected. This is a best practice for VMware environments anyway.
  • vMotion with snapshots should work starting from vSphere 4.0 http://kb.vmware.com/kb/1035550
  • Make  sure your I/O redirection datastore is a shared one and all your ESXi hosts have mounted it.
If you start the surebackup job the VM's will be started on the selected ESXi host. However if you setup DRS to balance your cluster, the VMs will be balanced automatically if they are vmotionable (I'm not sure it is a word but ok :)). If you are actually working in a totally seperated lab environments, this might be one of those times that you want to change the recommendation settings of DRS so that balancing will be done faster


Then when you fire up your lab you should see this happen if the load is getting to high on your initial ESXi server.

If it does not, just try a manual vMotion to check why the system is not able to do a vMotion.



2013/06/12

Protect your Veeam backups from physical access to your repository

A feature that is not in Veeam is encrypted backups. The features is not one of the top requested features like tape but still, every now and then I get a mail asking on how you can store your backups in an encrypted way with Veeam. The short answer is, it is not possible. However with Linux repositories you can do some pretty neat stuff.

This blog article continues on my previous article "Veeam and Linux Repository" . What I will show you in this article is how you encrypt the home volume so that all your backups are stored in an encrypted way. If someone would steal your server, the data would be worthless without the key thus protecting you from physical access.

So lets continue. Just after you have configured the firewall, you can create the repo group
groupadd repos;
echo "%repos ALL=(root) NOPASSWD: ALL" >> /etc/sudoers.d/repos;
However just before you create the repo01 user, we will encrypt the home volume. To do this, you will need take the home volume offline. Also encrypting the volume will destroy all the data, so do this before you put the server in production or migrate the data first.

To put home volume offline, go to to the console and go to runlevel 1 so that all remote users and other users will be disconnected. This should clear all the file locks but will also disable networking so you really need to do this on the console and not via ssh. Afterwards we will switch back to runlevel 4
telinit 1
umount /home
telinit 4
Now check your /etc/fstab file and look for the logical volume that you want to encrypt. In my case it is  /dev/mapper/vg_repo-lv_repository


Then you can use shred to clear any existing data on the disk. If you are using thin provisioning in VMware this is not recommended
shred -v --iterations=1 /dev/mapper/vg_repo-lv_repository
Then you can encrypt the disk with cryptsetup and open it. This will create a new disk under /dev/mapper
cryptsetup --verbose --verify-passphrase luksFormat /dev/mapper/vg_repo-lv_repository;
cryptsetup luksOpen /dev/mapper/vg_repo-lv_repository encrypted_home;

You can check if the disk is properly mapped:
fdisk -l /dev/mapper/encrypted_home
Now that the disk is under /dev/mapper/encrypted_home, you can format the disk with ext4
mkfs.ext4 /dev/mapper/encrypted_home
Finally you will need to add some lines to crypttab and fstab so that the disk is mounted at boot
echo "encrypted_home /dev/mapper/vg_repo-lv_repository none" >> /etc/crypttab 
echo "/dev/mapper/encrypted_home /home                   ext4    defaults        1 2" >> /etc/fstab
You will also have to comment out or remove the line in /etc/fstab that is responsible for mounting the old unencrypted volume /dev/mapper/vg_repo-lv_repository


Now you can execute "mount -a" to mount the encrypted volume or just reboot the machine. During the boot, the machine will ask for a password to write and read from the encrypted volume:



Now that you have an encrypted home volume, you can create the user an add the repository to Veeam
useradd -m -G repos repo01;
echo "repo01:repo01" | chpasswd;

Now you are able to write backups to your encrypted volume

In my test, the repository was not the bottleneck, however I only have a limited lab environment so there might be some overhead when you try instant vm recovery or while running the backups. If in doubt add more CPU and Memory :)


2013/05/31

Veeam FLR and Linux searches

Veeam has an excellent framework for searching and restoring files for Windows. The one click file restore is a feature well appreciated by users. However the functionality is not available for Linux servers. I thought about this and came up with some possible solutions.

Using the Veeam FLR

The first possible solution is using the Veeam FLR appliance. In the end it is just a Linux appliance and guess what, you can just logon to it. You can find all the info you need on the KB (http://www.veeam.com/kb1447)

Once you are logged in you can use the "mount" commando to find out where Veeam mounts the partitions. This seems to be on the pretty standard location "/media". Then you can dive into those partitions and use "find" to locate your file. In the example below you can see I used the FLR to search for the file /tmp/processfollow


Once you know the path, you can go back to the Explorer and do the restore operation

Using the native mlocate method

Linux has a standard tool for doing Indexing. This tool called mlocate, can be easily installed on Redhat system by executing
$ yum install mlocate

To create your initial database, just run the updatedb commando. Then if you want to update the file you can use the updatedb command again
$ yum install updatedb

The great thing is that Centos for example automatically creates a daily cron job so that this updating is automatically. Just take a look at "/etc/cron.daily/mlocate.cron" . In the script you will also see that Centos uses renice and ionice so that the process of indexing does not take all the available resource.

Once you have a database, you can use "locate" to find a file. For example
$ locate findthisfile 


Multiple local versions of the index

The index or database is just a flat file you can find under "/var/lib/mlocate/mlocate.db" . In fact you could adjust your cron job so that it copies the index file and renames it using the current date. You can then use locate to find a file that might already be deleted from disk. You can see an example below. The copy statement is 

$ cp /var/lib/mlocate/mlocate.db  /var/lib/mlocate/mlocate-$(date +%y%m%d).db

Then you can use the "-d" parameter to find a file in an older index
$ locate -d  /var/lib/mlocate/mlocate-date findthisfile

In the screenshot below you will see a trial run, where you can see that I am unable to find a file in the current db but I am able to find the file in an old index


Multiple versions of the index on a remote server

The other great thing is that you seem to be able to copy those indexes to a central server and use locate to search for files that are or were on a specific server.

In my example I have a central server 192.168.149.55. I used the following statement to copy my index to this central server
$ scp /var/lib/mlocate/mlocate.db index@192.168.149.45:/home/index/$(date +%y%m%d)-$(hostname).db

Then I created a small script on this central server called lsearch
#!/bin/sh
INDEXDIR=/home/index
for curdb in $(ls -t /home/index/*.db)
do
        echo ">>>>> $curdb"
        locate -d $curdb $1
done

After chmoding it, I am able to search for a file using ./lsearch
$./lsearch findthisfile


Of course I am not sure about the performance with bigger machines and of course with more servers. The script itself is very very very basic but I do hope it might inspire some people to create nicer and better implementations.