Veeam Backup

VMware SureBackup How-To and Troubleshooting Guide

Veeam Backup & Replication 6.1 July, 2012
By Ben Milligan Technical Lead, Veeam Software

 2012 Veeam Software. All rights reserved. All trademarks are the property of their respective owners. No part of this publication may be reproduced, transmitted, transcribed, stored in a retrieval system, or translated into any language in any form by any means, without written permission from Veeam Software Inc (Veeam). The information contained in this document represents the current view of Veeam on the issue discussed as of the date of publication and is subject to change without notice. Veeam shall not be liable for technical or editorial errors or omissions contained herein. Veeam makes no warranties, express or implied, in this document. Veeam may have patents, patent applications, trademark, copyright, or other intellectual property rights covering the subject matter of this document. All other trademarks mentioned herein are the property of their respective owners. Except as expressly provided in any written license agreement from Veeam, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.

Important!

Please read the End User Software License Agreement before using the accompanying software program(s). Using any part of the software indicates that you accept the terms of the End User Software License Agreement.

2 | Veeam Backup & Replication | SureBackup | HOW-TO AND TROUBLESHOOTING GUIDE

CONTENTS
CONTACTING VEEAM SOFTWARE............................................................................... 4 CUSTOMER SUPPORT................................................................................................................................ 4 ONLINE SUPPORT ..................................................................................................................................... 4 COMPANY CONTACTS .............................................................................................................................. 4 APPLICATION GROUPS ................................................................................................ 5 CREATING AN APPLICATION GROUP ....................................................................................................... 5 USING AN APPLICATION GROUP ............................................................................................................. 7 VIRTUAL LABS ........................................................................................................................................... 7 VIRTUAL LAB CONFIGURATION/CREATION............................................................................................. 7 SUREBACKUP JOBS.................................................................................................... 12 WHAT IS A SUREBACKUP JOB, AND WHAT IS ITS PURPOSE? ............................................................... 12 HOW TO SETUP A SUREBACKUP JOB ................................................................................................... 12 LINKING BACKUP JOBS .......................................................................................................................... 14 HOW TO USE A SUREBACKUP JOB ....................................................................................................... 15 MORE RESOURCES ..................................................................................................... 22

3 | Veeam Backup & Replication | SureBackup | HOW-TO AND TROUBLESHOOTING GUIDE

CONTACTING VEEAM SOFTWARE

At Veeam Software we value the feedback from our customers. It is important not only to help you quickly with your technical issues, but it is our mission to listen to your input, and build products that incorporate your suggestions.

Customer Support
Should you have a product issue, suggestion or question, please visit our Customer Center Portal at cp.veeam.com, to open a case, search our knowledge base, reference documentation, manage your license or obtain the latest product release.

Online Support
If you have any questions about Veeam Backup & Replication, you may use the following resources: Full documentation set: www.veeam.com/vmware-esx-backup/resources.html Community forum: forums.veeam.com

Company Contacts
For the most up to date information about company contacts and offices location, please visit www.veeam.com/contacts.html.

4 | Veeam Backup & Replication | SureBackup | HOW-TO AND TROUBLESHOOTING GUIDE

APPLICATION GROUPS
An Application Group is very close in definition to its name. Put simply, it is a grouping of Servers/Virtual Machines that have been backed up, that have dependencies on one another in an environment. One should create application groups that mimic production servers in their native environment based on dependencies. Some examples of these dependencies can be: Microsoft Exchange/ Global Catalog Server. Domain Controller/ SQL Server/ Web Server. These groupings should be defined by the production environment that Veeam is creating backups of/from. Example is Microsoft Exchange server if you power it on in the isolated environment which does not have DNS server and Domain Controller present, mailbox information store will not start. Creating an application group in Veeam Backup requires the Enterprise Licensed edition of Veeam Backup & Replication. If a customer is on a Standard license level, then they will not have this capability. Also confirm all machines in Application Group have VMware tools installed in production. This is used for testing/verification purposes.

Creating an Application Group

First navigate to the Infrastructure Pane in Veeam Backup, and right clicking on Application Groups provides the ability to create a new Application group as the screenshot shows:

Next, name the Application group, and add the guest VMs you wish to boot in this group.

5 | Veeam Backup & Replication | SureBackup | HOW-TO AND TROUBLESHOOTING GUIDE

As explained earlier, you will want to group together dependent guests for your need. Most common is Exchange with the Domain Controller from the same production environment:

This is where your settings become important. Particularly the roles set for the servers. The role set will also correspond to values contained under Edit:

Each role will dictate the Maximum allowed boot time. Domain Controllers as an example will set at 2100 Seconds because they reboot after initial boot phase of the server. This adds time to the boot operation, and we reflect that with a higher timeout. Once that timeout is hit, the job stops, fails, shuts down the lab, and sends a failure notification. The common error we receive here is: VMware Tools failed to start in a Timely Manner This is extremely common and occurs whenever the machine does not boot in time according to Maximum allowed boot time. VM roles are listed in .xml files stored in the SbRoles folder under the Veeam Backup installation. You can create your own roles by creating new .xml files and specifying the scripts to be performed in them.

6 | Veeam Backup & Replication | SureBackup | HOW-TO AND TROUBLESHOOTING GUIDE

Using an Application Group

Typical use the Application Group(s) is for Verification/U-AIR Restore/Sandbox testing. Users with large infrastructures will usually create an Application group with their main infrastructure servers such as Domain Controller/DNS Server. These are then used to verify all their backups by using this application group for linked backup jobs for verification testing. Small/static environments can place all their VMs in an application group, simply ensuring that they boot in the correct order in the application group settings.

Virtual Labs
I think it is best to think of Virtual Lab(s) as a router, as this helps simplify how it is used. This is also the area where most users run into configuration related problems, so it is hopeful this guide is used to eliminate some of this confusion. The simplest way to understand what is being done here is to, again, think of the Virtual Lab as a router, or just simply put, IP Tables, that allow an address to remain the same as production inside the lab while presenting it as something altogether different outside the lab. We want to mimic the production environment these machines live in normally as much as possible so we can realize any problems with the backup/restore before it is necessary.

Virtual Lab Configuration/Creation

Right click on Virtual Labs under Infrastructure pane in Veeam Backup console, selecting Add Virtual Lab. The first 3 pieces are simple.

1. Virtual Lab Name

The name is inconsequential of course, but keep in mind it will use this name for your Lab Virtual Network created by the wizard: Example:

2. Select your Host

a) Make sure that this host has a VMKernel port group that can communicate with the Veeam Server. You should be able to ping the VMkernel IP from the Veeam server, and/or VMKping the Veeam Server from the ESX host. This is a common problem, and the common error is Unable to Mount NFS Datastore (http://www.veeam.com/KB1055).

b) Make sure the host is not already overloaded with Memory/CPU, as the machines will utilize these resources.

3. Select your Datastore

a) Make sure that the free space on the datastore is more than the total amount of memory of the servers you are booting. In VMware a swap file is created when you power on a virtual machine that equals the amount of memory for the machine. If you see any errors when a machine is attempted to power on, in vSphere, that states: There is not enough free disk space on datastore.

7 | Veeam Backup & Replication | SureBackup | HOW-TO AND TROUBLESHOOTING GUIDE

b) Make sure the datastore you choose here also has a blocksize that can support the disks of the VMs booting as well, or you will get the error that the destination does not support large files.

4. Proxy Settings
a) All you need to assure here is that the network you choose should be communicable with the Veeam Server. The proxy needs to be able to talk to Veeam Backup server, period. In some cases it is simply preferred to set a static IP to this machine, but it should not be necessary if there is a DHCP server on that network.

b) Name, Resource Pool, Folder are all settings that are specific to the customer and environment. c) You can power on the Virtual Lab manually after creation, to ensure IP is applied, and that Veeam can ping that lab VM. Simply power it back down after this test (via vSphere Client).

5. Networking
This is where most users find difficulty in the setup process. a) I suggest starting out all configurations with Advanced (manual configuration) option. This allows more granular control over the network settings.

b) Isolated Networks this mapping should be the production network(s) that the Virtual Machines that you are booting in the SureBackup job/virtual lab are connected to. So if you look at the VM in vSphere, and look under Summary/Network, or under Edit Settings, when you select the vNIC:

So for example this above virtual machine would need to have the virtual lab mapped to the SSA-LAB network. Note The port group for the production network should be selected from the ESX host where the original VM is stored/managed. In some situations there are hosts where port group names are the same, but selecting incorrect ESX host could possibly prevent lab from communicating properly.

8 | Veeam Backup & Replication | SureBackup | HOW-TO AND TROUBLESHOOTING GUIDE

Example:

You can add multiple network mappings, as certain virtual machines will have multiple networks they are connected to in vSphere. You will notice the Virtual Lab will have a connection to each of these networks when you look at its settings in vSphere Client, respectfully. Note You can only add up to 4 networks here to the virtual lab, as this is the limit for VM Hardware version 4 of VMware (VI3). You can however increase this ability, by updating the virtual machine to version 7/8 hardware version of VMware. Simply right click on the virtual lab virtual machine in vSphere client after creation, and select Upgrade Virtual Hardware. This will allow you to add additional network connections to the lab, up to 10.

6. Network Settings
This section requires knowledge around how IP and subnets function. If you are unsure about how the 32 bit addressing works, I encourage you to review this information: http://en.wikipedia.org/wiki/IP_address; IP Calculator: http://jodies.de/ipcalc a) You will want to select the vNIC, and click Edit. This will bring up the connection settings:

9 | Veeam Backup & Replication | SureBackup | HOW-TO AND TROUBLESHOOTING GUIDE

b) The IP address you set here on the vNIC should match the gateway address of the production network it is mapped to (from the previous Isolated Networks part of the setup). The subnet mask should simply be the same as production network, and the masquerade is used to give the ability to communicate to these isolated machines from outside the lab. This will auto-adjust per your settings of subnet mask and IP, but may need tweaked depending on the network configuration of the customer. Notes The masquerade has to be unique in your environment! This is important. They will want to ensure there is no duplication of this IP on the network. It should sit outside the subnet mask set. Just have them run IPCONFIG via CMD on a virtual machine that is on their production network we are mapping to gain the gateway and mask needed. c) You do not need to enable DHCP if you are booting the DHCP server (typically a Domain Controller) in the lab. If the application group to be started in the virtual lab does not have a DHCP server and some applications in this group as well as verified applications require DHCP, you can enable the DHCP service on a vNIC adapter for each isolated network. You can also select specific DNS servers from the production network that should be started in the isolated network. Keep in mind that to be able to add a DNS server, you should have it virtualized in your production environment, and you should also have its backup. Again, the Domain Controller being booted commonly has this role (DNS) and this setting is not needed when that is true.

7. Static Mapping
This setting is not required. It allows you to map a specific address to machine being booted in the SureBackup lab so it is accessible from other machines on the production network. This is useful for Development access, U-AIR restore usage, etc. Again, this setting is not required for SureBackup/U-AIR to function but does provide further flexibility. The IP addresses would be set as external IPs of our proxy appliance. The proxy appliance redirects requests to these IPs set to the corresponding VMs in the Virtual Lab.

8. Ready to Apply!
For troubleshooting purposes this screen is very useful. You can review these settings to check network communication overall.

10 | Veeam Backup & Replication | SureBackup | HOW-TO AND TROUBLESHOOTING GUIDE

The lab now is created. Some problems you may see include Failed to Mount NFS Datastore or unable to Mount NFS Datastore during NFS storage creation on the host. You may also see this: Create NAS datastore: The specified key, name, or identifier already exists. This can happen when the host can ping the Veeam server by IP but not hostname. So the NFS datastore is mounted via IP instead of hostname, and Veeam Backup server does not recognize this. So we attempt to mount the NFS datastore again. Then we get the failure because its already mounted. To correct this, simply ensure that the host we are mounting the NFS volume to, can resolve the hostname of the Veeam server, over the VMkernel connection. In Version 6.1, we need to pay mind to Repository location as well. Your repository needs to have the NFS service enabled to allow it to run Virtual Machines from the backup file. This will not work across a WAN or DMZ type environment where the Virtual Lab Appliance is segmented from the Veeam Backup server unless you can NAT in both directions between the Lab proxy appliance and the Veeam Backup server. The Virtual Lab appliance should be in the same subnet as the Veeam Backup console. Lastly, you may get an error in vSphere in vCenter when you attempt to remove a Veeam NFS mount: Item has been deleted or not completely created Simply log into the ESX host directly that has this datastore mounted, and unmount it there. You should be able to remove/remount the NFS storage after this, or simply rescan the storage in vCenter to update the removal.

11 | Veeam Backup & Replication | SureBackup | HOW-TO AND TROUBLESHOOTING GUIDE

SUREBACKUP JOBS
What is a SureBackup job, and what is its purpose?
Now that you have setup your Application groups, and Virtual Lab, its time to now configure the SureBackup job(s) that will be utilizing these. In version 6.1, the GUI is dynamic, so you will not have the option to create a SureBackup job until a Virtual Lab is created. SureBackup jobs perform recovery verification by powering on one or more VMs in the isolated environment, and verifying recovery by checking if VM was started, OS was booted, VM responds to ping, and VM applications are running fine. SureBackup jobs are also the key component of U-AIR functionality. Video Resources: 1. www.youtube.com/watch?v=o3QzOoHxAJw this is a high level description of how SureBackup functions. www.youtube.com/watch?v=JkmUnZ0SPe8 technical Demo (Earlier code). Checks for successful VM startup (API call results). For example, if a virtual disk is missing from the backup file, or if disk descriptor file is corrupted, VM will not start and vCenter will inform SureBackup/Veeam Backup. Checks for successful OS boot up by checking VM heartbeat. If OS does not boot, then VMware Tools service will never start and heartbeat will never appear. The error message for this is VMware Tools failed to start in a timely manner (you will recall that this also can mean simply we did not give the machine enough time to boot). Checks for network connectivity by pinging VM. If VM never appears on the network, this is also indicated by a failure to ping the machine in our summary. (This can also mean that the virtual lab network is not configured properly). Checks VM applications by running test scripts against them. If the application does not respond to test script with expected results, this indicates another possible recovery issue.

As discussed in before, the operations that the SureBackup job takes are the following:

2.

3.

4.

These are the essential basics of SureBackup jobs. As you will learn, they add quite bit of flexibility to you utilize in isolated environments.

How to Setup a SureBackup Job

In the Veeam Backup & Replication pane, simply right-click on the SureBackup and select SureBackup. Then give it a name. Or click the SureBackup Job button under the Home tab. You will be prompted now to select a Virtual Lab in which the machines you are using in the job are run inside of. This ties in directly to the network settings discussed under our Virtual Labs section of this training documentation. You need to use the Virtual Lab that is mimicking the production network of the machines booting in the job. If they do not have this mapping correctly, the network(s) will not connect, and you will see an error message: 08/08/2011 3:40:18 PM Error Test_DC: Powering on (no VM networks) You can only use one Virtual Lab per SureBackup job, and vice versa, meaning you cannot run concurrent SureBackup jobs to the same Virtual Lab.

12 | Veeam Backup & Replication | SureBackup | HOW-TO AND TROUBLESHOOTING GUIDE

Next, you will set your Application Group:

Note that it lists the contents here of the machines within the Application group. Same as the Virtual Lab setting, you can only use one application group in a SureBackup job. Note the backup status to see the latest available backup for the selected servers. Pay special mind to this portion at the bottom of this settings screen.

This setting allows some further flexibility for U-AIR, direct tested of lab VMs, etc. This is also how you would create a sandbox. If you do not select this option, the lab will simply shut itself down when it completes verification. If the application group incurs any failed task, such as heartbeat, ping, or script, it will also shut down the Application group servers and fail the SureBackup job. This is because it is expected to only check for dependencies required as set by the user to apply to the application group. If any dependency should fail, then it is expected to not work as expected. It is OK also to eliminate certain tests in the SureBackup job so you can log into the machine that is booting in the lab to check it first hand for any issues, etc. You can however open the summary (by doubleclicking the job), right click on the server(s) in the job, and select Start here. This will run the Application group in troubleshooting mode allowing you to avoid any automatic shutdown when any of the test/scripts fail. It gives you the time to open the console of the SureBackup machines in the lab, and check for any problems.

13 | Veeam Backup & Replication | SureBackup | HOW-TO AND TROUBLESHOOTING GUIDE

Linking Backup Jobs

The next page of the settings shows us the ability to link one or more backup jobs to the SureBackup job. This will process all the VMs that belong in that Backup job linked. These are typically used for verification. These are completely optional, and not required for SureBackup job to function. You can use either Application group and/or linked backup jobs as well, for your testing needs. Important! Please note that failed VM in a Linked Backup job will not cause the lab to shutdown, however an Application group will shut down if any one VM fails on any given step. It is assumed that the Backup job is verification and the rest of the VMs are needed to still be processed, but in an Application group, it is assumed that all are dependencies and will fail if any one VM does. The most common configuration is to have a Domain Controller where DHCP/DNS and other roles that the VMs linked in the backup job need to pass verification reside in production. Put simply, you use the Application group as the base, and the VMs linked in the job are cycled through, while the application group stays powered on throughout. Click Add and you will see your jobs listed for selection. You can edit the role and other verification settings exactly as you did for Application group(s).

You can click Advanced, to set individual roles to the machines that are coming from the Backup job. This allows you to set the scripts for each server per its role for verification. At the bottom, you can also select simultaneous processing of these VMs linked via their backup job. It is suggested to start with just 1 and add more simultaneous that are allowed that your setup can perform consistently. This is useful when you do not wish for the resource impact on the host we are using, or latency introduced by running too many Virtual Machines via SureBackup on the Veeam Server.

14 | Veeam Backup & Replication | SureBackup | HOW-TO AND TROUBLESHOOTING GUIDE

The next two settings include notifications and schedule for these to run on. These are fairly nonconsequential. There is one setting when you set the job run automatically, where you can force the job to wait for a Backup job to complete when it is linked:

This is of course intended to prevent overlap. If a SureBackup job kicks off that is linked to a Backup job that is running, it will fail with the error that the job is locked by running session job name. This is normal and intended to prevent corruption of the backup(s). On the other side of things, Backup jobs will automatically stop any running SureBackup jobs which are locking the backup files that backup job needs to update. So, backup jobs should always complete fine. Click next, check the summary, then finish. Your SureBackup job is ready to run!

How to Use a SureBackup Job

Upon right clicking a SureBackup job, you will be presented with some options:

Youll notice we see Start job and Start job to This is an important feature to understand, as the Start job to is used to run a job from a previous point in time:

15 | Veeam Backup & Replication | SureBackup | HOW-TO AND TROUBLESHOOTING GUIDE

Important!

This is a common misunderstanding by users, so please pay mind to it. If you simply choose Start, we will assume to use the latest backup for each machine contained in the job. (This can cause problems, such as if the latest job was sent offsite). When you do start the job, the process begins. First, the Proxy appliance is booted:

If the Proxy Appliance cannot communicate with the Veeam server, you will receive the following error: Timed out waiting for proxy appliance to respond. Possible causes: 1. Appliance is configured to run on different network than backup server. 2. Appliance is configured to obtain IP address automatically, but DHCP server is not available. Since we tested communication during Virtual Lab creation earlier in this guide, this should not be a problem. After the proxy appliance is booted and running, we continue to process each guest in the job:

16 | Veeam Backup & Replication | SureBackup | HOW-TO AND TROUBLESHOOTING GUIDE

It will now configure the Domain Controller, which is a necessary step to ensure that the DC is booted in non-authoritative mode. This is why the role is important to set for this server type. If it is not set, the sysvol will not be shared, and Active Directory will not work. You can highlight the VM in the job, to get more granular about that particular guest in the SureBackup job as well:

You can also access the console (MKS) directly from the Veeam Server here, if you click on the hyperlink in the session details. This is helpful in monitoring boot state. While this machine is booting, you can also certainly use the vSphere Client to open the console of this guest, and watch as the machine boots up. You can also use this to check for any problems that you may be seeing. The machines will show up under your Virtual Lab Resource Pool with name extensions to avoid any conflict:

You can see this is my proxy appliance along with my Domain Controller booting the lab. You can also look at the machines summary to gain an idea of what VMware is advising about this guest as it comes to life:

17 | Veeam Backup & Replication | SureBackup | HOW-TO AND TROUBLESHOOTING GUIDE

As you can see, I am still waiting for VMware Tools and an IP address for this machine. These are the pieces we are awaiting to confirm validation. Here is when we have this information:

You will also notice in our logs for SureBackup as we are checking for these to complete (from SureBackup log for this job). First, we show we are waiting 2100 seconds for the heartbeat, and IP address: <01> Info [SureBackup] [SSA-DC01] [PowerOnVm] [StableIp] Waiting for IP-address during 2100 second(s) Now you can see the name of the machine along with these parameters, currently not yet ready:

You can check how it is progressing as we move along:

Now that an IP address has been detected, we assure it is stable: [SureBackup] [SSA-DC01] [PowerOnVm] [StableIp] IP-address has been detected in 00:06:32 Since this is a Domain Controller, you also should see the following: [SureBackup] [SSA-DC01] [PowerOnVm] [StableIp] Either VM or VMware Tools have been restarted in 00:01:39 seconds(s)

18 | Veeam Backup & Replication | SureBackup | HOW-TO AND TROUBLESHOOTING GUIDE

We continue the timer of 2100 seconds here, minus how long we have waited already. This directly relates to the Maximum allowed boot time setting in the Verification options of the Linked job and/or Application group. You can see now why we want to allow more time for Domain Controllers! [SureBackup] <Powering on> Waiting for IP-address during XX second(s) After some time, and if everything boots normally you should see this: [SureBackup] <Powering on> | VM: 'SSA-DC01_9ac6b87b78574b6fa93d81d101511f13' | IP: '10.0.20.21' | power: 'PowerOn' | machine: 'Running' | tools: 'Ok' | heartbeat: 'Green' | Once we see this, we should soon see the machine pass the heartbeat test, and move on to the ping. If the ping test succeeds, we move on to the script/role portion of the SureBackup job. There are several reasons a ping test can fail. One is simply that the network is not configured properly. A rarer problem is when the Virtual Lab has been vmotioned/migrated to another host. This can sometimes happen with DRS and DPM enabled in a cluster, but is manually done in most cases. The network which the isolated VM is supposed to connect to is not listed on that new host its been migrated to, so it cannot connect to a network it does not know about. Note For SBS servers (Small Business Server), this time can be even longer than 2100 seconds, due to the complexity and number of domain services they provide. Again, please set the boot timeout for longer time periods, whenever you see VMware Tools failed to start in a timely manner. Also, it is typical you are required to use the local administrative account used to first configure SBS when you log into the lab machine running in SureBackup. After the ping test is successful, it will move on to the Role test as mentioned above. The timeout value: Application Initialization Timeout is enforced here, so it forces the job to wait how long this setting is for. In this case it is 120 Seconds:

19 | Veeam Backup & Replication | SureBackup | HOW-TO AND TROUBLESHOOTING GUIDE

Our GUI will advise you when the next step will proceed as you can see where it states Note: operation will be continued at said time. Once the role test succeeds, the job will move on the next guest set to run in the SureBackup job. By highlighting the VM that is in progress, you can also see the time stated for when the job will proceed to next stage based on settings in application group/linked backup job(s):

Then hopefully we see success:

Again, in an application group, if any one check should fail, the whole lab fails and it shuts itself down. After all have started successfully, the lab will either shut down and send notification, or continue running. This is dependent upon the Keep Application Group Running setting in the SureBackup job settings described earlier. When the SureBackup verification is completed, or the Application group is set to not continue running, it will shut down all machines, remove them from inventory, and shut down the proxy appliance. If the Application group was successful booting, you can click Close on this summary. You will then see the State in the SureBackup > Jobs pane as 100% Completed:

20 | Veeam Backup & Replication | SureBackup | HOW-TO AND TROUBLESHOOTING GUIDE

As you can see, once it is ready at 100%, you can simply open the summary, and right click on the guest you wish to initiate a U-AIR restore on. Please note this will only work if the U-AIR components are installed on the Veeam Server. You should be prompted for download if you choose one of the wizards that is not installed.

21 | Veeam Backup & Replication | SureBackup | HOW-TO AND TROUBLESHOOTING GUIDE

MORE RESOURCES
http://www.veeam.com/blog/automate-recovery-verification-of-vm-backups-withveeam-surebackup.html http://www.veeam.com/videos/5-new-ways-to-use-surebackup-31.html http://www.youtube.com/watch?v=JkmUnZ0SPe8 http://www.youtube.com/watch?v=PrZmKznKe9Y http://www.veeam.com/vm-backup-recovery-replication-software.html?ad=home

22 | Veeam Backup & Replication | SureBackup | HOW-TO AND TROUBLESHOOTING GUIDE