Personal Backups with Duply


backupstart

A month or two ago I finally went through all the old hard drives I’ve accumulated over the past decade. I mounted each of the disks and moved a bunch of files onto my desktop’s drive. There were lots of photos from the drives that I don’t want to lose so I decided to get a little more serious about backups.

I decided to give Duply a go. Duply is a wrapper for duplicity, which underneath it all uses the tried and trusted rsync.

  • Multiple Locations – I have duply configured to send various data to a USB Drive, Swift (Rackspace Cloud Files), and Another Server. These are easily configured with the .duply/backup scheme. 
  • Encrypted – duply works with GPG encryption
  • Customizable – duply has pre/post hooks which I leverage for notifications on backup success/failures 
  • Efficient – duply is capable of doing incremental backups and using compression

I’ve been really happy with testing restores with duply as well.

An example process that I have setup is as follows:

On my desktop system’s power resume, run an incremental backup to Swift. Notify on start and finish of the backups.

It required a little bit of Python and BASH to accomplish this but I’m happy with the end result. The scripts I used are published to Github under andyhky/duply-scripts. Getting started/installation are in the README.

backupcomplete

 

Advertisements

Guide to Backing up your Virtual Machines with VMware Consolidated Backup (VCB)

This has been a pretty fun project at work and I think it could benefit a lot of people, so I’ve decided to post a modified version of my documentation. It’s a DRAFT and not quite yet in production. I’m still working through a lot of the process. Here’s what I’ve learned…

There’s an awful lot of noise (and even worse, quite a bit of actual material) to wrap your mind around when doing a Disaster Recovery plan for virtual machines.

This guide is not a complete Disaster Recovery Plan. It’s intended to get you, the VMware administrator, to a point where you’re comfortable with passing the buck onto another backup solution (Tivoli, Veritas, USB Drive, whatever) so you can integrate it into your organization’s existing Disaster Recovery Plan.

The process, in a nutshell:

Windows, File-level

VCB talks to Virtual Center. Using a user that has limited permissions, creates a snapshot of each running Windows VM. VCB then mounts the .vmdk as a network share taking up no room on the VCB proxy. Tivoli, or any third-party backup software can then consume the files as if they were locally on the VCB proxy. When the backup process is finished, the snapshot is committed and the share removed.

All OSes, Image Level

VCB talks to Virtual Center. Using a user that has limited permissions, creates a snapshot of each running Windows VM. VCB then copies the .vmdk, .vmx, and all virtual machine files to the VCB proxy. Tivoli, or any third-party backup software can then consume the files. When Tivoli is finished, the snapshot is committed and the files are removed.

Step 0: Get VI-Toolkit for Windows PowerShell. It’s a powerful tool that we’ll use to make your backups change-friendly.

Step 1: Get a physical machine with san connectivity. It’s going to be your VCB proxy. I know… physical machine. Yuck. It’s worth it. We’re using an IBM x346. The machine doesn’t have to be amazing, but it does need to have a rather large amount of storage if you’re doing backups of Linux Guest VMs or full image-level backups.

Step 1b: Determine how much storage you will need for full VM backups. We chose to go with weekly fullVM backups of all VMs, nightly file-level backups of our Windows VMs, and nightly fullVM (image-level) backups of our Linux VMs. The beauty of VCB is the file-level backups do not consume any disk on the VCB Proxy.

From PowerShell… determine how much storage your non-Windows Disks use:

get-vm 
| where { $_.Guest.OSFullName -notmatch "Microsoft" -band $_.Guest.state -eq "Running" } 
| Get-HardDisk 
| measure-object -sum CapacityKB

Step 2: Install Windows 2003 Server. It’s a requirement for VCB. You’ll want to run these commands once it boots:

cmd
diskpart
automount disable
automount scrub

This is CRUCIAL. Not doing this will cause big problems for your VMFS Luns once they are made visible to the proxy.

(Search “Disabling Automatic Drive-Letter Assignment” for more information)

Step 3: Install .NET 2.0, VMware Consolidated Backup Framework, PowerShell, VI-Toolkit.

We need the PowerShell & VI-Toolkit to make our VCB proxy change friendly.

Since we’re being specific with the types of guests we’re backing up and the types of backups we’re running on those specific guests, we have two choices:

  1. Maintain a set of Windows VMs in a script and Maintain a set of Linux VMs in a script. Hope we remember to add/remove these VMs when a change occurs.
  2. Automate.

Step 4: From VirtualCenter, Create a Role called VCBuser and a user for VCB.

Here are the recommended permissions for the vcb role:

  • Virtual Machine
    • Configuration
      • Disk Lease
    • State
      • Create Snapshot
      • Remove Snapshot
    • Provisioning
      • Allow Virtual Machine Download
      • Allow Read-only Disk Access

It is crucial to add this user at the Hosts & Clusters level, otherwise you will run into permissions errors that are difficult to track down.

Step 5: Before taking backups, generate some scripts.

A lot of the enterprise-grade software has options for disk quiescence which basically means preparing the machine to take for a good, usable backup.

In this case, preparing the machine for good & usable backups means mounting our virtual machines to the VCB proxy.

Before a backup can take place:

  • Mount / Unmount scripts must be generated
  • Mount scripts must be run.

After the backup, you need to run the appropriate unmount scripts.

Here’s some powershell code that will create four .bat files:

  • file_level_mount_script.bat
  • file_level_unmount_script.bat
  • image_level_mount_script.bat
  • image_level_unmount_script.bat

Update – 9/05/2008: Added get-vc to grab a connection to VirtualCenter.

$vcbuser = "vcbuser";
$vcbpass = "password";
$vc = "virtual_center";
$mount_root = "c:\backup\"; # include trailing \...

get-vc -Server $vc -User $vcbuser -Password $vcbpass

$file_level_mount_script_name = "C:\file_level_mount_script.bat";
$file_level_unmount_script_name = "C:\file_level_unmount_script.bat";
$image_level_mount_script_name = "C:\image_level_mount_script.bat";
$image_level_unmount_script_name = "C:\image_level_unmount_script.bat";



# Begin Windows VM File Level Backup MOUNT
$file_level_mount_script ="cd `"\Program Files\VMware\VMware Consolidated Backup Framework\`""; 
get-vm  
| where { $_.Guest.OSFullName -match "Windows" -band $_.Guest.state -eq "Running" } 
| % { $file_level_mount_script += "`nvcbmounter.exe -h " + $vc + " -u " + $vcbuser + " -p " + $vcbpass + " -r " + $mount_root + "" + $_.Guest.Hostname + " -t `"file`" -m san -a ipaddr:" + $_.Guest.Hostname };
echo $file_level_mount_script | out-File $file_level_mount_script_name
# End Windows VM File Level Backup MOUNT

# Begin Windows VM File Level Backup UNMOUNT
$file_level_unmount_script ="cd `"\Program Files\VMware\VMware Consolidated Backup Framework\`""; 
get-vm  
| where { $_.Guest.OSFullName -match "Windows" -band $_.Guest.state -eq "Running" } 
| % { $file_level_unmount_script += "`nvcbmounter.exe -h " + $vc + " -u " + $vcbuser + " -p " + $vcbpass + " -U " + $mount_root + "" + $_.Guest.Hostname }; 
echo $file_level_unmount_script | out-File $file_level_unmount_script_name
# End Windows VM File Level Backup UNMOUNT
 
# Begin Full Image Level Mount All VMs
$image_level_mount_script ="cd `"\Program Files\VMware\VMware Consolidated Backup Framework\`""; 
get-vm  
| where { $_.Guest.state -eq "Running" } 
| % { $image_level_mount_script += "`nvcbmounter.exe -h " + $vc + " -u " + $vcbuser + " -p " + $vcbpass + " -r " + $mount_root + "" + $_.Guest.Hostname + " -t `"fullvm`" -m san -a ipaddr:" + $_.Guest.Hostname }; 
echo $image_level_mount_script | out-File $image_level_mount_script_name
# End Full Image Level Mount All VMs
 
# Begin Full Image Level UNMOUNT All VMs
$image_level_unmount_script ="cd `"\Program Files\VMware\VMware Consolidated Backup Framework\`""; 
get-vm  
| where { $_.Guest.state -eq "Running" } 
| % { $image_level_unmount_script += "`nvcbmounter.exe -h " + $vc + " -u " + $vcbuser + " -p " + $vcbpass + " -U " + $mount_root + "" + $_.Guest.Hostname }; 
echo $image_level_unmount_script | out-File $image_level_unmount_script_name
# End Full Image Level UNMOUNT All VMs