Re-Registering a Proxy Server

SSH into the proxy server and run "/etc/init.d/initproxyappliance register"

It's faster than rebooting it and pressing a key when prompted from teh console.

Proxy Server Failing to Boot - fsck Failed

If you restart a proxy and it halts boot because "fsck failed for at least one filesystems" it's likely that there was a snapshot mounted when it rebooted and it's trying to mount that. Edit the proxy's settings in VMware and remove the snapshot (whatever disk is named differently than the proxy server). Once that's done you can can reboot the proxy and it should come up normally.


Make sure to delete the snapshot of the server that you removed from the proxy

Booting off of USB to install fresh DDOS when it is auto booting off of the local drives.

Installing fresh DDOS is done off of a USB drive. The EMC documentation just says to insert it, reboot, and it will boot off the USB drive. For me it bypassed the USB drive and booted normally. At least on my hardware it took pressing F11 at boot time to get an option to select the USB drive as the boot source.

Bandwidth Throttling Regular Client and VMware Image Backups

Bandwidth throttling a regular windows/nix client simply takes creating a file called avtar.cmd and placing it in the var folder where you installed the avamar client.

Edit the the file and add the following line:
--net-throttle=0.5
(where the 0.5 = how many mbps you are allowing the client)


Since there is no client installed on VMware image backups you can do it dataset wide. Edit the dataset, click the options tab, select the correct plugin (Windows/VMware Image), click more, and enter [avtar]throttle in the attribute field and the mbps value you'd like in the attribute value field. Then press the + button and hit okay. 

Configuring SNMP, Syslog, or Email alerting

Avamar can be set up to send SNMP traps, syslog notifications, or emails based on when event codes occur. This is configured under Tools - Manage profiles. Simply create a new profile and select the notification method you want.

As of version 7.1 SNMP and syslog do not provide enough information to be useful for VMware image backups (like client name). When integrating with an event management solution we were forced to parse emails.

If you want to be alerted of every backup failure or timeout select the following event codes

25004
30900
30901
30902
30910
30927
30932
30998
30999
31001
31013

Using PowerShell to Automate Avamar Tasks with SSH

I wanted to give our operations group the ability to execute various scripts or commands on the Avamar utility node without giving them direct SSH access. This can be accomplished using key based authentication, Plink, and powershell.

1. Download Plink and PuTTYgen from http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html
2. Use PuTTYgen to create your key pair. Save both the public and private key
3. Log onto the Avamar utility node and add the contents of the private key to the authorized_keys file in the ~user/.ssh directory.
4. Save the private key and Plink on the windows computer that you will be running the powershell scripts on. Protect your private key. Anyone that gains access to the file will have access to the utility node. 
5. Create your PowerShell script. In this example you need to set the variables for where the key and Plink reside, the full path to the private key, and the command you want to execute.
• $plinkfolder="C:\Program Files (x86)\PuTTY"
  $privateKey="C:\Program Files (x86)\PuTTY\avamar_vmware_script_priv.ppk"
  $avamarunode="user@ahavmrun.ent.lolcentral.com"
  $Command="/usr/bin/java -jar /usr/local/avamar/bin/proxycp.jar --selectalldatastore"
  cd $plinkfolder
  ./plink.exe $avamarunode -ssh -i $privatekey $Command

Now anyone you give access to execute that script will be running  "/usr/bin/java -jar /usr/local/avamar/bin/proxycp.jar --selectalldatastore" on the utility node. Keep in mind that if they know what they are doing, can edit the file, or access the key, they can do whatever they want on the utility node.

Cleaning Up Snapshots mounted on Avamar Proxy Servers.

If you have a VMware image backup failing with the 10056 code and you can't consolidate snapshots on the VM because of a file IO lock chances are good you have a snapshot that's been left mounted on the Avamar proxy server.

You can either manually edit the Avamar proxy servers and remove any disks past the first two, or you can use the proxycp.jar utility to automatically clean them all up.

java -jar /usr/local/avamar/bin/proxycp.jar --listproxydisk --cleanup

will go though all of the proxy servers you have and unmount any snaps it left attached. I have run this while there was a VMware image backup running and it did not cause that backup to fail.

Registering HP-UX and Linux Clients via CLI Non-interactively

If you want to script registering a lot of clients using ssh from a single host you can use avagent.d instead of avregister to register without an interactive shell on the host.

/opt/AVMRclnt/etc/avagent.d register your_utility_node_name your_domain

Hello

I've received a lot of Avamar help from blogs and forums over the last couple years. I figured I'd drop some of the tips and tricks here to give a little back.

Scripting VM Recovery for DR or other mass restores

Our DR plan involves restoring a lot of VMware virtual machines to a new Vcetner. I wrote a script to be run on the Avamar utility node to automate it. One quirk about restoring from a replicated copy with this method is that when you replicate a VM backup to another grid it adds a tag to the end of the backup name so you can't just use a client name list. To make this script work you need to:

1. Put it and the other required files in /home/admin/vmware_restore_script
2. Go into the GUI and copy/paste all names of the backup names (clientname_avamartag) into a file called vmware_lookup_list 
3. create a file called vmlist containing all of the servers you want to restore. 

This script is also customized to our ESX host names and datastore names. You can set the number of hosts and datastores and it will loop through them to distribute the VMs and Hosts equally. 

It does a minor amount of error checking and logs to an output file so you can see what VMs did not successfully submit a restore.

If anyone knows how to get a backup list based on a domain from CLI when there are no clients registered on that grid (just replica copies) please let me know. That would fix the whole vm_lookup_list part of this.

Feel free to use this and modify as needed. I'd appreciate any feedback if you think there is something to make it better.

#!/bin/bash
## Christian Schmidt 4/6/2015
## cgschmidt@landolakes.com
## Script to bulk restore VMs @ DR. It must resides in /home/admin/vmware_restore_script/
## folder must have vm_lookup_list containing the full avamar name of the servers (cut and paste from the restore area in the GUI)
## folder must have vmlist containing a simple list of vmnames to be restored
## ESX hosts names must be formatted as DRVMhost(count) edit the count of ESX hosts by changing ESXHOSTCOUNT variable
## Datastores must be formatted as vol(count) edit the count of datastores by changing the DSCOUNT variable
## montitor /home/admin/vmware_restore_script/vmrestore.log for success or failure of particular backups

##Domain replicated backups exist in
DOMAIN=REPLICATE/ahavmrun.ent.lolcentral.com/vmware.ent.lolcentral.com/windows-prod
##fqdn of the Vcenter
TGTVCENTER=vmware2.ent.lolcentral.com
##VMware Datacenter
TGTDC="Atlanta"
##VMware folder to put the VM in
TGTFOLDER=avamar_restores
##number of esxhosts named DRVMhost#
ESXHOSTCOUNT=3
##number of datastores named vol#
DSCOUNT=2
##setting counters
ESXCOUNTER=0
DSCOUNTER=0

echo "============================================================================" >> /home/admin/vmware_restore_script/vmrestore.log
echo "Starting VMware restore script at $(date)" >> /home/admin/vmware_restore_script/vmrestore.log
echo "============================================================================" >> /home/admin/vmware_restore_script/vmrestore.log
for SHORTSERVER in $(cat /home/admin/vmware_restore_script/vmlist)
do
        if [[ $(grep -i -c $SHORTSERVER /home/admin/vmware_restore_script/vm_lookup_list) -eq 1 ]]
        then
                SERVER=$(grep -i $SHORTSERVER /home/admin/vmware_restore_script/vm_lookup_list)
        else
                echo "ERROR: failed to lookup full server name for" $SHORTSERVER >> /home/admin/vmware_restore_script/vmrestore.log
                continue
        fi
        LABELNUM=a
        LABELNUM=$(mccli backup show --domain=$DOMAIN --name=$SERVER | grep -E "CST|CDT" | awk '{print $4}' | sort -rn | head -1)
        if [[ $LABELNUM -gt 0 ]]
        then
                let ESXCOUNTER=$ESXCOUNTER+1
                let DSCOUNTER=$DSCOUNTER+1
                mccli backup restore --name=$SERVER --domain=$DOMAIN --plugin=3016 --labelNum=$LABELNUM --restore-vm-to=new --virtual-center-name=$TGTVCENTER --datacenter=$TGTDC --folder=$TGTFOLDER --dest-client-name=${SHORTSERVER} --esx-host-name=DRVMhost$ESXCOUNTER --datastore-name=vol$DSCOUNTER
                if [[ $? -eq 0 ]]
                then
                        echo "SUCCESS: $SERVER Restore started" >> /home/admin/vmware_restore_script/vmrestore.log
                else
                        echo "ERROR: $SERVER mccli restore command failed, mccli return code $?" >> /home/admin/vmware_restore_script/vmrestore.log
                        continue
                fi
        else
                echo "ERROR: Unable to lookup labelnum for " $SERVER >> /home/admin/vmware_restore_script/vmrestore.log
                continue
        fi
        if [[ $ESXCOUNTER -eq $ESXHOSTCOUNT ]]
        then
                ESXCOUNTER=0
        fi
        if [[ $DSCOUNTER -eq $DSCOUNT ]]
        then
                DSCOUNTER=0
        fi
done
echo "========================Script completed $(date)==============================" >> /home/admin/vmware_restore_script/vmrestore.log

Checking and setting CBT for VMware image backups

Using change block tracking greatly improves VM image backup time. To ensure Avamar is using the feature you must both check the option to use it in the dataset and ensure it is turned on in VMware.

To validate it's enabled on your VMs you can use proxycp.jar

•  java -jar proxycp.jar --cbtstatus  | grep -E "VM Name|CBT"
• add the --vm VMname flag to check a single server

If you would like to try and enable it on all VMs:

• java -jar proxycp.jar --cbtstatus --enablecbt

Avamar will attempt to enable it on all VMs it has registered. If the VMware hardware level is too low to support CBT it will just error out in VMware.