I've previously written and even recorded a video about the Pure plug-in for Veeam and some of the cool things you get when you use these two tools together. To briefly recap though, this integration allows Veeam Backup and Replication to get enhanced storage features for vSphere VMs hosted on FlashArray VMFS Datastores by being able to talk directly to the storage array. Some of these enhancements include: Backup from Storage Snapshots, Veeam Explorer for Storage Snapshots, and Snapshot Orchestration (aka Snapshot Only Backup Jobs). In this post I'm going to dive a little deeper on the last one, Snapshot Only Backup Jobs.
Why Snapshot Only Jobs?
First of all, why would you even care about creating a backup job that ONLY took a storage snapshot? I mean, "snapshot's aren't backup" right? Well no, of course they aren't (don't @ me), but they can be a very powerful part of a comprehensive data protection strategy. Consider this. Most folks create "daily" backup jobs. So your worst case RPO based on this backup alone would be 24 hours plus however long your job takes to complete. Failed backup? now it's doubled. Now let's say you need to restore. You're RTO is a function of how fast you can retrieve your data from where ever it is that you have your backups stored. Large restores from backups stored on tape or a traditional backup storage appliance could be a daunting prospect.
If these RPO/RTOs aren't acceptable for your business, what do you do? Well, fortunately there are actually plenty of options. I've highlighted how FlashArray//C is well suited as a backup repository because of its ability to backup and restore quickly, while also supporting advanced features like Instant VM Recovery and SafeMode. You could also leverage continuous replication features like Pure's ActiveDR or Veeam's CDP; however I would point out that DR is not backup either (more on that another day). Finally, assuming your backup windows are relatively small, you could just backup more often. Instead of a daily backup you could run a backup every 12 hours, or 6 hours. The challenge here is that for many environments, backup jobs take quite a lot of resources. Is your network, hypervisor, and storage all sized to handle this increased workload?
This is where array based snapshots can help. A popular strategy is to keep your daily backups, but then supplement those backups with much more frequent (hourly?) array based snapshots. FlashArray snapshots are nearly instantaneous to create, consume no additional space besides the changed or deleted data they protect, and when used with Veeam deliver application consistency and granular recovery. They're also crazy fast to restore from since they're right there on your production storage. FlashArray has plenty of high availability features such that these snapshot should be there when you need them, but even if something goes horribly wrong, you have the peace of mind knowing that you still have your traditional backup to fall back on (let's remember the "3-2-1 rule.")
Snapshot Jobs to a Secondary Storage Array
Hopefully by this point I've convinced you to include FlashArray snapshots as part of your data protection strategy. But that's not actually the point of this blog post. Snapshot only backup jobs are super easy to implement, just go check out Veeam's documentation. What this blog post is all about is helping you get those snapshots offloaded to a secondary FlashArray.
Let's say for example that you have a new FlashArray//C with tons of capacity because of all of that super-dense QLC flash. You like that Veeam can automate your FlashArray snapshots, provide application consistency, and give you granular VM, File, and App based restores. The problem is you just don't have the space to keep all those snaps around on your production array (remember, FlashArray snapshots are thin, but they still take space for changed and deleted data). Maybe however, it's a simple matter of just wanting to get them copied to your DR site for an extra layer of protection.
If you've glanced over Veeam's feature matrix for Integration with Storage Systems, you've no doubt noticed that there are a handful of individual storage products listed and then there is Universal Storage API Integrated Systems. FlashArray integration is built on this USAPI and sadly the USAPI doesn't support snapshot-only jobs for secondary arrays. I decided to take on this challenge and see if you could do it anyway (#spoiler, you can).
Adding Secondary Storage Support With a Post-Job Script.
Until quite recently there really wasn't a great workaround to replicate your Veeam created FlashArray snapshots to another FlashArray. FlashArray snapshot replication is based on the concept of a Protection Group and Veeam created snapshots aren't part of any Protection Group. In Purity//FA 6.1.0 however Pure introduce a brand new pair of CLI commands 'purevol send' and 'purevol get'. With these new commands, you can granularly control the replication of a FlashArray volume or snapshot. Now we have everything that we need to get Veeam to automate this process. The process will be to leverage a Veeam post-job PowerShell script to replicate any snapshots created during the backup to a secondary FlashArray.
How To Set It Up
Start off by making sure you have a connection set up between your source and target FlashArrays.
Next, download a copy of my script Invoke-PfaSendSnapAfterVBRJob.ps1 to your VBR Server. There are a few variables you'll need to customize for your environment.
Set the endpoint (IP or DNS name), username, and password of the FlashArray that is hosting the VMFS Datastore(s) for the VMs being protected (source) and the FlashArray where we will be copying the snapshots (target).
$sourceArrayEndpoint="fa-1.homelab.local"
$targetArrayEndpoint="fa-2.homelab.local"
$arrayUsername="pureuser"
$arrayPassword="pureuser"[Note on OAuth2] You can't use OAuth2 authentication yet. This is due to the fact that the purevol send command is so new that it hasn't been included in the PowerShell SDK just yet. That means the script has to pass it as a CLI call and for that we'll need user/password authentication. As soon as a new cmdlet is available to handle this API call I'll update the script and switch to OAuth2.
The next variable tells the script the name of a FlashArray Host Group. This host group should be created manually on the target FlashArray and each of your Veeam Proxy hosts should be added. The script will automatically connect our replicas to this Host Group in order for the Pure plug-in for Veeam to be able to see them and scan for the presence of VMs.
$PfaVeeamHostGroup="Veeam-Proxies"
Once you've made the necessary changes, save the script to a location on the VBR server which will be accessible by whatever service account is used to run Veeam. I saved mine to C:\VBRScripts\Invoke-PfaSendSnapAfterVBRJob.ps1.
Now we need to make sure the VBR server has all of the necessary prerequisites to run the script. It's tested to run in PowerShell 5.1 which is the default version on Windows Server 2019. Veeam installs the necessary PowerShell components when the VBR Server is deployed, so you should already have either the VeeamPSSnapIn (VBR v10) or VBR PowerShell Toolkit (VBR v11). The only thing you should really need to install is the PureStoragePowerShellSDK2. (Installation Instructions)
Create your backup job as usual, and for the backup repository select Pure Storage Snapshot (Primary snapshot only). Pick your retention policy (how many snapshots you want to keep.) Then open Advanced job settings.
Under the Scripts tab, check the box to run a script after the job and then browse for your script.
If you don't see the script, be sure PowerShell Files is selected under "File types to show."
Finish setting up the backup job as you normally would. The next time the job runs, you should see "Post-job script completed successfully." in the job log.
Let's See How It Works
Head over to the Source FlashArray and you should see your new snapshots. In my example I had a job that targeted VMs across two VMFS Datastores, so I ended up with two snapshots.
Now over to the Target FlashArray and sure enough, the snapshots have been replicated over here as well. Notice however that there are actually 2 additional snapshots. Why is this?
To answer that question, let's talk a bit more about how VBR is able to scan snapshots for the presence of VMs. It achieves this by actually mounting the snapshot's parent volume on one of the Veeam Proxy servers. Therein lies the problem. Since our snapshots were replicated from another FlashArray, they are disassociated (at least at the Target FlashArray level) from any source volume. To overcome this, the script clones a new volume from the replicated snapshot and then immediately takes a snapshot of that new volume. This snapshot will be identical in terms of the data it contains to the snapshot which was replicated; however it will now be connected to a parent volume that VBR can mount and scan.
You can also view the target FlashArray Storage to see the new volumes cloned from the replicated snapshots.
Drilling down into the details of one of these volumes will confirm that indeed one of the snapshots is a child of this volume. Additionally you can see that the script connected the volume to the Host Group for the Veeam Proxies as defined in the variable $PfaVeeamHostGroup. I want to reiterate that the script will only connect an existing host group, it will not attempt to create a host group if one does not exist. You must pre-create this host group and ensure all of your proxy servers with access to the FlashArray have been added as members if you want these snapshots to show up in Veeam under Storage Infrastructure.
Finally, let's take a look at the VBR Console and ensure our new snapshots are ready for recovery; and we find that indeed they are. These snapshots are fully recoverable including Instant VM, Guest Files, and Application Item Recovery.
Subsequent job runs will result in very similar behavior until you reach the retention policy set on the Veeam job. At that point snapshots that exceed the policy will be deleted on both the source and target FlashArrays.
There you have it, Snapshot-only Jobs for Secondary Storage Arrays with the Veeam Universal Storage API.