In a previous post I detailed the performance and RPO improvements that can be gained from implementing backup from FlashArray snapshots with Veeam Backup and Replication. In this post I'll explore the implications when that FlashArray is also configured for SafeMode.
How does FlashArray Handle Volume and Snapshot Deletion?
When FlashArray volumes and snapshots are destroyed (FlashArray's term for being deleted), these objects are not immediately removed from FlashArray. Instead, they are placed into a destroyed state. 24 hours after being destroyed, these objects will be permanently deleted in a process called eradication. At any time prior to eradication, a user may recover a destroyed object. Alternatively, a destroyed object may be immediately eradicated by a user prior to 24 hours. The 24 hour countdown from when an object is destroyed until when it is automatically eradicated is know as the eradication timer.
What is SafeMode?
SafeMode is a feature that may be enabled to further reduce the chance of unintentional data loss. SafeMode changes this process in three important ways.
1. SafeMode snapshots can't be eradicated (deleted), modified, or encrypted.
2. SafeMode snapshot cadence and eradication scheduling are customizable.
3. SafeMode may only be configured or modified by Pure Technical Support working with an authorized designee from your organization.
Pure Plug-In For Veeam
The Pure plug-in for Veeam can be a very useful tool and is in fact required to enable Veeam Backup from Storage Snapshots or Veeam Snapshot Only Jobs on FlashArray. In addition to those backup related features however, the plug-in also enables Veeam restores for VMs stored on FlashArray snapshots even if they were not created through the Veeam console. In order to accomplish this, Veeam periodically scans the FlashArray for the presence of VMs. Any FlashArray snapshots are temporarily copied so that they may be mounted to a Veeam Proxy server and scanned. Once the scan is complete, those temporary volume copies are deleted and immediately eradicated. If however SafeMode is enabled, these copies will be subject to the eradication timer and will persist until the timer has expired.
Here is an example of what those temporary copies will look like while pending eradication on FlashArray with SafeMode enabled. They begin "VEEAM-ExportLUNSnap-...".
Veeam Backup from Storage Snapshots
I detailed this process in depth in a previous post, but a few reminders...
When Veeam backup from storage snapshots is used to protect VMs stored on FlashArray hosted VMFS datastores, it creates volume copies on FlashArray. This might be a little confusing since the Veeam feature is called "backup from storage snapshots" but with FlashArray, creating volume copies instead of snapshots is actually a more efficient way of implementing this feature.
The number of volume copies created per job depends on a two factors:
One copy for each datastore that a VM in the job lives on. In other words, if your backup job is targeting 10 VMs, 5 on datastore-1 and 5 on datastore-2, Veeam will copy both datastores for each job run.
If the optional "Limit processed VM count per storage snapshot to [number]" setting is enabled in the Veeam backup job, this will create additional volume copies. The number of copies will be the total VM count in the backup job divided by the number of VMs per snapshot as defined in the setting.
How SafeMode Impacts Veeam Backup from Storage Snapshots
When you enable SafeMode on FlashArray, all volumes are protected against manual eradication. This includes those volume copies created by Veeam during a backup job with backup from storage snapshots enabled. This means that at the end of each Veeam job when it attempts to clean up the temporary volume copy that was created, it will be able to destroy the copy, but will not be able to eradicate it. The destroyed volume will continue to exist in the destroyed state until the eradication timer for that volume has expired.
You can see from the FlashArray audit trail that the volume in a test run was destroyed but not eradicated.
You can also see that the temporary volume begining with "VEEAM-StorageLUNSnap-..." has been destroyed but is pending eradication until the expiration of the eradication timer. In my lab, the SafeMode eradication timer is set to 24 hours.
Considerations
FlashArray volume copies are incredibly efficient and when created consume almost no capacity. Notice in the screenshot above, my destroyed volume only consumes about 950K. That said, they will increase in capacity over time as changes are made to the original volume they were copied from. This is because those copies will hold on to any data that has been changed or deleted from the original volume. With SafeMode enabled, you need to plan for the additional capacity required to store these destroyed but not yet eradicated volumes for the duration of your eradication timer. The capacity required will be based on several factors including:
How many Veeam jobs are enabled with backup from storage snapshots.
If there are multiple jobs that protect VMs on the same datastore, since that datastore will be copied multiple times, once for each job.
The change rate of the datastore volumes.
The duration of the SafeMode eradication timer.
If you are concerned about limiting the capacity impact of using Veeam backup from storage snapshots with FlashArray SafeMode, here are some things you can do.
Consider which VMs will benefit most from backup from storage snapshots and only enable the feature for backup jobs that protect those VMs. Typically, this would be VMs that are either very large or have a very high change rates.
Limit the VMs that are stored on a particular datastore to only those you want to protect with backup from storage snapshots. When Veeam creates a volume copy on FlashArray, it copies the entire datastore including all VMs that are stored there, whether or not they are part of the current backup job.
Make sure you don't have multiple recurring backup jobs that protect the same VMs. If so, you'll end up creating a volume copy for each job that runs.
Be aware of the "Limit processed VM count per storage snapshot to [number]" setting and if you decide to enable it, understand how it will increase your volume copy count.
Additionally you may want to reduce the number of volume copies that are created when Veeam scans FlashArray snapshots for the presence of VMs. To do this, edit the settings for your FlashArray in the Veeam Console.
In the Access Options section, customize the Volumes to Scan setting and select the minimum number of volumes that should be scanned. This list should only include those volumes that are used as VMFS datastores which contain VMs you may want to restore with Veeam.
Summary
Many Pure Storage FlashArray and Veeam Backup and Replication customers love leveraging the integration between these to solutions to improve backup performance and reduce RPO. FlashArray customers also enjoy the added peace of mind that FlashArray SafeMode delivers in reducing the chance of unintended or malicious data loss. With just a little planning, it's possible to bring both of these technologies together to increase the efficiency and security of your VMware environment.