More backup admins should use Veeam Persistent Guest Agent v11

The Veeam Persistent Guest Agent introduced in version 11 is a boon you may have missed for service providers and enterprise customers alike.

Veeam has not been shy in touting the benefits of the Veeam Persistent Guest Agent in blog posts and announcements. However, with all the other great features that came with version 11, I feel it has a chance of being overlooked by backup administrators. I want to reiterate some of the strengths of this feature to encourage more backup administrators to use it. Rather than repeat what they’ve said themselves, I’d like to share the successes we’ve seen at Deft with the new feature, as well as some unexpected ones.

With the massive increase of ransomware attacks over the last few years, it’s little doubt that the Persistent Guest Agent was developed in part to help mitigate against them. Why and how would that be so? I had a relatively uncommon experience long ago – I had seen firsthand a few of the earliest ransomware attacks all the way back in 2010. Ransomware back then came in much like it does today – an infected email attachment for example, that then spread through network shares, encrypting files and demanding a Bitcoin ransom or some other less-traceable monetary services that existed at the time and could be utilized by criminals.

However, these attacks were highly crude and ineffective compared to today. In the cases I experienced over a decade ago, we could simply use Windows Volume Shadow Copies we were already utilizing at the time to reverse cases of human error. Instead, in this case of this inept ransomware, we used Volume Shadow Copy restores to restore entire directory trees in an instant.

And even then, we still had our trusty backup server, only accessible by approved admins but still on the same domain and network, that we could have used to save the day as well. When we got familiar with these strange ransomware attacks we were seeing, it was clear that resolving the issue was easy. We still took another lesson however as well – we needed to mitigate risk by making our Windows file share permissions more granular based on an individual employee’s job role to minimize the impact of a single instance of ransomware, as well as just best security practice in a highly information-sensitive environment.

Over the years, however, ransomware criminals began to become much more clever with their tools. They began deleting Volume Shadow Copies, making the trivial fix of reverting back we used to use impossible. More troubling, they began using an increasing number of system exploits to gain privilege escalation, and attack systems only accessible by admins, even if the initial infection began on an unprivileged domain account. With these extra permissions, they started to attack backup servers. Not even just agent-based backup servers, but the backup servers themselves, a whole level of abstraction further, by compromising the server and altering backup retention, running several backups to age out the “good” backups, and more. As a result, we see increasing guidance these days from Veeam and other backup vendors to put your backup infrastructure off the domain – whether they be a non-domain machine or a separate domain just for backups.

What does this have to do with the Veeam Persistent Guest Agent? Well, if you’re now backing up systems cross-domain or even across a firewall, you may now have to rely on VIX for application-aware backups. The alternative to VIX is to take the difficult choice between putting a Guest Interaction Proxy itself in the DMZ and opening up a limited set of firewall ports directly to the Veeam backup server, or opening a large number of ports from the DMZ to your Guest Interaction Proxy in a trusted zone.

If you choose the administratively easier VIX path, this choice comes with several interesting and potentially dangerous security caveats – mainly that you must choose between either using the original default Local Administrator credentials or the default Domain Administrator credentials otherwise you’ll have to do things like completely disabling UAC on the client servers. Neither of these are particularly great security practices.

The other option is that you can skip VIX and go with a Guest Interaction Proxy that is reachable by application-aware servers directly. While this is labor-intensive to manage in service provider and complex enterprise network scenarios, the Veeam Persistent Guest Agent comes in and helps make this a more manageable choice now. It simplifies which ports need to be open on a firewall so you do not need to place the Guest Interaction Proxy on the same network as your DMZ servers if you don’t wish to do so. This is all great, but this is what Veeam blogs have already underlined. Taking this a step further, how does this help in practical terms?

Benefits for corporate environments:

• DMZ environments – larger corporate environments can now more safely and efficiently perform application-aware backups of environments even when the Guest Interaction Proxy is behind a firewall due to the simplified network firewall configurations afforded by the Veeam Persistent Guest Agent. With a much smaller set of ports to open, it is easier for your network engineers to maintain and allows more straightforward data to provide to security auditors.
• Cross-domain, across firewall backup environments – if you’re a backup administrator in a corporate environment that has followed newer best practices of keeping your backup server off your main corporate user domain, great for you! And now with the Veeam Persistent Guest Agent, you won’t have to rely on the built-in Local or Domain administrator accounts for application-aware backups. There is no longer any need to maintain VIX-level access, nor accounts with Admin-share-level access.

Benefits for service provider environments:

• Service Providers would prefer as a best practice to not have to request their customers disable UAC on guest VMs, nor have to ask for the default Local or Domain administrator accounts or even Admin-share-level access, a large benefit to both parties.

Benefits for all environments:

• There is a benefit I have personally experienced that I had not seen documented anywhere before that I would like to share. Anyone who has tried to implement Application-Aware SQL server Transaction Log backups, especially in a service provider environment, has likely had to take some time to get it working properly. It can take some trial and error to get them working properly and troubleshooting is complex, even with the help of Veeam Support. I suspect the same would be true for corporate environments trying to do this over a firewall and into a DMZ.

The benefit I found is that installing the Veeam Persistent Guest Agent on the SQL client VMs seems to offer much more precise error messages in some circumstances. In one case, it took a generic timeout message and clarified it into one that explicitly stated that the Persistent Guest Agent could not connect to the Guest Interaction Proxy. This led to the “ah-ha!” moment – the firewall was configured to allow traffic from the Veeam server to the SQL VM, but traffic sourced from the SQL VM was not allowed back due to the firewall configuration. Transaction Log backups require traffic initiation from the Guest VM to the Guest Interaction Proxy, so firewall rules that allow both directions between these two pieces of infrastructure are necessary.

I hope you had a chance to learn something from this post and how the Veeam Persistent Guest Agent may help the security posture and the operational efficiency of your environments. I personally will be standardizing on the Persistent Guest Agent for the benefits of the more verbose logging messages I have seen during troubleshooting.