Vsphere Permissions to "Delete from Disk"

I have a select number of users that need to be able to delete VMs from Vsphere. I created a new role with the virtual machine > Inventory > Remove privledge. The problem is, these users are also part of a domain group\role that does not have the remove priviledge. I understand least restrictive wins. Is there anyway around this (aside from removing the user from the less restrictive group or redoing all of our perms)?

Thanks!