GPU virtual addresses are constant now so this should never be getting hit
anyway and userspace shouldn't break from them being ignored.

This is being done in preference to teaching the code how to deal with BOs
that exist at different virtual addresses within separate VMs.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>