We previously only did this for push buffers, but an upcoming patch will
need to attach fences to all VMAs to resolve another issue.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>