Issues after forcibly shutting down a vGPU enabled Machine - Reset GPU on Xenserver

-
Had an odd scenario today where a VM with vGPU enabled on XenServer became unresponsive and would not restart/shutdown, force shutdown/reset, or really do anything.  So I had to reference a handful of Citrix articles to remember how to force it, but then came to another issue that it didn't kill the GPU portion...  So here we go!

First we have to kill the VM on XenServer: (https://support.citrix.com/article/CTX220777 & https://support.citrix.com/article/CTX215974)
  1. Disable High Availability (HA) so you don’t run into issues
  2. Log into the Xenserver host that is running your VM with issues via ssh or console via XenCenter
  3. Run the following command to list VMs and their UUIDs
    xe vm-list resident-on=<uuid_of_host>
  4. First you can try just the normal shutdown command with force
    xe vm-shutdown uuid=<UUID from step 3> force=true
  5. If that just hangs, use CONTROL+C to kill it off and try to reset the power state.  The force is required on this command
    xe vm-reset-powerstate uuid=<UUID from step 3> force=true
  6. If the VM is still not shutdown, we may need to destroy the domain
  7. Run this command to get the domain id of the VM.  It is the number in the first row of output. The list will be the VMs on the host.  Dom0 will be the host itself and all numbers after are running VM
    list_domains
  8. Now run this command using the domain ID from the output of step 7
    xl destroy <Domain>
Now at this point this normally solves it... BUT I couldn't start anything on the vacated pGPU... SO we continue...
  1. Check the gpu using nvidia's built in tools by running
    nvidia-smi This showed the GPU at 100% utilization with no processes running on it...


  2. At this point we wanted to reset the GPU by using
    nvidia-smi --gpu-reset -i * [replace * with number of the GPU in question from nvidia-smi]
    This did not work as it said it was in use by one more more processes...

  3. Run this command to see processes not listed in nvidia-smi
    fuser -v /dev/nvidia*
  4. Normally we could try to kill any processes listed here by using
    kill -9 PID
  5. We only had a service listed (xcp-rrdd-gpumon).  It would respawn if we killed it so we need to stop it using
    service xcp-rrdd-gpumon stop
  6. Then we checked processes again using the same command as 3
    fuser -v /dev/nvidia*
  7. Now there were no processes running and we could succesfully reset the GPU
    nvidia-smi --gpu-reset -i *
  8. Check nvidia-smi to ensure this worked and then restart the service we stopped
    service xcp-rrdd-gpumon start
At this point nvidia-smi should show the GPU reset with no load, no memory usage and VMs should be able to start on it again (at least ours did)!

 

Also learned there are so many other cool features packed into nvidia-smi I hadn't noticed before when searching for solutions here.  Maybe a learning journey for another day!

Comments

Post a Comment

Popular posts from this blog

Autodesk in Virtual Environments and What is "Allowed"

-
Yesterday at Autodesk University 2018, I attended a session (SD227281) that was supposed to be about "Best Practices for Virtualizing Your Autodesk Software."  What ended up happening was some sort of confusing and illogical announcement about new Terms and Conditions around what is allowed and what is not allowed in terms of licensing models for virtualized environments.  To categorize this session as devolving into an infuriating train wreck would be an accurate description. After cooling down and doing some research yesterday I am even more confused, and some of what we heard was seemingly wrong. First I'll summarize what was "announced" and what was discussed in that session by the two senior Autodesk people (John Looper and Chris Kayser). Announced in the presentation: Autodesk will permit virtualization for: - Subscriptions with single-user access - Enterprise Business Agreements - Server-based components (Think Autodesk Licensing Server
Read more

Dell R740 issues with XenServer 7.1/7.2

-
Image
My last post talked about our new Dell R740s.  Response to that post have pointed out that there are a few things worth following up with since people have been having some issues with this server/Intel Gold 61xx Processors. I had a few minor issues getting XenServer installed, but mostly around storage and networking. *I should note that my servers are showing crash dumps in XenCenter, but everything is working perfectly.  Waiting on my support case to have those SSRs analyzed (case originally started to ask about support for the H740p card in XS7.2).  I will post and update when I hear back tomorrow morning. Storage Controller - H740p We went with the H740p card which is not supported on XenServer 7.2 (although support tells me it may be soon) so we had to go to 7.1.1 (7.1 CU1) and install the driver at  https://support.citrix.com/article/CTX226401 .  If you had to load this to detect disks, make sure you install it AGAIN at the end during supplemental packs to actually have
Read more

NVIDIA GRID on Pascal - A Card Comparision

-
Image
I was all excited to post this, as I spent a decent amount of time on this subject leading up to ordering our new servers, and hadn't seen anyone else address it yet.  Then yesterday Jan Hendricks posted his thoughts on the subject which mirrored much of my findings ( https://jhmeier.com/2017/11/15/data-comparison-of-nvidia-grid-tesla-p4-and-p40/ ).  I guess that motivated me to finally finish my thoughts and get it posted as well! NVIDIA GRID 5.0 & Tesla Pascal Cards Photo Credit:  http://www.nvidia.com/object/accelerate-inference.html With NVIDIA GRID release 5.0 came support for Pascal GPUs.  While this brought many things (covered many other places), the most interesting part of this for me was that now ANY Tesla Pascal GPU works with NVIDIA GRID.  This opened us up to a wider array of options right away with 3 PCIe cards already generally available and an MXM option.  While NVIDIA pushed the P40 as the natural replacement for the M60, I wanted to look at all of o
Read more