Proxmox GPU Passthrough 가이드

DarkAcid

아래 내용은 레딧의 cjalas님의 가이드를 제 멋대로 번역한 글입니다
The Ultimate Beginner's Guide to GPU Passthrough (Proxmox, Windows 10) : homelab (reddit.com)

왕초보를 위한 Proxmox GPU Passthrough 가이드 

안녕하세요, 제 첫 초보를 위한 시리즈에 오신 여러분들을 환영합니다.  이 가이드는 Proxmox에서 VM에 GPU를 게스트 OS에 Passthrough 하는법을 알려드릴겁니다. 이 가이드는 Proxmox 가상화를 처음 사용하는 초보자에게 맞춰져있으며, GPU를  VM에서 사용할 수 있게 만드는 완전한 가이드입니다. 그러나 이 가이드는 모든 정보를 포함하고있지는 않으며 최선을 다해 처음부터 끝까지 GPU Passthrough에 필요한 자료들을 드릴 수 있도록 하겠습니다. 만약 문제가 있으시면 레딧에 /r/Proxmox/r/Homelabr/VFIO, or /r/linux4noobs 커뮤니티를 활용해 물어보세요.

시작하기전 (크래딧)

이 가이드는 레딧과, Proxmox 커뮤니티, 그리고 여러 가이드를 올려주신 분들의 도움으로 작성되었습니다. 만약 여기 포함되있지 않으시면 알려주세요.                                                   

주의사항: 이 가이드는 모든 시스템에 적용되지 않을 수 있습니다. 유저의 주의가 필요합니다. 이 가이드의 작성자는 당신의 서버, 집, 그리고 당신이 폭팔하는 일에 책임을 지지않습니다. Surgeon General Warning: do not operate this guide while under the influence of intoxicating substances. Do not let your cat operate this guide. You have been warned.

여기서부터 시작합니다 (Pre-configuration Checklist)

GPU Passthrough를 시작하기전 하드웨어와 소프트웨어의 설정은 중요합니다. 여기서 제 시스템과 소프트웨어의 사양을 알려드릴건데요, 이 가이드가 저와 같은 하드웨어와 소프트웨어에서도 작동하지 않을수있음을 명심하셔야 합니다. 이 가이드는 Proxmox 사용자를 대상으로 합니다. 당신의 하드웨어는 최소한 VT-d, interrupt mapping, and UEFI BIOS를 지원해야합니다.

 

하드웨어 사양:

Motherboard: Supermicro X9SCM-F (Rev 1.1 Board + Latest BIOS)

CPU: LGA1150 Socket, Xeon E3-1220 (version 2) 1

Memory: 16GB DDR3 (ECC, Unregistered)

GPU: 2x GTX 1050 Ti 4gb, 2x GTX 1060 6gb 2

소프트웨어 사양:

Latest Proxmox Build (5.3 as of this writing)

Windows 10 LTSC Enterprise (Virtual Machine) 3

노트:

1대부분 Xeon E3 CPUs에서 IOMMU 그룹이 지저분합니다, 추가적으로 설정이 필요합니다.

2같은 사양,모델, 브랜드의 멀티 GPU를 추천하지않습니다.

3아무 윈도우 10 ISO면 상관없습니다만, 마이크로소프트의 최신버젼 사용을 추천합니다.

Proxmox 설정

이 가이드는 당신이 Proxmox를 이미 설치했고 웹 인터페이스에 로그인하여 쉘 터미널에 접속할 수 있다고 가정하에 작성되었습니다. 만약 Proxmox의 설치법이 필요하시면 Proxmox공식 홈페이지에 "시작하기"를 읽어보시는 것을 추천드립니다

Step 1: Grub 설정변경하기

Intel CPU를 사용한다는 가정하에 Proxmox 서버에 SSH던 웹에서 노드 아래 novnc를 이용해 쉘 터미널을 사용하던,  /etc/default/grub 파일을 여십시오. 저는 nano를 사용합니다.

nano /etc/default/grub

아래 라인을 찾으십시오:

GRUB_CMDLINE_LINUX_DEFAULT="quiet"

그리고 이렇게 바꿔주세요:

Intel CPUs:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

AMD CPUs:

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on"

중요한 추가 커맨드

만약 passthrough가 실패했다면 추가 커맨드가 필요할 수있습니다. 만약 Xeon E3-12xx series와 비슷한 CPU를 사용중이라면 반드시 필요합니다. 이 추가 커맨드는 Proxmox에게 GPU를 사용하지 말라고 명령하는것입니다. 이것이 중요한 이유는 GPU와 CPU가 같은 IOMMU 그룹에 있을 경우 Passthrough가 실패할 위험이 있기때문입니다.

여기 제 grub 커맨드 설정입니다:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:off,efifb:off"

이 커맨드가 하는 일들은 다음과 같습니다:

A. Disabling the Framebuffer: video=vesafb:off,efifb:off (프레임 버퍼 무력화)

B. ACS Override for IOMMU groups: pcie_acs_override=downstream,multifunction (ASC에서 IOMMU 그룹 오버라이드)

grub 설정이 끝났으면 아래 커맨드를 실행하세요:

update-grub

Step 2: VFIO 모듈

Proxmox에서 몇가지의 VFIO 모듈들을 추가해야합니다.

파일 /etc/modules을 nano나 다른 텍스트 에디터로 수정합니다.

nano /etc/modules

아래 내용을 /etc/modules file에 복사/붙여넣기 해주세요:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

저장하시고 나가세요.

Step 3: IOMMU interrupt remapping

이 내용은 많이 설명하지 않습니다 그냥 아래 커맨드를 쉘에서 실행시키시면됩니다:

echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf
echo "options kvm ignore_msrs=1" > /etc/modprobe.d/kvm.conf

Step 4: Blacklisting Drivers

우리는 호스트에서 GPU를 사용하는걸 원하지 않습니다 그러므로 GPU 드라이버들을 블랙리스트에 추가합니다. 아래 커맨드를 쉘에서 실행해주세요:

echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf

Step 5: Adding GPU to VFIO

이 커맨드를 실행하세요:

lspci -v

쉘에 아래와 같은 내용이 뜰겁니다. VGA라고 적혀있는 라인을 찾아보세요 이렇게 생겼습니다.:

01:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1) (prog-if 00 [VGA controller])

01:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)

가장 앞에 번호를 기록해주세요 (예제: 01:00.0 and 01:00.1). 다음 설정을 위해 필요합니다.

아래 커맨드를 실행해주세요. 01:00를 위에서 적은 번호로 바꿔주세요:

lspci -n -s 01:00

이렇게 하시면 GPU카드의 벤더 ID가 output될겁니다 대부분 GPU ID하나 그리고 오디오 ID하나가 보입니다. 이렇게 생겼습니다:

01:00.0 0000: 10de:1b81 (rev a1)

01:00.1 0000: 10de:10f0 (rev a1)

우리가 원하는건 벤더 ID를 기록해두는 것입니다: 10de:1b81과 그리고 10de:10f0.

그리고 GPU 벤더 ID를  VFIO에 추가합니다 (본인의 벤더 ID로 바꿔주세요!!!!):

echo "options vfio-pci ids=10de:1b81,10de:10f0 disable_vga=1"> /etc/modprobe.d/vfio.conf

마지막으로 아래 커맨드를 실행해주세요:

update-initramfs -u

재시작하세요:

reset

이제 당신의 Proxmox 호스트는 GPU passthrough할 준비가 되었습니다.

 

 

------------------------------------------------------------------------------------------------------------------------------------------------------

 

 

 

Configuring the VM (Windows 10)

Now comes the 'fun' part. It took me many, many different configuration attempts to get things just right. Hopefully my pain will be your gain, and help you get things done right, the first time around.

Step 1: Create a VM

Making a Virtual Machine is pretty easy and self-explanatory, but if you are having issues, I suggest looking up the official Proxmox Wiki and How-To guides.

For this guide, you'll need a Windows ISO for your Virtual Machine. Here's a handy guide on how to download an ISO file directly into Proxmox. You'll want to copy ALL your .ISO files to the proper repository folder under Proxmox (including the VirtIO driver ISO file mentioned below).

Example Menu Screens

General => OS => Hard disk => CPU => Memory => Network => Confirm

IMPORTANT: DO NOT START YOUR VM (yet)

Step 1a (Optional, but RECOMMENDED): Download VirtIO drivers

If you follow this guide and are using VirtIO, then you'll need this ISO file of the VirtIO drivers to mount as a CD-ROM in order to install Windows 10 using VirtIO (SCSI).

For the CD-Rom, it's fine if you use IDE or SATA. Make sure CD-ROM is selected as the primary boot device under the Options tab, when you're done creating the VM. Also, you'll want to make sure you select VirtIO (SCSI, not VirtIO Block) for your Hard disk and Network Adapter.

Step 2: Enable OMVF (UEFI) for the VM

Under your VM's Options Tab/Window, set the following up like so:

Boot Order: CD-ROM, Disk (scsi0)
SCSI Controller: VirtIO SCSI Single
BIOS: OMVF (UEFI)

Don't Forget: When you change the BIOS from SeaBIOS (Default) to OMVF (UEFI), Proxmox will say something about adding an EFI disk. So you'll go to your Hardware Tab/Window and do that. Add > EFI Disk.

Step 3: Edit the VM Config File

Going back to the Shell window, we need to edit /etc/pve/qemu-server/<vmid>.conf, where <vmid> is the VM ID Number you used during the VM creation (General Tab).

nano /etc/pve/qemu-server/<vmid>.conf

In the editor, let's add these command lines (doesn't matter where you add them, so long as they are on new lines. Proxmox will move things around for you after you save):

machine: q35
cpu: host,hidden=1,flags=+pcid
args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'

Save and exit the editor.

Step 4: Add PCI Devices (Your GPU) to VM

r/homelab - The Ultimate Beginner's Guide to GPU Passthrough (Proxmox, Windows 10)

Look at all those GPUs

Under the VM's Hardware Tab/Window, click on the Add button towards the top. Then under the drop-down menu, click PCI Device.

Look for your GPU in the list, and select it. On the PCI options screen, you should only need to configure it like so:

All Functions: YES
Rom-Bar: YES
Primary GPU: NO
PCI-Express: YES (requires 'machine: q35' in vm config file)

Here's an example image of what your Hardware Tab/Window should look like when you're done creating the VM.

r/homelab - The Ultimate Beginner's Guide to GPU Passthrough (Proxmox, Windows 10)

Oopsies, make sure “All Functions” is CHECKED.

Step 4a (Optional): ROM File Issues

In the off chance that things don't work properly at the end, you MIGHT need to come back to this step and specify the ROM file for your GPU. This is a process unto itself, and requires some extra steps, as outlined below.

Step 4a1:

Download your GPU's ROM file

OR

Dump your GPU's ROM File:

cd /sys/bus/pci/devices/0000:01:00.0/
echo 1 > rom
cat rom > /usr/share/kvm/<GPURomFileName>.bin
echo 0 > rom

Alternative Methods to Dump ROM File:

a. Using GPU-Z (recommended)

b. Using NVFlash

Step 4a2: Copy the ROM file (if you downloaded it) to the /usr/share/kvm/ directory.

You can use SFTP for this, or directly through Windows' Command Prompt:

scp /path/to/<romfilename>.rom myusername@proxmoxserveraddress:/usr/share/kvm/<romfilename>.rom

Step 4a3: Add the ROM file to your VM Config (EXAMPLE):

hostpci0: 01:00,pcie=1,romfile=<GTX1050ti>.rom

NVIDIA USERS: If you're still experiencing issues, or the ROM file is causing issues on its own, you might need to patch the ROM file (particularly for NVIDIA cards). There's a great tool for patching GTX 10XX series cards here: https://github.com/sk1080/nvidia-kvm-patcher and here https://github.com/Matoking/NVIDIA-vBIOS-VFIO-Patcher. It only works for 10XX series though. If you have something older, you'll have to patch the ROM file manually using a hex editor, which is beyond the scope of this tutorial guide.

r/homelab - The Ultimate Beginner's Guide to GPU Passthrough (Proxmox, Windows 10)

Example of the Hardware Tab/Window, Before Windows 10 Installation.

Step 5: START THE VM!

We're almost at the home stretch! Once you start your VM, open your noVNC / Shell Tab/Window (under the VM Tab), and you should see the Windows installer booting up. Let's quickly go through the process, since it can be easy to mess things up at this junction.

Final Setup: Installing / Configuring Windows 10

r/homelab - The Ultimate Beginner's Guide to GPU Passthrough (Proxmox, Windows 10)

Copyright(c) Jon Spraggins (https://jonspraggins.com)

If you followed the guide so far and are using VirtIO SCSI, you'll run into an issue during the Windows 10 installation, when it tries to find your hard drive. Don't worry!

r/homelab - The Ultimate Beginner's Guide to GPU Passthrough (Proxmox, Windows 10)

Copyright(c) Jon Spraggins (https://jonspraggins.com)

Step 1: VirtIO Driver Installation

Simply go to your VM's Hardware Tab/Window (again), double click the CD-ROM drive file (it should currently have the Windows 10 ISO loaded), and switch the ISO image to the VirtIO ISO file.

r/homelab - The Ultimate Beginner's Guide to GPU Passthrough (Proxmox, Windows 10)

Copyright(c) Jon Spraggins (https://jonspraggins.com)

Tabbing back to your noVNC Shell window, click Browse, find your newly loaded VirtIO CD-ROM drive, and go to the vioscsi > w10 > amd64 sub-directory. Click OK.

Now the Windows installer should do its thing and load the Red Hat VirtIO SCSI driver for your hard drive. Before you start installing to the drive, go back again to the VirtIO CD-Rom, and also install your Network Adapter VirtIO drivers from NetKVM > w10 > amd64 sub-directory.

r/homelab - The Ultimate Beginner's Guide to GPU Passthrough (Proxmox, Windows 10)

Copyright(c) Jon Spraggins (https://jonspraggins.com)

IMPORTANT #1: Don't forget to switch back the ISO file from the VirtIO ISO image to your Windows installer ISO image under the VM Hardware > CD-Rom.

When you're done changing the CD-ROM drive back to your Windows installer ISO, go back to your Shell window and click Refresh. The installer should then have your VM's hard disk appear and have windows ready to be installed. Finish your Windows installation.

IMPORTANT #2: When Windows asks you to restart, right click your VM and hit 'Stop'. Then go to your VM's Hardware Tab/Window, and Unmount the Windows ISO from your CD-Rom drive. Now 'Start' your VM again.

Step 2: Enable Windows Remote Desktop

If all went well, you should now be seeing your Windows 10 VM screen! It's important for us to enable some sort of remote desktop access, since we will be disabling Proxmox's noVNC / Shell access to the VM shortly. I prefer to use Windows' built-in Remote Desktop Client. Here's a great, simple tutorial on enabling RDP access.

NOTE: While you're in the Windows VM, make sure to make note of your VM's Username, internal IP address and/or computer name.

Step 3: Disabling Proxmox noVNC / Shell Access

To make sure everything is properly configured before we get the GPU drivers installed, we want to disable the built-in video display adapter that shows up in the Windows VM. To do this, we simply go to the VM's Hardware Tab/Window, and under the Display entry, we select None (none) from the drop-down list. Easy. Now 'Stop' and then 'Start' your Virtual Machine.

NOTE: If you are not able to (re)connect to your VM via Remote Desktop (using the given internal IP address or computer name / hostname), go back to the VM's Hardware Tab/Window, and under the PCI Device Settings for your GPU, checkmark Primary GPU**. Save it, then 'Stop' and 'Start' your VM again.**

Step 4: Installing GPU Drivers

At long last, we are almost done. The final step is to get your GPU's video card drivers installed. Since I'm using NVIDIA for this tutorial, we simply go to http://nvidia.com and browse for our specific GPU model's driver (in this case, GTX 10XX series). While doing this, I like to check Windows' Device Manager (under Control Panel) to see if there are any missing VirtIO drivers, and/or if the GPU is giving me a Code 43 Error. You'll most likely see the Code 43 error on your GPU, which is why we are installing the drivers. If you're missing any VirtIO (usually shows up as 'PCI Device' in Device Manager, with a yellow exclamation), just go back to your VM's Hardware Tab/Window, repeat the steps to mount your VirtIO ISO file on the CD-Rom drive, then point the Device Manager in Windows to the CD-Rom drive when it asks you to add/update drivers for the Unknown device.

Sometimes just installing the plain NVIDIA drivers will throw an error (something about being unable to install the drivers). In this case, you'll have to install using NVIDIA's crappy GeForce Experience(tm) installer. It sucks because you have to create an account and all that, but your driver installation should work after that.

Congratulations!

After a reboot or two, you should now be able to see NVIDIA Control Panel installed in your Windows VM, as well as Device Manager showing no Code 43 Errors on your GPU(s). Pat yourself on the back, do some jumping jacks, order a cake! You've done it!

r/homelab - The Ultimate Beginner's Guide to GPU Passthrough (Proxmox, Windows 10)

Multi-GPU Passthrough, it CAN be done!

Credits / Resources / Citations

  1. https://pve.proxmox.com/wiki/Pci_passthrough

  2. https://forum.proxmox.com/threads/gpu-passthrough-tutorial-reference.34303/

  3. https://vfio.blogspot.com/2014/08/iommu-groups-inside-and-out.html

  4. https://forum.proxmox.com/threads/nvidia-single-gpu-passthrough-with-ryzen.38798/

  5. https://heiko-sieger.info/iommu-groups-what-you-need-to-consider/

  6. https://heiko-sieger.info/running-windows-10-on-linux-using-kvm-with-vga-passthrough/

  7. http://vfio.blogspot.com/2014/08/vfiovga-faq.html

  8. https://passthroughpo.st/explaining-csm-efifboff-setting-boot-gpu-manually/

  9. http://bart.vanhauwaert.org/hints/installing-win10-on-KVM.html

  10. https://jonspraggins.com/the-idiot-installs-windows-10-on-proxmox/

  11. https://pve.proxmox.com/wiki/Windows_10_guest_best_practices

  12. https://docs.fedoraproject.org/en-US/quick-docs/creating-windows-virtual-machines-using-virtio-drivers/index.html

  13. https://nvidia.custhelp.com/app/answers/detail/a_id/4188/~/extracting-the-geforce-video-bios-rom-file

  14. https://www.overclock.net/forum/69-nvidia/1523391-easy-nvflash-guide-pictures-gtx-970-980-a.html

  15. https://medium.com/@konpat/kvm-gpu-pass-through-finding-the-right-bios-for-your-nvidia-pascal-gpu-dd97084b0313

  16. https://www.groovypost.com/howto/setup-use-remote-desktop-windows-10/

Thank you everyone!

서버운영 초보

메인컴퓨터: 5800x/x570itx/16G/1080evga se

보조컴퓨터: 2700x/B450m/16G/1060 msi 3gb

- EXSI: Linux-Plex, Linux-SMB서버, Linux-Zerotier Bridge(예정)

 

클라우드

Oracle: Zerotie, Linode: 공부용 웹사이트 2개, Vultr: Pritunl, AWS: 미디어 프로덕션

적용중인 도장이 없습니다.

* 가입일 : 2021-11-05
* 총 게시글 수 : 66
* 총 댓글 수 : 489
* 추천받은 게시글 개수 : 56
* 추천받은 댓글 개수 : 202
댓글
9
  • 만곰이
    2022.09.15

    감사합니다. 퇴근하고 다시 삽질 열심히 해보겠습니다. ^^

  • theb1ue
    2022.09.15

    대단한 가이드네요! 작성하시느라 고생 하셨습니다

  • 행복하세
    2022.09.15

    아 esxi 에서 지포스 패스쓰루 하다하다 안되서 포기했는데

    프록스목스로 넘어가야 되나........

  • 행복하세
    DarkAcid
    작성자
    2022.09.16
    @행복하세 님에게 보내는 답글

    저는 개인적으로는 ESXI에서 패스쓰루가 한 100만배 쉬웠던거 같긴합니다. Proxmox 그럽은 너무 어려웠어요. 저도 레딧에서 찾은 이 가이드가 아니라면 포기했을거에요 ㅎㅎ

  • 만곰이
    2022.09.19

    지난주 Test 결과 실패입니다. ㅠ.ㅠ 공유해주신 내용대로 진행했는데 이번주, 다음주에 차근차근 다시 해보려고 합니다.

  • 만곰이
    DarkAcid
    작성자
    2022.09.29
    @만곰이 님에게 보내는 답글

    저도 커널 업데이트 이후 안되더군요 5.13버젼은 된다고하니 돌아가야할거 같아요

  • DarkAcid
    만곰이
    2022.09.30
    @DarkAcid 님에게 보내는 답글

    혹시나 해서 최신 커널로 업뎃해 봤는데 안되더라구요. 일단 손 놓고 있는 상황이고, 5.13으로 내려야 하는데 방법을 몰라서 딴짓만 하고 있습니다. ㅎㅎㅎㅎ

  • DarkAcid
    만곰이
    2022.10.05
    @DarkAcid 님에게 보내는 답글

    Proxmox Forum에서 찾은 댓글인데 아직 적용은 못해봤습니다.

     

    https://forum.proxmox.com/threads/gpu-passthrough-issues-on-kernel-5-15.108702/#post-467404

     

    ------------------------------------------------------------------------------------------------------------------------------

    I finally figured it out how to get this working. All I had to do were the following:

    1. Create a bash job that runs a boot with the follow script

     

    Code:

    #!/bin/sh

     

    echo 1 > /sys/bus/pci/devices/0000\:0x\:00.0/remove

    echo 1 > /sys/bus/pci/rescan

     

     

    echo 0 > /sys/class/vtconsole/vtcon0/bind

    echo 0 > /sys/class/vtconsole/vtcon1/bind

     

    echo vesa-framebuffer.0 > /sys/bus/platform/drivers/vesa-framebuffer/unbind

    echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind

    echo simple-framebuffer.0 > /sys/bus/platform/drivers/simple-framebuffer/unbind

     

    Please replace the device ID to the one that you have for your video card

     

    2. Make sure that GRUB_CMDLINE_LINUX_DEFAULT has efifb:off as part of the options

     

    3. update-grub (I sometimes forget this step but just in case)

     

    Hopefully this should work for you. If you like, here is where I found the solution: https://forum.proxmox.com/threads/problem-with-gpu-passthrough.55918/

     

    Happy passthrough!

  • 우아아아앙
    2022.09.27

    감사합니다. PROXMOX 설치하면 한번 해보겠습니다.

댓글 쓰기
권한이 없습니다.