Here you will find required patches and notes about NVIDIA and VGA Pass Through.

Please be informed that

ABOUT PATCHES
  1. I am not the author who wrote the original patches. I've just tried to maintain the patches I found from Xen mailing-list. Since Summer 2011, I've begun to test Xen VGA PassThrough
  2. I try to test patches once a week (week-end)
  3. LAST UPDATE: 03/25/2012 / LAST XEN REVISION: 25099
BEFORE YOUR TEST
  1. You have to read the required documentation taken from Xen Wiki: Xen VGA PassThrough
  2. You must have the specific hardware before doing any test: NVIDIA graphic card, specific processor Intel (Vt-d) or AMD (IOMMU), a specific motherboard VT-d capable

1. Examples: Video for playing Crysis 2 single player and screenshots for Windows XP

  1. Here is a (poor) video for Crysis 2 http://www.youtube.com/watch?v=3SaYO0ERW44 I did using my old GT 440 (before I bought my GTX 460).
  2. Here are a few screenshots using CPUZ64



2. Tested Hardware

Item Reference Comment
Processor Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz
Motherboard Manufacturer: MSI / Product Name: H61MU-E35 (MS-7680) Bios Version V10.11
Graphic Card (*) GTX 460 GTX 460 SE EVGA 1024 MB 01G-P3-1366-ET
Graphic Card GTX 560 TI GTX 560 TI Point Of View 2048 MB GDDR5
(*) : It was my previous card ( before 01/09/2012)

3. Xen Patches

All patches are applied from xen-unstable staging (http://xenbits.xensource.com/staging/xen-unstable.hg/)
From Revision To Revision Kernel / Linux Distrib Download Comments
25099 24888 3.1.10 / Debian 6.0.4 64 bits hhttp://www.davidgis.fr/download/xen-4.2_rev24798_gfx-passthrough-patchs.tar.bz2 A new file tools/firmware/hvmloader/rombios.c has been pushed by Julian (maintainer). A lot of things have been moved to this new file (revisions from 24782 to 24785)
24563 24779 3.1.10 / Debian 6.0.2 64 bits http://www.davidgis.fr/download/xen-4.2_rev24463_gfx-passthrough-patchs.tar.bz2 Path for patch 'pass-through.c' has changed from 'tools/ioemu-remote/hw/pass-through.c' to 'tools/qemu-xen-traditional-dir-remote/hw/pass-through.c'
24407 24547 3.1.4, 3.1.6, 3.1.8 / Debian 6.0.2 64 bits http://www.davidgis.fr/download/xen-4.2_rev24232_gfx-passthrough-patchs.tar.bz2 May not work from 24492 to 24519 due to revision 24492. Fixed in revision 24520
24232 24407 3.1.4 / Debian 6.0.2 64 bits http://www.davidgis.fr/download/xen-4.2_rev24232_gfx-passthrough-patchs.tar.bz2
24014 24228 3.1.0 / Debian 6.0.2 64 bits http://www.davidgis.fr/download/gfx_patchs_gtx-460-se-evga.tar.bz2
23799 23799 2.6.39.3 / Debian 6.0.2 64 bits http://www.davidgis.fr/download/gfx_patch.tar.bz2

4. Linux Kernel .config

My tests are based on a simple Debian 6.0.2 64 bits.

Notice: since 3.1.0, I used to compile Xen including all "Xen capabilitites" in the kernel (=Y) not as modules (=M).

root@mercury:~# grep -i xen /boot/config-3.1.6
CONFIG_XEN=y
CONFIG_XEN_DOM0=y
CONFIG_XEN_PRIVILEGED_GUEST=y
CONFIG_XEN_PVHVM=y
CONFIG_XEN_MAX_DOMAIN_MEMORY=128
CONFIG_XEN_SAVE_RESTORE=y
# CONFIG_XEN_DEBUG_FS is not set
# CONFIG_XEN_DEBUG is not set
CONFIG_PCI_XEN=y
CONFIG_XEN_PCIDEV_FRONTEND=y
CONFIG_XEN_PCIDEV_FE_DEBUG=y
CONFIG_XEN_BLKDEV_FRONTEND=y
CONFIG_XEN_BLKDEV_BACKEND=y
CONFIG_NETXEN_NIC=y
CONFIG_XEN_NETDEV_FRONTEND=y
CONFIG_XEN_NETDEV_BACKEND=y
CONFIG_INPUT_XEN_KBDDEV_FRONTEND=y
CONFIG_HVC_XEN=y
CONFIG_XEN_WDT=y
CONFIG_XEN_FBDEV_FRONTEND=y
# Xen driver support
CONFIG_XEN_BALLOON=y
CONFIG_XEN_SELFBALLOONING=y
CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_XEN_DEV_EVTCHN=y
CONFIG_XEN_BACKEND=y
CONFIG_XENFS=y
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
CONFIG_XEN_XENBUS_FRONTEND=y
CONFIG_XEN_GNTDEV=y
CONFIG_XEN_GRANT_DEV_ALLOC=y
CONFIG_XEN_PLATFORM_PCI=y
CONFIG_SWIOTLB_XEN=y
CONFIG_XEN_TMEM=y
CONFIG_XEN_PCIDEV_BACKEND=y
Here are my configs
Kernel Download Comments
3.1.10 http://www.davidgis.fr/download/config-3.1.10.tar.bz2 Tested 01/28/2012
3.1.8 http://www.davidgis.fr/download/config-3.1.8.tar.bz2 Tested 01/10/2012
3.1.6 http://www.davidgis.fr/download/config-3.1.6.tar.bz2 Tested 12/25/2011
3.1.4 http://www.davidgis.fr/download/config-3.1.4.tar.bz2 Tested date < 12/25/2011
3.1.0 http://www.davidgis.fr/download/config-3.1.0.tar.bz2
2.6.39.3 http://www.davidgis.fr/download/config-2.6.39.3.tar.bz2

5. Tested domU

domU Graphic card Nvidia driver (*) Comments
Windows XP 32 / 64 Bits
  • GTX 460 SE EVGA 1024 MB
  • GTX 560 Ti Point Of View 2048 MB GDDR5
  • 275.33
  • 275.50
  • [+] Crysis 2 works very well, single player mode, resolution 1920x1080
  • [-] It it not possible to restart domU without restarting dom0
Ubuntu Lucid 10.04.4 64 Bits
  • GTX 460 SE EVGA 1024 MB
  • GTX 560 Ti Point Of View 2048 MB GDDR5
  • 275.28 aka NVIDIA-Linux-x86_64-275.28.run
  • 275.43 aka NVIDIA-Linux-x86_64-275.43.run
  • [+] Quake 3 works very well using ioquake3 but you need to build ioquake3 from sources
    • svn co svn://svn.icculus.org/quake3/trunk ioquake3
      cd ioquake3
      make
    • If you've got the following error
      LD build/release-linux-x86_64/renderer_opengl1_x86_64.so
      /usr/bin/ld: cannot find -lGL
      collect2: ld returned 1 exit status
      just do
       rm -vf /usr/lib/libGL.so
      ln -s /usr/lib/libGL.so.275.28 /usr/lib/libGL.so
      make
      cd build/release-linux-x86_64/
      cp -rvf . /home/david/ioquake3/
      
  • [+] It is possible to restart domU without restarting dom 0 but Linux domU needs to be the first domU started - for HVM domU list - since dom0 booted

(*): This is the required version, may not work with version >= 280.XX

6. Quick instructions to install Xen with patches

  1. Extract your EEPROM from your NVIDIA graphic card

    Use nvtools or your favorite tool (Please Google for that).

    Optional way If you've got a USB Key (small size), you can use the following way. I made a ISO for nvflash 5.100 so you can boot from a USB Key.
    • Plug your USB Key
    • Download iso
      wget http://www.davidgis.fr/download/nvflash_5.100.1_usb.iso.tar.bz2
      tar xvzf nvflash_5.100.1_usb.iso.tar.bz2
    • Now copy iso to the USB Key

      Let's suppose that we have the USB key mounted on /dev/sdg1

      david@gemini:~$ mount |grep media
      /dev/sdf1 on /media/lmmmmnmkddc type vfat (rw,nosuid,nodev,uhelper=udisks,uid=1000,gid=1000,shortname=mixed,dmask=0077,utf8=1,flush)
      /dev/sdg1 on /media/LEXAR type vfat (rw,nosuid,nodev,uhelper=udisks,uid=1000,gid=1000,shortname=mixed,dmask=0077,utf8=1,flush)
      
      Umount
      umount /media/LEXAR
      The magic command
      cat nvflash_5.100.1_usb.iso > /dev/sdg
      Notice: There is no error. It is /dev/sdg and not /dev/sdg1 in the command above

      Unplug the USB Key. Boot on the USB Key using BIOS and use

      • nvflash.exe --list
        to find your card
      • nvflash.exe --save vgabios.rom
        to get your EEPROM
      • Use [CTRL]+[ALT]+[DEL] to quit.

    Once it is done,copy file vgabios.rom somewhere on your computer and rename it to vgabios-pt.bin

  2. Download Xen form staging
    rev=24407;hg clone -r $rev http://xenbits.xensource.com/staging/xen-unstable.hg/ xen-unstable.hg-rev-${rev}
    
  3. Make a init compil in tools
    cd xen-unstable.hg-rev-24407/
    cd tools/
    make 
    make clean
    cd ..
    
  4. Download patches and apply (See section 3 "Xen Patches")
    wget -q http://www.davidgis.fr/download/xen-4.2_rev24232_gfx-passthrough-patchs.tar.bz2
    tar xvjf xen-4.2_rev24232_gfx-passthrough-patchs.tar.bz2 
    for file in $(ls xen-4.2_rev24232_gfx-passthrough-patchs/*);do patch -p1 < $file;done
    
  5. Modify the values assigned to your graphic card to tools/firmware/hvmloader/acpi/dsdt.asl. So here is an example for my own video graphic card
    • In order to getting the required values, first of all find the identifier for your card
      root@mercury:~# lspci |grep VGA
      01:00.0 VGA compatible controller: nVidia Corporation Device 0e23 (rev a1)
      
    • Use dmesg to get the required range
      root@mercury:~# dmesg | grep 01:00.0 | grep BAR
      [    2.883158] pci 0000:01:00.0: BAR 0: reserving [mem 0xf8000000-0xf9ffffff flags 0x40200] (d=0, p=0)
      [    2.883161] pci 0000:01:00.0: BAR 1: reserving [mem 0xd0000000-0xd7ffffff flags 0x14220c] (d=0, p=0)
      [    2.883163] pci 0000:01:00.0: BAR 3: reserving [mem 0xd8000000-0xdbffffff flags 0x14220c] (d=0, p=0)
      [    2.883166] pci 0000:01:00.0: BAR 5: reserving [io  0xe000-0xe07f flags 0x40101] (d=0, p=0)
      
      The 3 ranges [0xf8000000-0xf9fffff], [0xd0000000-0xd7ffffff], [0xd8000000-0xdbfffff] need to be specify in dsdt.asl.patch.

      You need to use the formula

      Maximum - Minimum + 1 for the difference
    • Let' have a try for the first range [0xf8000000-0xf9fffff]
      Maximum = (0xf9fffff) / Minimum = 0xf8000000 => Difference = 4194303999 - 4160749568 + 1 = 33554432 => hexadecimal = 0x02000000
      so for the first range
      Maximum = 0xf9fffff
      Minimum = 0xf8000000
      Difference = 0x02000000
      
      Do the same for the two other ranges.
    • Then open tools/firmware/hvmloader/acpi/dsdt.asl. Using your favorite editor (nano, vim...) ,try to find the text reserve MMIO BARs of gfx and put the required values
      ...
                          /* reserve MMIO BARs of gfx for 1:1 mapping */
                          DWordMemory(
                              ResourceProducer, PosDecode, MinFixed, MaxFixed,
                              Cacheable, ReadWrite,
                              0x00000000,
                              0xF8000000,
                              0xF9FFFFFF,
                              0x00000000,
                              0x02000000)
      
                          DWordMemory(
                              ResourceProducer, PosDecode, MinFixed, MaxFixed,
                              NonCacheable, ReadWrite,
                              0x00000000,
                              0xD0000000,
                              0xD7FFFFFF,
                              0x00000000,
                              0x08000000)
      
                          DWordMemory(
                              ResourceProducer, PosDecode, MinFixed, MaxFixed,
                              Cacheable, ReadWrite,
                              0x00000000,
                              0xD8000000,
                              0xDBFFFFFF,
                              0x00000000,
                              0x4000000)
      ...
      
  6. Copy your vgabios-pt.bin (see point 1 - extracting EEPROM) in tools/firmware/vgabios/
  7. cp -vf /root/xen-unstable.hg-rev-24109/tools/firmware/vgabios/vgabios-pt.bin tools/firmware/vgabios/vgabios-pt.bin
  8. Compil and install xen
    make xen && make tools && make stubdom
    make install-xen && make install-tools PYTHON_PREFIX_ARG= && make install-stubdom && reboot
    

7. Notes for domU configuration

Here is the configuration of my domU
root@mercury:~# grep -vE '^(#|$)|#' /etc/xen/machines/mercury-xen03.cfg
builder='hvm'
memory = 3072
shadow_memory = 1024
name = "mercury-xen03"
vcpus='4'
vif = [ 'bridge=eth0,mac=00:14:3e:00:8f:c2' ]
disk = [
		'phy:/dev/xenvg/mercury-xen03-disk-6,xvda,w',		
		'phy:/dev/sr0,xvdc:cdrom,r',
         ]
boot="cd"
sdl=0
vnc=1
vnclisten="0.0.0.0"
vncconsole=1
vncpasswd=''
acpi=1
apic=1
xen_extended_power_mgmt=1
pae=1
arch='x86_64'
hpet = 1
hap = 1
viridian = 1
monitor=1
serial='pty'
keymap='fr'
soundhw = "all"
audio="on"
ne2000=0
on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'
xen_platform_pci=1 
gfx_passthru=1
pci  = [ '01:00.0','00:1a.0','00:1d.0' ]
pci_msitranslate = 1
pci_power_mgmt = 1
acpi_s3 = 1
acpi_s4 = 1
Now we will use PCI-STUB in order to being able to boot the domU. Please refer to Xen VT-d Wiki Section "Binding Devices to pci-stub"
  1. root@mercury:~# lspci | grep VGA
    01:00.0 VGA compatible controller: nVidia Corporation Device 0e23 (rev a1)
  2. root@mercury:~# lspci -n | grep "01:00.0"
    01:00.0 0300: 10de:0e23 (rev a1)
  3. Now we can boot the domU
    root@mercury:~# grep -vE '^(#|$)' start_windows.sh 
    echo "10de 0e23" > /sys/bus/pci/drivers/pci-stub/new_id
    echo "0000:01:00.0" > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
    echo "0000:01:00.0" > /sys/bus/pci/drivers/pci-stub/bind
    xl   create /etc/xen/machines/mercury-xen03.cfg