Here you will find required patches and notes about NVIDIA and VGA Pass Through.
Please be informed that
ABOUT PATCHES
- I am not the author who wrote the original patches. I've just tried to maintain the patches I found from Xen mailing-list. Since Summer 2011, I've begun to test Xen VGA PassThrough
- I try to test patches once a week (week-end)
- LAST UPDATE: 03/25/2012 / LAST XEN REVISION: 25099
BEFORE YOUR TEST
- You have to read the required documentation taken from Xen Wiki: Xen VGA PassThrough
- You must have the specific hardware before doing any test: NVIDIA graphic card, specific processor Intel (Vt-d) or AMD (IOMMU), a specific motherboard VT-d capable
|
1. Examples: Video for playing Crysis 2 single player and screenshots for Windows XP
- Here is a (poor) video for Crysis 2 http://www.youtube.com/watch?v=3SaYO0ERW44 I did using my old GT 440 (before I bought my GTX 460).
- Here are a few screenshots using CPUZ64
2. Tested Hardware
| Item |
Reference |
Comment |
| Processor |
Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz |
| Motherboard |
Manufacturer: MSI / Product Name: H61MU-E35 (MS-7680)
|
Bios Version V10.11 |
| Graphic Card (*) |
GTX 460 |
GTX 460 SE EVGA 1024 MB 01G-P3-1366-ET |
| Graphic Card |
GTX 560 TI |
GTX 560 TI Point Of View 2048 MB GDDR5 |
(*) : It was my previous card ( before 01/09/2012)
3. Xen Patches
All patches are applied from xen-unstable staging (
http://xenbits.xensource.com/staging/xen-unstable.hg/)
4. Linux Kernel .config
My tests are based on a simple Debian 6.0.2 64 bits.
Notice: since 3.1.0, I used to compile Xen including all "Xen capabilitites" in the kernel (=Y) not as modules (=M).
root@mercury:~# grep -i xen /boot/config-3.1.6
CONFIG_XEN=y
CONFIG_XEN_DOM0=y
CONFIG_XEN_PRIVILEGED_GUEST=y
CONFIG_XEN_PVHVM=y
CONFIG_XEN_MAX_DOMAIN_MEMORY=128
CONFIG_XEN_SAVE_RESTORE=y
# CONFIG_XEN_DEBUG_FS is not set
# CONFIG_XEN_DEBUG is not set
CONFIG_PCI_XEN=y
CONFIG_XEN_PCIDEV_FRONTEND=y
CONFIG_XEN_PCIDEV_FE_DEBUG=y
CONFIG_XEN_BLKDEV_FRONTEND=y
CONFIG_XEN_BLKDEV_BACKEND=y
CONFIG_NETXEN_NIC=y
CONFIG_XEN_NETDEV_FRONTEND=y
CONFIG_XEN_NETDEV_BACKEND=y
CONFIG_INPUT_XEN_KBDDEV_FRONTEND=y
CONFIG_HVC_XEN=y
CONFIG_XEN_WDT=y
CONFIG_XEN_FBDEV_FRONTEND=y
# Xen driver support
CONFIG_XEN_BALLOON=y
CONFIG_XEN_SELFBALLOONING=y
CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_XEN_DEV_EVTCHN=y
CONFIG_XEN_BACKEND=y
CONFIG_XENFS=y
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
CONFIG_XEN_XENBUS_FRONTEND=y
CONFIG_XEN_GNTDEV=y
CONFIG_XEN_GRANT_DEV_ALLOC=y
CONFIG_XEN_PLATFORM_PCI=y
CONFIG_SWIOTLB_XEN=y
CONFIG_XEN_TMEM=y
CONFIG_XEN_PCIDEV_BACKEND=y
Here are my configs
5. Tested domU
| domU |
Graphic card |
Nvidia driver (*) |
Comments |
| Windows XP 32 / 64 Bits |
- GTX 460 SE EVGA 1024 MB
- GTX 560 Ti Point Of View 2048 MB GDDR5
|
|
- [+] Crysis 2 works very well, single player mode, resolution 1920x1080
- [-] It it not possible to restart domU without restarting dom0
|
| Ubuntu Lucid 10.04.4 64 Bits |
- GTX 460 SE EVGA 1024 MB
- GTX 560 Ti Point Of View 2048 MB GDDR5
|
- 275.28 aka NVIDIA-Linux-x86_64-275.28.run
- 275.43 aka NVIDIA-Linux-x86_64-275.43.run
|
- [+] Quake 3 works very well using ioquake3 but you need to build ioquake3 from sources
-
svn co svn://svn.icculus.org/quake3/trunk ioquake3
cd ioquake3
make
- If you've got the following error
LD build/release-linux-x86_64/renderer_opengl1_x86_64.so
/usr/bin/ld: cannot find -lGL
collect2: ld returned 1 exit status
just do
rm -vf /usr/lib/libGL.so
ln -s /usr/lib/libGL.so.275.28 /usr/lib/libGL.so
make
cd build/release-linux-x86_64/
cp -rvf . /home/david/ioquake3/
- [+] It is possible to restart domU without restarting dom 0 but Linux domU needs to be the first domU started - for HVM domU list - since dom0 booted
|
(*): This is the required version, may not work with version >= 280.XX
6. Quick instructions to install Xen with patches
- Extract your EEPROM from your NVIDIA graphic card
Use nvtools or your favorite tool (Please Google for that).
Optional way
If you've got a USB Key (small size), you can use the following way. I made a ISO for nvflash 5.100 so you can boot from a USB Key.
- Plug your USB Key
- Download iso
wget http://www.davidgis.fr/download/nvflash_5.100.1_usb.iso.tar.bz2
tar xvzf nvflash_5.100.1_usb.iso.tar.bz2
- Now copy iso to the USB Key
Let's suppose that we have the USB key mounted on /dev/sdg1
david@gemini:~$ mount |grep media
/dev/sdf1 on /media/lmmmmnmkddc type vfat (rw,nosuid,nodev,uhelper=udisks,uid=1000,gid=1000,shortname=mixed,dmask=0077,utf8=1,flush)
/dev/sdg1 on /media/LEXAR type vfat (rw,nosuid,nodev,uhelper=udisks,uid=1000,gid=1000,shortname=mixed,dmask=0077,utf8=1,flush)
Umount
umount /media/LEXAR
The magic command
cat nvflash_5.100.1_usb.iso > /dev/sdg
Notice: There is no error. It is /dev/sdg and not /dev/sdg1 in the command above
Unplug the USB Key. Boot on the USB Key using BIOS and use
Once it is done,copy file vgabios.rom somewhere on your computer and rename it to vgabios-pt.bin
- Download Xen form staging
rev=24407;hg clone -r $rev http://xenbits.xensource.com/staging/xen-unstable.hg/ xen-unstable.hg-rev-${rev}
- Make a init compil in tools
cd xen-unstable.hg-rev-24407/
cd tools/
make
make clean
cd ..
- Download patches and apply (See section 3 "Xen Patches")
wget -q http://www.davidgis.fr/download/xen-4.2_rev24232_gfx-passthrough-patchs.tar.bz2
tar xvjf xen-4.2_rev24232_gfx-passthrough-patchs.tar.bz2
for file in $(ls xen-4.2_rev24232_gfx-passthrough-patchs/*);do patch -p1 < $file;done
- Modify the values assigned to your graphic card to
tools/firmware/hvmloader/acpi/dsdt.asl. So here is an example for my own video graphic card
-
In order to getting the required values, first of all find the identifier for your card
root@mercury:~# lspci |grep VGA
01:00.0 VGA compatible controller: nVidia Corporation Device 0e23 (rev a1)
- Use dmesg to get the required range
root@mercury:~# dmesg | grep 01:00.0 | grep BAR
[ 2.883158] pci 0000:01:00.0: BAR 0: reserving [mem 0xf8000000-0xf9ffffff flags 0x40200] (d=0, p=0)
[ 2.883161] pci 0000:01:00.0: BAR 1: reserving [mem 0xd0000000-0xd7ffffff flags 0x14220c] (d=0, p=0)
[ 2.883163] pci 0000:01:00.0: BAR 3: reserving [mem 0xd8000000-0xdbffffff flags 0x14220c] (d=0, p=0)
[ 2.883166] pci 0000:01:00.0: BAR 5: reserving [io 0xe000-0xe07f flags 0x40101] (d=0, p=0)
The 3 ranges [0xf8000000-0xf9fffff], [0xd0000000-0xd7ffffff], [0xd8000000-0xdbfffff] need to be specify in dsdt.asl.patch.
You need to use the formula
Maximum - Minimum + 1 for the difference
- Let' have a try for the first range [0xf8000000-0xf9fffff]
Maximum = (0xf9fffff) / Minimum = 0xf8000000 => Difference = 4194303999 - 4160749568 + 1 = 33554432 => hexadecimal = 0x02000000
so for the first range
Maximum = 0xf9fffff
Minimum = 0xf8000000
Difference = 0x02000000
Do the same for the two other ranges.
- Then open
tools/firmware/hvmloader/acpi/dsdt.asl. Using your favorite editor (nano, vim...) ,try to find the text reserve MMIO BARs of gfx and put the required values
...
/* reserve MMIO BARs of gfx for 1:1 mapping */
DWordMemory(
ResourceProducer, PosDecode, MinFixed, MaxFixed,
Cacheable, ReadWrite,
0x00000000,
0xF8000000,
0xF9FFFFFF,
0x00000000,
0x02000000)
DWordMemory(
ResourceProducer, PosDecode, MinFixed, MaxFixed,
NonCacheable, ReadWrite,
0x00000000,
0xD0000000,
0xD7FFFFFF,
0x00000000,
0x08000000)
DWordMemory(
ResourceProducer, PosDecode, MinFixed, MaxFixed,
Cacheable, ReadWrite,
0x00000000,
0xD8000000,
0xDBFFFFFF,
0x00000000,
0x4000000)
...
- Copy your vgabios-pt.bin (see point 1 - extracting EEPROM) in
tools/firmware/vgabios/
cp -vf /root/xen-unstable.hg-rev-24109/tools/firmware/vgabios/vgabios-pt.bin tools/firmware/vgabios/vgabios-pt.bin
- Compil and install xen
make xen && make tools && make stubdom
make install-xen && make install-tools PYTHON_PREFIX_ARG= && make install-stubdom && reboot
7. Notes for domU configuration
Here is the configuration of my domU
root@mercury:~# grep -vE '^(#|$)|#' /etc/xen/machines/mercury-xen03.cfg
builder='hvm'
memory = 3072
shadow_memory = 1024
name = "mercury-xen03"
vcpus='4'
vif = [ 'bridge=eth0,mac=00:14:3e:00:8f:c2' ]
disk = [
'phy:/dev/xenvg/mercury-xen03-disk-6,xvda,w',
'phy:/dev/sr0,xvdc:cdrom,r',
]
boot="cd"
sdl=0
vnc=1
vnclisten="0.0.0.0"
vncconsole=1
vncpasswd=''
acpi=1
apic=1
xen_extended_power_mgmt=1
pae=1
arch='x86_64'
hpet = 1
hap = 1
viridian = 1
monitor=1
serial='pty'
keymap='fr'
soundhw = "all"
audio="on"
ne2000=0
on_poweroff = 'destroy'
on_reboot = 'restart'
on_crash = 'restart'
xen_platform_pci=1
gfx_passthru=1
pci = [ '01:00.0','00:1a.0','00:1d.0' ]
pci_msitranslate = 1
pci_power_mgmt = 1
acpi_s3 = 1
acpi_s4 = 1
Now we will use PCI-STUB in order to being able to boot the domU. Please refer to
Xen VT-d Wiki Section "Binding Devices to pci-stub"
-
root@mercury:~# lspci | grep VGA
01:00.0 VGA compatible controller: nVidia Corporation Device 0e23 (rev a1)
-
root@mercury:~# lspci -n | grep "01:00.0"
01:00.0 0300: 10de:0e23 (rev a1)
-
Now we can boot the domU
root@mercury:~# grep -vE '^(#|$)' start_windows.sh
echo "10de 0e23" > /sys/bus/pci/drivers/pci-stub/new_id
echo "0000:01:00.0" > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
echo "0000:01:00.0" > /sys/bus/pci/drivers/pci-stub/bind
xl create /etc/xen/machines/mercury-xen03.cfg
If on all nodes (both master and slaves), you did a fresh backup of the node using 