Two hugely significant patches were posted this week. The first was the Kernel Virtual Machine patch, for integrating Linux KVM support into upstream QEMU. The second was the addition of the Tiny Code Generator (TCG), which replaces dyngen and offers the potential to finally break QEMU's dependency on the gcc 3.x series once all targets are converted over.
Igor Kovalenko posted a patch to upgrade SCSI support for large disks.
From what I see there are more bits of scsi INQUIRY command which are mandatory, and this patch adds two mandatory and one optional scsi VPD page to that command reply. Also handled optional "command support data" request, with error reply. Please apply qemu-scsi-disk-inquiry-20080129-2.patch attached. With this patch I'm able to install openSUSE x86 on 10Gb scsi disk (emulated, image). Without this patch I see the following emulation error messages: scsi-disk: Orphaned request lsi_scsi: error: IO with unknown tag 65579 (tag number varies) and once lsi_scsi: error: Reselect with pending DMA Seems like Linux sym driver really does not like missing VPD replies, or there is some problem with lsi emulation in qemu.
Scott Pakin submitted a patch to make SLIRP 64 bit clean. (Whether this involved emulating 64 bit targets, working on 64 bit hosts, or both, was never quite explained during the thread.)
The attached patch corrects a bug in qemu/slirp/tcp_var.h that defines the seg_next field in struct tcpcb to be 32 bits wide regardless of 32/64-bitness. seg_next is assigned a pointer value in qemu/slirp/tcp_subr.c, then cast back to a pointer in qemu/slirp/tcp_input.c and dereferenced. That produces a SIGSEGV on my system. For more information, see the thread "[ 1881532 ] Network access seg faults KVM on large-memory machine" on the KVM Bugs page on SourceForge (http://tinyurl.com/2fxfbx).
In the ensuing discussion, Blue Swirl pointed out that fixing it wasn't that easy, and Scott submitted updated patches.
Alexander Graf offered a patch to implement multiboot support for qemu's -kernel option, but since it was only needed to boot Xen and Hurd kernels it attracted little interest.
rainer haage announced another qemu gui/launcher/job scheduler:
You might be interested to add a link to the qemulator GUI on the qemu homepage. It's now an official part of the Ubuntu universe packages and it's also in debian unstable and testing.
Alexander Graf submitted a patch series to improve MacOS X host support:
supporting Mac OS X on Intel has been a long standing issue. The Q project has done a fabulous job maintaining patches that make things work for i386 Mac OS X, but do invasive changes to qemu internals. This set of patches attempts to make Mac OS X host support as compatible as possible to the existing Linux host and PowerPC Mac OS X support. Of course this means, that as long as Linux gcc4 support is broken, it won't work on Mac OS X either. Nevertheless all changes necessary to support the binary format (Mach-O) and several other minor issues that are Mac specific can be easily added to the existing code base without harming other platforms. This way people who want to run qemu on Mac OS X only have to maintain gcc4 patches and no Apple specific ones.
The patch set posted earlier this month was for running MacOS X targets under QEMU. This series is for running QEMU on MacOS X hosts. It teaches qemu application emulation to understand Mach-O (the file format MacOS X uses instead of ELF), upgrades ./configure to detect MacOS X hosts, and includes MacOS X related build fixes for i386 and x86_64 hosts.
What it doesn't fix is building under gcc 4.x, which is the default compiler of MacOS X. It doesn't make that problem any worse, but it doesn't solve it either.
The specific patches were:
[PATCH 1/4] dyngen Mach-O support for i386 and x86_64
This patch extends the existing support for the Mach-O binary format in dyngen from PowerPC to PowerPC, i386 and x86_64.
[PATCH 2/4] Mac OS X x86_64 support
This patch implements Mac OS X specific parts that are necessary to get x86_64 versions of qemu working. It does: - add x86_64 detection to configure - add -fomit-frame-pointer if available - set the pagezero size to <4GB, so 32bit lea still works - fix redeclarations of int64_t and uint64_t - add direct jump support for x86_64
[PATCH 3/4] Mac OS X i386 support
This patch implements Mac OS X specific parts that are necessary to get i386 versions of qemu working. It does: - make the GOTO_TB macro more portable
[PATCH 4/4] Mac OS X generic support
This patch implements Mac OS X specific parts that are necessary to get i386 and x86_64 versions of qemu working. Since both hosts need this patch, I seperated it from the architecture specific ones. It does: - not force always_inline - only define [u]intXX types if they are not already defined - nullify spin-functions using #define instead of inline function
Following up on last week's discussion, Anthony Liguori submitted the main KVM support patch:
KVM is a Linux interface for providing userspace interfaces for accelerated virtualization. It has been included since 2.6.20 and supports Intel VT and AMD-V. Ports are under way for ia64, embedded PowerPC, and s390. This set of patches provide basic support for KVM in QEMU. It does not include all of the changes in the KVM QEMU branch (such as virtio, live migration, extboot, etc). However, if we can get these first portions merged, I will follow up with the remainder of the changes and I believe we can be fully merged in the very near future. The first 5 patches of this series are not KVM specific but are critical fixes for KVM to be functional. The 6th patch provides KVM support. The goal in providing KVM support is to make sure that when KVM support is not compiled in, the code paths aren't changed at all. I hope this makes it very easy to merge. KVM moves very quickly, so I'd appreciate if these patches can be reviewed as soon as possible as it's going to be tough to keep them in sync with the main KVM tree while they're out of tree. To enable KVM support, you have to have to libkvm installed. You should also explicitly specify the location of your kernel tree (with KVM headers) with the --kernel-path option. We will improve libkvm such that this isn't required in future versions. KVM also has an enhanced Bochs BIOS. I've tested these patches with out it and it's not strictly necessary for basic functionality. I would recommend pulling in a copy of it though as it has useful fixes even in the absence of KVM. A very large number of people have contributed to these patches with Avi Kivity being the main developer of this support. For a full listing of contributers, please consult the KVM ChangeLog[1]. [1] http://kvm.qumranet.com/kvmwiki/ChangeLog
Much of what the KVM project offered here are general upgrades to qemu, and were closely reviewed. The first and last patches in the series even attracted Fabrice's personal attention. The longest thread was for the first patch, which removes the 2 gigabyte limit on the amount of memory a 64-bit QEMU host can provide to a guest system (by replacing various signed 32 bit ints with a 64 bit type). The QEMU developers reviewed the patch closely, pointing out numerous possible cleanups. Xen developer Ian Jackson asked if the patch could be extended to support Xen, but Anthony replied that Xen's design made that problematic:
Support for the map cache in the Xen tree is a rather big change that I'm not going to attempt to support it in this patch series. I'd rather preserve the phys_ram_base + PA assumption because it allows us to be able to do support > 1 page DMA operations for our virtual IO drivers. If you break the assumption that physically contiguous memory in the guest is virtual contiguous memory in the host, things get pretty ugly.
Paul brook left open the possibility of someday exending the QEMU infrastructure to support non-contiguous host memory, through virtual IOMMUs:
That's why you should be using the cpu_physical_memory_rw routines :-) Anything that assume large linear accesses (Currently only some of the embedded LCD controllers) is going to break as soon as you start introducing IOMMUs. There have been several threads on this list about having a sane DMA infrastructure.
But the Xen project's interest in providing more than 2 gigabytes of memory to 64 bit guests on 32 bit hosts using the x86 Page Addressing Extensions was not well received by the QEMU developers. They seem to prefer to keep the code simple, since 64 bit host hardware gets cheaper every day.
[Editorial comment: Also, the Xen project is clearly receeding in importance within the open source community. KQEMU works on both Linux and Windows, and KVM takes advantage of new hardware features to do provide the same acceleration as KQEMU more simply and efficiently on hardware that supports it. Xen is much larger and more complicated than either, without a significant increase in functionality. It really hasn't got much reason to exist anymore, and the project doesn't seem to produce much interesting or high quality code of general use to the rest of QEMU the way KVM does.]
A second and third round of patches were submitted, with only minor technical discussion:
Since v2, I've incorporated all the feedback received for v1. I was able to drop the option ROM refactoring too but introduced a new patch to fix an obvious bug with the VMware vga initialization pointed out by Paul Brook. Since v3, I've incorporated a few changes suggested by Paul Brook and rebased the patches to apply against the latest CVS.
Gervase Lam finished out the month with a question:
Awhile back, somebody did a patch to allow guest accelerated Windows OpenGL graphics. From what I can tell, he basically wrote a Windows OpenGL API that 'calls' interrupts, which QEMU traps. QEMU then interprets the interrupts and calls the OpenGL API of the Linux host. <http://qemu-forum.ipi.fi/viewtopic.php?f=2&t=2984&st=0&sk=t&sd=a&sid=1c320bec1bea28ff21ed2204b2df4c60> In the thread is also the interesting suggestion of using the dlxgl wrapper API to convert Direct3D to OpenGL in Windows. This theoretically means that you could run Direct3D apps in QEMU. I think it would be worthwhile getting Direct3D and OpenGL to work on QEMU. This is especially when Win95/98 cannot run on the more up-to- date PC chipsets (due to manufacturers not being will to provide drivers or whatever). I know of a least two applications that can run on Win95/98 but not XP. Fabrice made a comment on this patch: <http://lists.gnu.org/archive/html/qemu-devel/2006-11/msg00149.html> From my minimal understanding of what he is saying, it seems he would prefer there to be a BOCHS graphics adapter, which would then pass the OpenGL commands from Windows to Linux OpenGL. Also, Direct3D commands would be passed from Windows to Linux Wine. I wouldn't mind having a go at this as I would like to run 3D accelerated Windows apps on Linux. However, due to a couple of other projects I want to complete first (including a PC upgrade), I don't think I'll have time to do this until may be later this year. Also, I will need to do some learning. So don't your breath. Anybody want to volunteer to do this? What are people's current feelings about this? I've been wanting to run 3D accelerated Win98/95 in Linux (on what will be an up-to-date chipset soon...). But I kept on finding out about PCI pass through on Xen to allow Direct3D to do its work, and the open "source" specifications of ATI cards (though I read that nVidia are allegedly deliberating about doing the same thing sometime this year).
Nobody else volunteered to do the work this week. A couple people expressed interest in the simpler project of a 2D graphics driver supporting arbitrary screen resolutions, but Paul Brook pointed out that QEMU's VMware adapter already does this.
Fabrice Bellard dropped a bombshell, introducing a new Tiny Code Generator to eventually replace dyngen, capable of breaking QEMU's dependency on gcc 3.X while simultaneously speeding up QEMU:
I added a new code generator (TCG) in QEMU. Read the file qemu/tcg/README to have technical information. A new code generator was needed in order to avoid problems with the various GCC versions and to get better performance. I made minimal modifications in each target so that they can still work by using TCG and legacy "dyngen" micro operations. More work will be needed to convert each target to TCG, but it can be done progressively. Only the x86 and x86_64 targets have been significantly modified to use TCG. TCG currently only supports x86 and x86_64 hosts. The support of other hosts is broken in QEMU. I don't consider it is a priority to support other hosts although TCG is generic enough to allow it. The performance should not change much at this point. On x86_64 hosts, the x86 and x86_64 targets are faster because of some assembly language optimizations. On x86 hosts, the x86_64 target is faster because TCG can do some optimizations related to the conversion from 64 to 32 bits. No performance change should be noticed on non x86 targets because they are not converted yet.
The design of dyngen is described in detail in this usenix paper, and the problems with dyngen were covered in an earlier thread. This is the new code generator Fabrice mentioned in that thread.
This is an earthquake in qemu development. The core of qemu is a translation layer that converts blocks of target code into blocks of code to run on the host. For years, that translation layer was dyngen, based on memorizing special snippets of code produced by gcc back when qemu is compiled, and then playing those snippets back in various orders to produce blocks of host code. Dyngen is what tied qemu to a specific version of gcc, both convincing gcc to produce properly transplantable snippets and knowing how unwrap the object files produced by that version of gcc to extract the snippets.
Now QEMU can act as its own host compiler. By redesigning the core of qemu to produce its own executable code, Fabrice is breaking the dependence on GCC 3.x, although it won't be gone until all the targets are fully converted and dyngen can be removed. (In the meantime, the translator uses TCG for some operations and dyngen for others.) Also, to run on hosts other than x86 and x86-64, TCG must now be taught to produce code for each new host, just like any other compiler. Bu unlike dyngen's blind playback of prebuilt code, TCG can also be taught to optimize the code it produces.
Samuel Thibault asked a question that
Samuel Thibault wrote: > We were wondering whether it could be useful to have e.g. a git > repository holding the patches from Xen and KVM: while they are being > merged upstream, they could at least be shared by both projects... > KVM currently maintains a git tree of it's QEMU changes so unfortunately, things aren't nicely separated into patches. As you may have noticed, I've posted the core KVM support patches and I'm hoping we can merge them pretty soon. I am then going to follow up and try to get the remainder of the patches merged. In the short term, we will probably continue maintaining a git tree against QEMU as a staging repository but I hope that we can regularly sync up (just like we do with Linux). In the long term, I'd like KVM not to maintain a QEMU tree at all. In a broader sense, I know there are concerns about patches getting missed on qemu-devel. I don't think the solution is to maintain a separate patch queue, but to get the patches merged more quickly in the first place.
[From this editor's viewpoint, the subtext looks like Xen asking to leverage KVM's generally higher patch quality to get its own patches in, and KVM turning them down gently.]
After a little discussion on the list, I made this patch for cocoa.m, to replace CoreGraphic by OpenGL. It's way faster than CG, but it requires a Mac with OpenGL capable Graphics Card and at least 8mb of VRAM. I think starting with G4 and Highend G3, this requirements are met.
To clarify, this isn't to emulate a 3D card within qemu, but instead using opengl on MacOS X hosts as a faster alternative to SDL for displaying 2D video.
A longish thread ensued, much of it wondering whether this should be genericized to provide an opengl alternative to SDL on other hosts, such as Linux. The counter-argument is that many versions of SDL can already use OpenGL to render their output, and perhaps performance problems on specific targets (such as 64-bit MacOS X, where benchmarks show opengl clearly winning over SDL) should be fixed in SDL, not worked around in QEMU.
Eddie Kohler submitted a patch:
The following patch, against cvs, supports read-only snapshots on read-only qcow2 image files. Snapshots can be loaded, but not saved. This is really useful in my context, which is operating systems projects; a read-only memory snapshot allows students to share a snapshot that skips the boot process, allowing them to run their code right away.
Asheesh Laroia complained that his local LAN used the same address range as qemu's default virtual network, 10.0.2.x. He suggested hardwiring in 10.0.3.x instead, because then it's not his problem.
Xen developer Ian Jackson suggested quite possibly the only way of making it worse: choosing an address randomly, which could make the problem hit more people in an intermittent and unpredictable manner. (Luckily, this suggestion was not well received. He later clarified he hadn't meant it.)
Much bikeshedding ensued. The suggestion was made to make it a command line option, but nobody volunteered to do the work, and the thread petered out.
Big news this week is svn 3943 and 3944: the Tiny Code Generator (TCG). It's a gigantic patch (7000 line, although much of it's ripping out old dyngen code) fundamentally changing the way QEMU works, in a good way. (This is the key to breaking the gcc 3.x dependency.)
Other interesting commits add a new gigabit card (Intel PRO/1000), and handle more SCSI commands.
Fabrice also yanked last week's new -translation option.
This week's commits: