QEMU weekly news: Jan 29, 2008 - Feb 4, 2008

1 Mailing list

2 Source control


1 Mailing list

Two hugely significant patches were posted this week. The first was the Kernel Virtual Machine patch, for integrating Linux KVM support into upstream QEMU. The second was the addition of the Tiny Code Generator (TCG), which replaces dyngen and offers the potential to finally break QEMU's dependency on the gcc 3.x series once all targets are converted over.

1.1 Jan 29, 2008 - [PATCH] implement more INQUIRY commands

Igor Kovalenko posted a patch to upgrade SCSI support for large disks.

From what I see there are more bits of scsi INQUIRY command which are mandatory,
and this patch adds two mandatory and one optional scsi VPD page to that command
reply. Also handled optional "command support data" request, with error reply.

Please apply qemu-scsi-disk-inquiry-20080129-2.patch attached.

With this patch I'm able to install openSUSE x86 on 10Gb scsi disk
(emulated, image).
Without this patch I see the following emulation error messages:
scsi-disk: Orphaned request
lsi_scsi: error: IO with unknown tag 65579 (tag number varies)
and once
lsi_scsi: error: Reselect with pending DMA
Seems like Linux sym driver really does not like missing VPD replies,
or there is some problem with lsi emulation in qemu.

1.2 Jan 29, 2008 - [PATCH] Making SLIRP code more 64-bit clean

Scott Pakin submitted a patch to make SLIRP 64 bit clean. (Whether this involved emulating 64 bit targets, working on 64 bit hosts, or both, was never quite explained during the thread.)

The attached patch corrects a bug in qemu/slirp/tcp_var.h that defines
the seg_next field in struct tcpcb to be 32 bits wide regardless of
32/64-bitness.  seg_next is assigned a pointer value in
qemu/slirp/tcp_subr.c, then cast back to a pointer in qemu/slirp/tcp_input.c
and dereferenced.  That produces a SIGSEGV on my system.

For more information, see the thread "[ 1881532 ] Network access seg faults
KVM on large-memory machine" on the KVM Bugs page on SourceForge
(http://tinyurl.com/2fxfbx).

In the ensuing discussion, Blue Swirl pointed out that fixing it wasn't that easy, and Scott submitted updated patches.

1.3 Jan 30, 2008 - [PATCH] x86 Multiboot support (extended)

Alexander Graf offered a patch to implement multiboot support for qemu's -kernel option, but since it was only needed to boot Xen and Hurd kernels it attracted little interest.

1.4 Jan 30, 2008 - Qemulator

rainer haage announced another qemu gui/launcher/job scheduler:

You might be interested to add a link to the qemulator GUI on the qemu 
homepage. It's now an official part of the Ubuntu universe packages and it's 
also in debian unstable and testing.

1.5 Jan 31, 2008 - [PATCH 0/4] Intel Mac OS X Host support

Alexander Graf submitted a patch series to improve MacOS X host support:

supporting Mac OS X on Intel has been a long standing issue. The Q  
project has done a fabulous job maintaining patches that make things  
work for i386 Mac OS X, but do invasive changes to qemu internals.

This set of patches attempts to make Mac OS X host support as  
compatible as possible to the existing Linux host and PowerPC Mac OS X  
support. Of course this means, that as long as Linux gcc4 support is  
broken, it won't work on Mac OS X either. Nevertheless all changes  
necessary to support the binary format (Mach-O) and several other  
minor issues that are Mac specific can be easily added to the existing  
code base without harming other platforms.

This way people who want to run qemu on Mac OS X only have to maintain  
gcc4 patches and no Apple specific ones.

The patch set posted earlier this month was for running MacOS X targets under QEMU. This series is for running QEMU on MacOS X hosts. It teaches qemu application emulation to understand Mach-O (the file format MacOS X uses instead of ELF), upgrades ./configure to detect MacOS X hosts, and includes MacOS X related build fixes for i386 and x86_64 hosts.

What it doesn't fix is building under gcc 4.x, which is the default compiler of MacOS X. It doesn't make that problem any worse, but it doesn't solve it either.

The specific patches were:

  • [PATCH 1/4] dyngen Mach-O support for i386 and x86_64

    This patch extends the existing support for the Mach-O binary format  
    in dyngen from PowerPC to PowerPC, i386 and x86_64.
    
  • [PATCH 2/4] Mac OS X x86_64 support

    This patch implements Mac OS X specific parts that are necessary to  
    get x86_64 versions of qemu working.
    
    It does:
    
    - add x86_64 detection to configure
    - add -fomit-frame-pointer if available
    - set the pagezero size to <4GB, so 32bit lea still works
    - fix redeclarations of int64_t and uint64_t
    - add direct jump support for x86_64
    
  • [PATCH 3/4] Mac OS X i386 support

    This patch implements Mac OS X specific parts that are necessary to  
    get i386 versions of qemu working.
    
    It does:
    
    - make the GOTO_TB macro more portable
    
  • [PATCH 4/4] Mac OS X generic support

    This patch implements Mac OS X specific parts that are necessary to  
    get i386 and x86_64 versions of qemu working. Since both hosts need  
    this patch, I seperated it from the architecture specific ones.
    
    It does:
    
    - not force always_inline
    - only define [u]intXX types if they are not already defined
    - nullify spin-functions using #define instead of inline function
    

1.6 Jan 31, 2008 - Support for the Kernel Virtual Machine interface

Following up on last week's discussion, Anthony Liguori submitted the main KVM support patch:

KVM is a Linux interface for providing userspace interfaces for accelerated
virtualization.  It has been included since 2.6.20 and supports Intel VT and
AMD-V.  Ports are under way for ia64, embedded PowerPC, and s390.

This set of patches provide basic support for KVM in QEMU.  It does not include
all of the changes in the KVM QEMU branch (such as virtio, live migration,
extboot, etc).  However, if we can get these first portions merged, I will
follow up with the remainder of the changes and I believe we can be fully
merged in the very near future.

The first 5 patches of this series are not KVM specific but are critical fixes
for KVM to be functional.  The 6th patch provides KVM support.  The goal in
providing KVM support is to make sure that when KVM support is not compiled in,
the code paths aren't changed at all.  I hope this makes it very easy to merge.

KVM moves very quickly, so I'd appreciate if these patches can be reviewed as
soon as possible as it's going to be tough to keep them in sync with the main
KVM tree while they're out of tree.

To enable KVM support, you have to have to libkvm installed.  You should also
explicitly specify the location of your kernel tree (with KVM headers) with the
--kernel-path option.  We will improve libkvm such that this isn't required in
future versions.

KVM also has an enhanced Bochs BIOS.  I've tested these patches with out it and
it's not strictly necessary for basic functionality.  I would recommend pulling
in a copy of it though as it has useful fixes even in the absence of KVM.

A very large number of people have contributed to these patches with Avi Kivity
being the main developer of this support.  For a full listing of contributers,
please consult the KVM ChangeLog[1].

[1] http://kvm.qumranet.com/kvmwiki/ChangeLog

Much of what the KVM project offered here are general upgrades to qemu, and were closely reviewed. The first and last patches in the series even attracted Fabrice's personal attention. The longest thread was for the first patch, which removes the 2 gigabyte limit on the amount of memory a 64-bit QEMU host can provide to a guest system (by replacing various signed 32 bit ints with a 64 bit type). The QEMU developers reviewed the patch closely, pointing out numerous possible cleanups. Xen developer Ian Jackson asked if the patch could be extended to support Xen, but Anthony replied that Xen's design made that problematic:

Support for the map cache in the Xen tree is a rather big change that 
I'm not going to attempt to support it in this patch series.

I'd rather preserve the phys_ram_base + PA assumption because it allows 
us to be able to do support > 1 page DMA operations for our virtual IO 
drivers.  If you break the assumption that physically contiguous memory 
in the guest is virtual contiguous memory in the host, things get pretty 
ugly.

Paul brook left open the possibility of someday exending the QEMU infrastructure to support non-contiguous host memory, through virtual IOMMUs:

That's why you should be using the cpu_physical_memory_rw routines :-)

Anything that assume large linear accesses (Currently only some of the 
embedded LCD controllers) is going to break as soon as you start introducing 
IOMMUs. There have been several threads on this list about having a sane DMA 
infrastructure.

But the Xen project's interest in providing more than 2 gigabytes of memory to 64 bit guests on 32 bit hosts using the x86 Page Addressing Extensions was not well received by the QEMU developers. They seem to prefer to keep the code simple, since 64 bit host hardware gets cheaper every day.

[Editorial comment: Also, the Xen project is clearly receeding in importance within the open source community. KQEMU works on both Linux and Windows, and KVM takes advantage of new hardware features to do provide the same acceleration as KQEMU more simply and efficiently on hardware that supports it. Xen is much larger and more complicated than either, without a significant increase in functionality. It really hasn't got much reason to exist anymore, and the project doesn't seem to produce much interesting or high quality code of general use to the rest of QEMU the way KVM does.]

A second and third round of patches were submitted, with only minor technical discussion:

Since v2, I've incorporated all the feedback received for v1.  I was able to
drop the option ROM refactoring too but introduced a new patch to fix an
obvious bug with the VMware vga initialization pointed out by Paul Brook.

Since v3, I've incorporated a few changes suggested by Paul Brook and rebased
the patches to apply against the latest CVS.

1.7 Jan 31, 2008 - 3D acceleration

Gervase Lam finished out the month with a question:

Awhile back, somebody did a patch to allow guest accelerated Windows
OpenGL graphics.  From what I can tell, he basically wrote a Windows
OpenGL API that 'calls' interrupts, which QEMU traps.  QEMU then
interprets the interrupts and calls the OpenGL API of the Linux host.

<http://qemu-forum.ipi.fi/viewtopic.php?f=2&t=2984&st=0&sk=t&sd=a&sid=1c320bec1bea28ff21ed2204b2df4c60>

In the thread is also the interesting suggestion of using the dlxgl
wrapper API to convert Direct3D to OpenGL in Windows.  This
theoretically means that you could run Direct3D apps in QEMU.

I think it would be worthwhile getting Direct3D and OpenGL to work on
QEMU.  This is especially when Win95/98 cannot run on the more up-to-
date PC chipsets (due to manufacturers not being will to provide drivers
or whatever).  I know of a least two applications that can run on
Win95/98 but not XP.

Fabrice made a comment on this patch:

<http://lists.gnu.org/archive/html/qemu-devel/2006-11/msg00149.html>

From my minimal understanding of what he is saying, it seems he would
prefer there to be a BOCHS graphics adapter, which would then pass the
OpenGL commands from Windows to Linux OpenGL.  Also, Direct3D commands
would be passed from Windows to Linux Wine.

I wouldn't mind having a go at this as I would like to run 3D
accelerated Windows apps on Linux.  However, due to a couple of other
projects I want to complete first (including a PC upgrade), I don't
think I'll have time to do this until may be later this year.  Also, I
will need to do some learning.  So don't your breath.  Anybody want to
volunteer to do this?

What are people's current feelings about this?  I've been wanting to run
3D accelerated Win98/95 in Linux (on what will be an up-to-date chipset
soon...).  But I kept on finding out about PCI  pass through on Xen to
allow Direct3D to do its work, and the open "source" specifications of
ATI cards (though I read that nVidia are allegedly deliberating about
doing the same thing sometime this year).

Nobody else volunteered to do the work this week. A couple people expressed interest in the simpler project of a 2D graphics driver supporting arbitrary screen resolutions, but Paul Brook pointed out that QEMU's VMware adapter already does this.

1.8 Feb 1, 2008 - TCG

Fabrice Bellard dropped a bombshell, introducing a new Tiny Code Generator to eventually replace dyngen, capable of breaking QEMU's dependency on gcc 3.X while simultaneously speeding up QEMU:

I added a new code generator (TCG) in QEMU. Read the file 
qemu/tcg/README to have technical information. A new code generator was 
needed in order to avoid problems with the various GCC versions and to 
get better performance.

I made minimal modifications in each target so that they can still work 
by using TCG and legacy "dyngen" micro operations. More work will be 
needed to convert each target to TCG, but it can be done progressively. 
Only the x86 and x86_64 targets have been significantly modified to use TCG.

TCG currently only supports x86 and x86_64 hosts. The support of other 
hosts is broken in QEMU. I don't consider it is a priority to support 
other hosts although TCG is generic enough to allow it.

The performance should not change much at this point. On x86_64 hosts, 
the x86 and x86_64 targets are faster because of some assembly language 
optimizations. On x86 hosts, the x86_64 target is faster because TCG can 
do some optimizations related to the conversion from 64 to 32 bits. No 
performance change should be noticed on non x86 targets because they are 
not converted yet.

The design of dyngen is described in detail in this usenix paper, and the problems with dyngen were covered in an earlier thread. This is the new code generator Fabrice mentioned in that thread.

This is an earthquake in qemu development. The core of qemu is a translation layer that converts blocks of target code into blocks of code to run on the host. For years, that translation layer was dyngen, based on memorizing special snippets of code produced by gcc back when qemu is compiled, and then playing those snippets back in various orders to produce blocks of host code. Dyngen is what tied qemu to a specific version of gcc, both convincing gcc to produce properly transplantable snippets and knowing how unwrap the object files produced by that version of gcc to extract the snippets.

Now QEMU can act as its own host compiler. By redesigning the core of qemu to produce its own executable code, Fabrice is breaking the dependence on GCC 3.x, although it won't be gone until all the targets are fully converted and dyngen can be removed. (In the meantime, the translator uses TCG for some operations and dyngen for others.) Also, to run on hosts other than x86 and x86-64, TCG must now be taught to produce code for each new host, just like any other compiler. Bu unlike dyngen's blind playback of prebuilt code, TCG can also be taught to optimize the code it produces.

1.9 Feb 1, 2008 - Common Xen/KVM patchqueue repository?

Samuel Thibault asked a question that

Anthony Liguori answered:

Samuel Thibault wrote:
> We were wondering whether it could be useful to have e.g. a git
> repository holding the patches from Xen and KVM: while they are being
> merged upstream, they could at least be shared by both projects...
>   

KVM currently maintains a git tree of it's QEMU changes so 
unfortunately, things aren't nicely separated into patches.  As you may 
have noticed, I've posted the core KVM support patches and I'm hoping we 
can merge them pretty soon.

I am then going to follow up and try to get the remainder of the patches 
merged.  In the short term, we will probably continue maintaining a git 
tree against QEMU as a staging repository but I hope that we can 
regularly sync up (just like we do with Linux).  In the long term, I'd 
like KVM not to maintain a QEMU tree at all.

In a broader sense, I know there are concerns about patches getting 
missed on qemu-devel.  I don't think the solution is to maintain a 
separate patch queue, but to get the patches merged more quickly in the 
first place.

[From this editor's viewpoint, the subtext looks like Xen asking to leverage KVM's generally higher patch quality to get its own patches in, and KVM turning them down gently.]

1.10 Feb 1, 2008 - [PATCH] OpenGL for OSX

Mike Kronenberg offered:

After a little discussion on the list, I made this patch for cocoa.m,  
to replace CoreGraphic by OpenGL.

It's way faster than CG, but it requires a Mac with OpenGL capable  
Graphics Card and at least 8mb of VRAM.
I think starting with G4 and Highend G3, this requirements are met.

To clarify, this isn't to emulate a 3D card within qemu, but instead using opengl on MacOS X hosts as a faster alternative to SDL for displaying 2D video.

A longish thread ensued, much of it wondering whether this should be genericized to provide an opengl alternative to SDL on other hosts, such as Linux. The counter-argument is that many versions of SDL can already use OpenGL to render their output, and perhaps performance problems on specific targets (such as 64-bit MacOS X, where benchmarks show opengl clearly winning over SDL) should be fixed in SDL, not worked around in QEMU.

1.11 Feb 3, 2008 - [PATCH] loadvm for read-only snapshot files

Eddie Kohler submitted a patch:

The following patch, against cvs, supports read-only snapshots on read-only 
qcow2 image files.  Snapshots can be loaded, but not saved.  This is really 
useful in my context, which is operating systems projects; a read-only memory 
snapshot allows students to share a snapshot that skips the boot process, 
allowing them to run their code right away.

1.12 Feb 4, 2008 - Making qemu use 10.0.3.x not 10.0.2.x

Asheesh Laroia complained that his local LAN used the same address range as qemu's default virtual network, 10.0.2.x. He suggested hardwiring in 10.0.3.x instead, because then it's not his problem.

Xen developer Ian Jackson suggested quite possibly the only way of making it worse: choosing an address randomly, which could make the problem hit more people in an intermittent and unpredictable manner. (Luckily, this suggestion was not well received. He later clarified he hadn't meant it.)

Much bikeshedding ensued. The suggestion was made to make it a command line option, but nobody volunteered to do the work, and the thread petered out.

2 Source control

Big news this week is svn 3943 and 3944: the Tiny Code Generator (TCG). It's a gigantic patch (7000 line, although much of it's ripping out old dyngen code) fundamentally changing the way QEMU works, in a good way. (This is the key to breaking the gcc 3.x dependency.)

Other interesting commits add a new gigabit card (Intel PRO/1000), and handle more SCSI commands.

Fabrice also yanked last week's new -translation option.

This week's commits: