The big news this week was Fabrice Bellard announcing work on a new code generator, which (when it goes in) should finally allow qemu to build with compilers other than gcc 3.x.
Other topics include support for VNC reverse connections (allowing a VNC server to connect to a waiting qemu instance), how and why to build a static qemu binary, the project's painful position caught between GPLv2 and GPLv3, some disagreement about the proper way to support the VMWare tools interface, a bug report affecting support for PS/2 mice, and more work on MacOS X host support.
Jeremy C. Reed asked:
Any way to startup qemu with my own desired speed?
It does not have to be precise. And I don't want to slow down the host system itself.
For example, I am using a 1900MHz system and need to test some applications usability and performance as if it were running on a 233 MHz CPU (or close).
Laurent Desnogues insisted it couldn't be done, and then later that it hadn't really been done after Mulyadi Santosa pointed out a qemu-brake patch which worked for Jeremy.
Eddie Kohler subitted a patch:
This patch against current CVS adds VNC reverse connections, where the server connects actively to a waiting client, as in "-vnc rev:5500" or "-vnc rev:read.cs.ucla.edu:5500". This is quite useful if the user expects to run QEMU many times in succession (for example, is debugging a toy OS), and doesn't want to reopen a VNC client each time.
Daniel P. Berrange suggested the command line use a postfix flag instead. Eddie came up with such a patch, but pointed out that -vnc without the flag uses a display number, while reverse uses a port number, so it wasn't actually any cleaner. Daniel withdrew his objection, but the postfix syntax is what the maintainers decided to merge anyway.
Salil Bijur noticed that the "--static" option to qemu's configure still doesn't work, and asked if it had ever been fixed:
I've been trying to build QEMU statically by first configuring it using the --static option. The compiling gives me the same linker errors as mentioned here: http://www.mail-archive.com/qemu-devel@nongnu.org/msg10721.htmlI know this has been asked before but it hasn't been followed up. Any ideas on how to fix this?
The problem is that qemu hasn't got a "--chroot" option, so users wanting to run target applications using application emulation have a bit of a problem. Their options are:
Given a dynamically linked target application, the first option isn't enough. The second option mixes host and target libraries on the same system, which is messy and tends to cause conflicts between the two sets of files.
Copying a dynamically linked qemu into a target filesystem requires copying all the host's shared libraries (and library loader) qemu needs into the target root filesystem. This just moves the problem around without solving it.
Hence the need for a statically linked qemu, to run out of a chroot environment. Building such a static qemu turns out to be possible but not easy.
(Note: In March I posted a --chroot patch, in belated response to this thread. This allows a dynamic qemu to perform its own chroot after initializing itself but before trying to run the target binary, eliminating the need for a static binary in the target directory, at the cost of running qemu as root so it can chroot itself.)
Christian Erhardt brought over a thread from the kqemu list about sign of char variables.
On some platforms (such as x86) the "char" type defaults to signed (which is what you'd think if you looked at short or int). On other platforms (such as arm) "char" defaults to unsigned (which is what you'd think if you looked at strings of 8 bit ascii values). QEMU is treating char literals as unsigned, which means on platforms where char defaults to signed, assigning a "literal string" to an unsigned char * generates a warning about a pointer type mismatch due to the differing sign.
Suggested solutions included telling the compiler not to generate these warnings (-Wno-pointer-sign) and adding typecasts to every occurrence. (Nobody mentioned the -funsigned-char flag, to consistently specify the sign regardless of the platform's default.)
Alexander Graf took the annual stab at getting qemu to build with gcc 4.x:
Compiling qemu on gcc4 has been a long standing issue. During the time several approaches came up, each more or less intrusive. I have collected the ones I found to be the cleanest, fixed PowerPC host support with Michael Matz and am sending everything as a patchset now, hoping that someone comments on these patches and they will finally get included. Please test these patches on all platforms you have available.
This started an enormous thread. Much debugging ensued, with many more patches and reports that it only worked on gcc 4.1 and not gcc 4.3, that it broke building under windows via a mingw based on gcc 3.4. Andreas Farber set about testing it on MacOS X.
Johannes Schindelin got it to work for him, and pushed the result into a gcc4 git branch. (He and Michael Matz then went off into a long sub-thread about tweaking assembly statements to work around bugs in different versions of gcc's register reload.)
Fabrice bellard brought the party to a sudden stop:
I must say I don't like such patches because they are likely to break with every new GCC version. Moreover, I will commit in the next few days a new code generator in QEMU which will utimately solve the compilation problems. In its current form it still relies on "dyngen" so that legacy micro operations still work, but in the end dyngen will be completely suppressed. A side effect of these commits will be that only the x86 and x86_64 hosts will be supported during some time.
This seems to be a successor to Paul Brook's old "qops" patch, and requires some backstory.
QEMU has always needed to know about each target it supports, so it can read that particular type of machine language. But the QEMU developers wanted to avoid having a whole second set of dependencies for each _host_ it could run on, so it could generate the appropriate type of machine code.
To avoid the need for explicit support for each different host, Fabrice invented dyngen. During the qemu build, dyngen would ask gcc to compile lots of small functions to individual object files, and then strip the machine language out of each of those little object files and save it for use at runtime. When qemu needed to generate host code at runtime, it would output the saved chunks of code dyngen had saved (from those .o files at build time), stitching them together into a translated block of code.
Unfortunately, this approach just traded one type of dependency for another. Now instead of running on a restricted set of hosts it know how to produce code for, QEMU only built with a stricted set of compilers that produced object files that didn't violate any of dyngen's (many) expectations. And most unfortunately, gcc 4.x started violating dyngen's expectations (doing things like producing a return instruction in the middle of each object file rather than at the end where it could be easily stripped off).
The QEMU developers have long said the proper way to fix this is to bite the bullet and teach QEMU to directly produce machine code for each supported host, it's just a really big job without an easy transition path from one to the other except to break everything. Paul Brook had an a code generator called "qops" working as far back as 2006, but it produced substantially slower code than dyngen. Theimo Suefer says that Fabrice's new code generator produces code that runs about as fast as dyngen's.
So what Fabrice's above message means is that he's introducing a new code generator that can directly produce machine langauge, but initially only implementing it for x86 and x86_64 hosts. Those hosts should be able to use it to emulate all supported targets, but running qemu on an arm or ppc host will still require dyngen (and thus gcc 3.x) until those hosts get their own code generators.
A few developers protested that they'd spent so much time jury-rigging the old approach to sort of work with gcc 4.x, and it seemed a shame to just discard that code even if it caused regressions and wasn't the right approach going forward. Most just grumbled a bit, although one threw an outright temper tantrum until the flame petered out.
In a later thread on the same topic, Julian Seward (the author of Valigrind and bzip2) gave a nice summary of the issue:
It's not a terribly big deal. Writing backends is a lot easier than writing front ends, since the back end can just emit some small convenient subset of target instructions, whereas the front ends have to deal with every stupid, obscure, weird-ass instruction that ever shows up. QEMU is not the first project to post-process gcc's output. The Glasgow Haskell Compiler (http://en.wikipedia.org/wiki/Glasgow_Haskell_Compiler) did that for many years and it was always an immense amount of hassle tracking the changes to gcc's code generation. Having a completely-independent-of-everything, standalone code generator is definitely a lot easier in the end.
Ulrich Hecht of SuSE kicked off this thread:
Our Teenage Mutant Legal Turtles have discovered that s390-dis.c contains GPLv3 code. Fortunately, the actual code has not changed since the last GPLv2 binutils release, save for the license headers, so it suffices to change those back to fix the problem.
The problem is that for many years, the Free Software Foundation assumed that all softwrare licensed under version 2 of the GPL was dual licensed with a "GPLv2 or later version" phrasing, which allowed future releases to move to GPlv3 (dropping GPLv2 from its multiple licenses). In reality, many projects such as the Linux kernel long ago specified) that they were licensed GPLv2 only. And other projects licensed "GPLv2 or later" don't want to leave GPlv2 behind, and thus can't accept code licensed _only_ under GPlv3.
Other projects have wrestled with this problem before, and now it's QEMU's turn.
Over the course of the thread, the qemu developers identified several files in qemu taken from GPLv2 only sources (mostly the Linux kernel), and identified other GPLv2 projects (such as xen and kvm) which use qemu code and would like to stay in sync with new releases of QEMU.
The license statement on QEMU itself doesn't specify a version, and the text of the GPLv2 license says that in that case you can use any version (including the long-obsolete GPL version 1). Since QEMU already includes GPLv2 only code, and the practical upshot of this thread was to remove a GPLv3 license notice (which applied to binutils code identical to a previous GPLv2 release), for the moment QEMU can only be distributed under the terms of GPL version 2. Clearly, some developers are open to allowing the code they've contributed to QEMU to be used under other licenses, but which code and which licenses remain undefined.
The QEMU developers did not take an official position on this issue in this thread. They're caught between being unable to use code from new versions of gcc/binutils/gdb and being unable to use code from any version of the Linux kernel (or have their own code used by linux kernel projects such as kvm). Currently, the status quo sides them with the Linux kernel and GPLv2.
Someone whose name looks a bit like Mapk Kopehbepr asked if he could install m68k Linux under qemu.
A patch posted back in July provided userspace emulation, but the patch's author states:
Even with the patch there are still quite some things missing. For a full system emulation you need more supervisor level instructions, and a proper fpu emulation is missing as well.
Laurent Vivier suggested some alternatives:
Debian m68k developers use Aranym (http://aranym.org/) for their buildd, which is an Atari Emulator, based on UAE CPU emulator. I think you can install Debian m68k on a disk with it, binaries are the same for mac (except the kernel), then you can move all files to a mac partition (atari partition table and mac partition table are not compatible). Then, to boot, I (modestly) advise you to use EMILE ;-) (http://emile.sourceforge.net)
The thread started with a question whether QEMU could emulate the new Intel VT extensions needed to run things like Xen and KVM. (The answer was to emulate a 64-bit system with qemu-system-x86_64, which emulates a newer processor supporting the VT extensions which is still able to run the old 32-bit code.
Along the way, Alexander Graf commented on the inability of KVM (a linux kernel based derivative of qemu, sort of an alternative to kqemu) to emulate the VT bits it needs to run:
Simply said, you can not run KVM inside of a KVM virtualized machine.
To which andrzej zaborowski asked:
What would be the problem with supporting that in KVM? The host SVM should just report that the guest is trying to execute an SVM insn and generate an Illegal Operation which would be handled by qemu.
right now kvm never call to qemu for cpu emulation, qemu is used just for the devices,
Alexander Graf replied to andrzej:
Yes, you're completely right about this. The main reason it isn't implemented in kvm is that nobody did it yet. It is actually one of the things on my "would like to do"-list, so if things stay the way they are, maybe you will see this feature in kvm someday.
Theimo Seufer noted the following about the 0.9.1 release:
Only the most notable changes are mentioned in the ChangeLog. It seems some of them (-like -drive) were forgotten.
It started when Filip Navara noted the lack of security in VMWare's backdoor, and proposed making qemu's emulation of VMWare's backdoor equally insecure:
the current version of QEMU emulates the VMware backdoor I/O port and it works quite well. Unfortunately it doesn't emulate the VMware behavior of ignoring the I/O permissions when accessing this special port. The attached patch corrects it. It's important to ignore the permissions, so that user mode VMware tools can communicate to the backdoor.
(For those who don't know, VMWare tools is a set of utilities and drivers that run inside an emulated operating system to allow things like cut and paste between host and guest OS windows, which require the client OS to communicate with the host OS through the emulator in order to work properly.)
I really dislike that VMware relies on this. It's very hard to implement in kqemu or KVM. I think it would be better to modify open-vm-tools than to modify QEMU.
>Alexander Graf opined:
I think it would be great to maintain compatibility with the binary-only versions of the vm tools though.
To which Anthony Liguori replied:
But you're changing the semantics of the x86 instruction set. You potentially break a real operating system. It also eliminates the possibility of nesting with something like kqemu because you can't trap all PIO operations.
Mark Williamson suggested a command line flag for qemu to specify the desired behavior.
Alexander Graf continued the thread:
I completely agree with the point of breaking x86 semantics is bad. Yes, it is. What is the point in emulating the VMWare interface though, if the only program actually requiring that interface does not work, namely vmware tools, especially the windows version. So as far as I know VMWare uses VMX to run 64-bit code on Intel as well, so there has to be a way to forcefully break the checks.
Jamie Lokier gave a detailed explanation of the problems with VMWare tools' current design, and suggested potential workarounds.
Finally Anthony Liguori explained:
vmmouse uses the vmport interface but runs in ring 0 under Linux so it's not an issue. FWIW, the folks on open-vm-tools-devel have expressed an interest in moving to a different interface then their "backdoor" interface.
Even Roualt noted a bug in QEMU 0.9.1 mouse support:
I've tried QEMU 0.9.1 and PS/2 mouse support for FC4 guest seems to be broken. I was using previously a CVS version dating back to 2007/09/07 and it works fine with it. It can be tested by simply running qemu on the FC4-i386-DVD.iso installation ISO DVD. Anaconda doesn't manage to find the mouse and fall back to text mode. On an already installed image, the mouse doesn't work in X or in text mode.
(This bug may also be the reason why the Knoppix 4.0.2 iso image I have lying around no longer has a mouse when booted up under QEMU 0.9.1, but worked fine under 0.9.0.)
An hour and change later, Even reported that he'd found the guilty change, and that reverting it fixed the problem:
After quite a lot of CVS bisection, I've identified revision 1.24 of
hw/pckbd.c ("QEMU keyboard issue with Gujin-2.2") to be responsible for the
regression with the PS/2 mouse for FC4 guests.
With r1.24, I can read in the log of the kernel boot sequence :
"PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU@ at 0x60,0x64 irq 1,12"
Failed to disable AUX port, but continuing anyway... Is this a Sis?
If AUX port is really absent please use the 'i8042.noaux' option.
serio: i8042 KBD PORT at 0x60,0x64 irq 1"
With r1.23, I only get :
"PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU@ at 0x60,0x64 irq 1,12"
serio: i8042 KBD PORT at 0x60,0x64 irq 1"
Johannes Schindelin chimed in to explain how Even could have found it more quickly by using Johannes' git mirror and doing a git bisect.
Even thanked him, and politely asked if anybody would like to comment on the actual problem he'd found.
Eventually, andrzej zaborowski suggested:
I have no clue about ps2 but seeing as redirecting the command from kbd to mouse (0 -> 1 in the kbd_queue call) fixes the keyboard issue for Gujin, but breaks mouse, the obvious conclusion is that the command should not be done for either kbd or mouse. Can you try with line 214 (previously line 207) simply removed? If you can test also how that affects Gujin that would be even better.
Even debugged further:
I've tried as you suggested to remove line 214 ( "kbd_queue(s, s->mode, 1)" ), and it breaks both keyboard and mouse support for the FC4 install DVD or my FC4 disk image. When booting the FC4 disk image, I can read "i8042.c : Can't read CTR while initializing i8042." in the kernel boot log. As far as Gujin is concerned, it would be great if people familiar with it could try. I'm looking for an available disk image with Gujin as the boot loader but no success for the moment. I guess I'll have to figure how to install and use it.
The thread petered out there with no resolution.
C.W. Betts asked a question that would be in the FAQ, if qemu had one:
I was thinking, maybe qemu could use threads for at least every processor it emulates (on emulated smp computers) and, at the most, every single device emulated. This would help users who have multiple cores, but it might impact performance on those of us who don't. Just an idea I'm throwing out.
Paul Brook replied:
Please read previous discussions on this mailing list. I'd be surprised if putting device emulation in a separate thread makes much difference. The really slow bits (waiting for IO to complete) are already asynchronous. Most other device accesses are very short, so you'd waste more time through synchronisation than you gain from putting them is a separate thread. Splitting multiple CPUs into multiple threads is extremely hard to get right, especially when your host system provides less strict ordering and atomicity guarantees than those required by the guest system.
Johannes Schindelin gave a link to a previous thread on the issue.
C.W. Betts then asked
Still, is there a way to make qemu take advantage of multiple cores? They are pretty commonplace in new computers (is there any selling computer that doesn't have multiple cores?).
And Daniel P. Berrange suggested:
If you've got recent x86 CPUs, then they may well have hardware virtualization support, in which case you can use KVM which can take advantage of multiple cores. At least for native host/guest i686 & x86_64 combos.
William Pearson suggested a clustering approach:
It depends on what you want to do. You could always run two or more copies of qemu and set up a small networked cluster of the architecture you want to emulate. Each emulator should go on to a different core. Might be useful when compiling things in qemu or doing other processor bound tasks.
(For the record, I note my own Firmware Linux project's use of distcc to address a similar problem.)
Mike Kronenberg started the ball rolling:
This is a complete rewrite of cocoa.m to support Core Graphics. As mentioned in earlier threads, the QuickDraw API is depreciated starting with OS X 10.4. Now with OS X 10.5 it won't even compile QuickDraw code on x86_64.
Mike's patch was quickly applied [LINK], but a week later, Alexandar Graf replied:
I'm sorry that I didn't find the time to test this implementation before. It's damn slow. I ran it using my x86_64 on 10.5.1, targetting x86_64-softmmu and booting a linux kernel. I could literally see every like getting repainted (which btw did not happen with my quick hacky version I sent to the list some time ago). I think the major problem is that too much is being done during drawRect. If I understand the code correctly, you create CGImage objects on every repaint, which is prone to be slow. Why not simply reuse the framebuffer qemu provides anyway and leave everything else to CG?
It's great to see someone working on Mac OS X support and thanks a lot for writing a compatible implementation of cocoa.m that works with every Mac OS X version out there.
Mike replied:
yes, as stated earlyer, it is slower than Quickdraw, especially if the whole screen is redrawn. Overal emulation speed for GUI apps is faster, dough, as only small portions of the screen are redrawn. Unfortunateley, there is no "official" direct access to the framebuffer anymore, since apple depreciated QuickDraw. [1] This way, there is no sharing/direct mapping of the cg framebuffer and qemu screenbuffer anymore.
Pierre d'Herbemont suggested an approach using an Apple extension to OpenGL to tell MacOS X to share 3D texture memory with a userspace buffer, but there was no reply to his message.
Instead the conversation went towards detecting partial updates. Mike replied to Alexander:
> I ran it using my x86_64 on 10.5.1, targetting x86_64-softmmu and > booting a linux kernel. I could literally see every like getting > repainted (which btw did not happen with my quick hacky version I > sent to the list some time ago). You did not notice the effect with Your implementation, because the the whole screen is redrawn every time. With this implementation, only the parts requested by qemu are updated.
Anthony Liguori explained further:
VGA framebuffer operations come in as memory operations. They're tracked by watching what memory gets dirtied. This can only operate at a page-granularity so this results in scan-line granularity updates. The VNC front-end goes to great lengths to keep a shadowed framebuffer and reduce these updates to a smaller update region. You could possibly look at refactoring that code. However...
Jamie Lokier chimed in:
That update region code should probably be moved to something generic and made into a generic display option. Reducing update region is logically orthogonal, and could work with any update method (e.g. local X11, remote X11, local X11-OpenGL, remote X11-OpenGL, SDL etc.). With some of those, for some people (especially some but not all remote setups) it might be worth it.
To which Julian Seward replied:
For exactly these reasons I developed a shadow framebuffer patch which ... > makes QEMU's graphics emulation much more usable over remote > X connections, by reducing the amount of data sent to the X server. > This is particularly noticeable for small display updates, most > importantly mouse cursor movements, which become faster and so > generally make the guest's GUI more pleasant to use. See http://lists.gnu.org/archive/html/qemu-devel/2007-03/msg00141.html for the patch and short associated thread. It never got included, though.
There was no reply. Along the way, Jamie Lokier pointed out that emulating a "hardware mouse cursor" could improve the redraw situation. Nobody replied to that either.
Slow week, just bugfixes and documentation.
This week's commits: