On Thursday 13 September 2007 3:11:24 am Michael Kerrisk wrote:
> Hey Rob,
>
> I just got myself subscribed to linux-doc, finally. Since I
> do stuff in the nearby space (kernel-userland API docs), and
> I haven't been following your endeavours closely on LKML,
> I'm curious how things are going overall with kernel
> documentation.
Pretty well.
I just got a skeleton of the master index up at
http://kernel.org/doc/master.html (which is converted from
http://kernel.org/doc/master.idx via make/indexsections.py). I'm not really
happy with the section headers that the index links to. I just added section
numbers which is makes it less bad but I'm still not quite comfortable with
it. Maybe I need
tags in there or something. Possibly they should be
. I need to look at it in Firefox rather than just Konqueror...
Currently I'm taking a break from that and working out a plan of attack for
the EXPORT_SYMBOL macros in the source. There are over 12,000 of them, and
they should all be documented. Probably as a "make htmldocs" book. I have
an email about that which I sent to my Linux Foundation boss off-list, which
I should probably dig up and repost here.
> What are your rough master plans and projects going
> forward, in terms of kernel documentation? (I ask
> somewhat naively, because I'm not exactly sure what your
> remit is for the LF fellowship year.)
What's a "remit"? Do you mean my expected output?
What I'm trying to do is:
A) Get everything documentation-related from the kernel tarball onto the web
where Google can find it. This includes the Documentation directory, "make
htmldocs", the README files scattered in the source code, the menuconfig
help, and miscelaneous like the output of "make help". Currently, all of
that _is_ on the web, and updated by scripts I run when I remember to. (I'll
probably make a cron job out of this eventually, but I don't want to run
anything cpu-intensive on kernel.org and any use of "xmlto" falls deeply in
that category.)
B) Index the great heaps of documentation out there on the web, to create a
central go-to site for people who want to know things about the Linux kernel.
This includes lots of stuff that _won't_ be merged into the kernel tarball.
There was some discussion about making the Documentation/* directory the
central repository of all kernel knowledge and wisdom, but this doesn't work.
Ignoring the fact that it's currently totally disorganized (which can be
fixed), and also ignoring the fact that it's not even the central repository
for in-kernel documentation (there are at least three other major sources of
documentation in the kernel tarball listed above), there's a design problem.
The Documentation directory is full of _text_files_, but lots of
documentation is out on the web. Yes, there are good text files that go by
(often on linux-kernel as patch [0/##], or as git checkin comments, or in
developer blogs...) But there are also Linux Weekly News articles in html,
OLS papers in PDF format, audio files in mp3 and ogg, video files in whatever
Google puts its Tech Talks in (flash video, I think)... There are literally
gigabytes of this stuff out there, in varying formats under varying licenses.
It isn't even feasible to mirror all of it on kernel.org, let alone merge it
into the kernel tarball, let alone convert it all to ascii text.
If you're linking out to stuff on the web, the format you want to use is html,
not text. Converting Documentation to html would be a big controversial
change, and I'm Not Going There (tm). Documentation's job is to host
documentation, but _my_ job is primarily editorial. I find it, link to it,
and clean it up/fill in the holes as I have time.
So I'm creating a master index of kernel documentation. Some of it will be
local content, some of it will be mirrored locally, and some of it the index
just links to and runs a 404 checker against (with reliance on the wayback
machine for stuff we don't mirror until it goes down).
> I'm not sure whether it's in the remit, or whether you have
> the time or interest in helping out in kernel-userland
> documentation (i.e., man-pages), but there are certainly
> things to do there if you want. Let me know if you are
> interested.
The interface between kernel and userland is actually my main area of
interest.
There's not much point in documenting the stuff that's going away again in
three releases, which includes most of the deep kernel internals. It's
_nice_ having that stuff documented, but not the world's best use of time
trying to nail jello to the wall. The kernel-userland interface is _stable_,
so once we document that it STAYS DOCUMENTED (more or less), so there's no
much point going after the more ephemeral stuff until the stable stuff's
nailed down.
The question was raised at the kernel summit: is man-pages documenting what
the kernel exports, or is it documenting the interface glibc exposes to
userspace (often through a heavy wrapper)?
http://lwn.net/Articles/248376/
http://lwn.net/SubscriberLink/248376/24c8808f1c680d5e/
> Part of Michael's proposal is that new system calls should come equipped
> with manual pages. It was suggested that this requirement will be hard to
> enforce unless the man pages are packaged with the kernel itself. That led
> to an interesting question: the man pages, as currently written, document
> the system call interface as presented by the C library. But the API
> exported directly by the kernel can be different, and often is. Which API
> should be documented? It seems that the kernel-implemented API is the one
> to cover, especially considering that glibc is not the only C library and
> that other library implementors may well be very interested in that
> information.
Once again, "distribute it with the kernel" is seen as some kind of magic.
It's not. Honest and truly it's not. It just means that updating the
documentation results in your post getting lost in the noise on linux-kernel
and having to go through the four-layer bureaucracy (developer, maintainer,
sour cream, lieutenant, baked beans, Andrew, Linus) with somebody objecting
along the way that your patch has the signed-off-by line in the wrong
position and you forgot to include three magic dashes as mentioned section 14
paragraph 21 of Documentation/SubmittingPatches (no seriously, look it up),
and in any case you weren't supposed to send it to THIS list you should have
sent it to netfilter-devel (didn't you read
http://vger.kernel.org/vger-lists.html? What's wrong with you?) and then
it's still being ignored after the third repost two months later...
And of course the stuff that's in the kernel now is such a marevlous example
of up-to-dateness. Documentation/scsi/scsi.txt is primarily a link to two
online documents (http://www.tldp.org/HOWTO/SCSI-2.4-HOWTO and
http://www.torque.net/scsi/SCSI-2.4-HOWTO) which were each last updated about
5 years ago and are for the 2.4 kernel, yet despite that are the most
up-to-date and comprehensive documentation available about a major kernel
subsystem. (Yes, I asked.) I'd like to improve this, and plan to, but I'm
not a scsi guy. I have to learn it before I can document it, along with
everything else I have to do.
Anyway, yes I am interested in helping out here, it's a question of free
time. :) You may notice that I've kept up the Doclifter html translation of
the man pages (http://kernel.org/doc/xmlman) since
http://kernel.org/doc/man-pages remains empty. :)
Among other things, I'm interested in documenting system calls and ioctls. I
didn't even know "man 2 syscalls" existed until last week. (I was pulling up
man 2 syscall and not getting an index. :) There isn't a man page for ioctls
but I can start with: find /usr/include/linux -type f | xargs grep "_IO.("
Part of what I'd like to do is index the existing man pages more coherently.
There's lots of great stuff there I just haven't had time to _read_. But
then that's true about every decent source of documentation, you'll notice I
have a large list of sources already. They need to be _integrated_, and
that's serious work...
> Best regards,
>
> Michael
Did I answer your question? :)
Rob
--
"One of my most productive days was throwing away 1000 lines of code."
- Ken Thompson.