Chris Ball

I took over maintaining the Linux kernel’s MMC/SD/SDIO subsystem recently, and quickly found that I was spending too much time saving, applying and compile-testing submitted patches (and trying to remember which of these I’d done for a given patch). The following Emacs/Gnus function helps with that — with a single keypress when looking at a mail that contains a patch, it applies the patch to my git tree, runs the kernel’s “checkpatch” tool to check for common errors, and kicks off a compile test in the background. I’m not much of an elisp coder, so feel free to critique it if you can.

(defun apply-mmc-patch ()
    "Take a gnus patch: apply; compile-test; checkpatch."
    (interactive)
    (setq default-directory "/home/cjb/git/mmc/")
    (setq compilation-directory "/home/cjb/git/mmc/")
    ; First, apply the patch.
    (dvc-gnus-article-apply-patch 2)
    ; Run 'git format-patch', and save the filename.
    (let ((patchfile (dvc-run-dvc-sync
        'xgit (delq nil (list "format-patch" "-k" "-1"))
        :finished 'dvc-output-buffer-handler)))
      ; Compile the result.
      (compile "make modules")
      ; Now run checkpatch.
      (let ((exit-code (call-process "perl" nil nil nil
                     "scripts/checkpatch.pl"
                     patchfile)))
    (if (eq exit-code 0)
        (message "Checkpatch: OK")
      (message "Checkpatch: Failed")))))

(define-key gnus-summary-mode-map "A" 'apply-mmc-patch)

First, some background: KDB (a kernel debugger shell) and KMS (kernel mode-setting) combine to let you drop into a graphical shell when something debugger-worthy happens on your Linux machine. That thing might be a panic, or a breakpoint, or a hardware trap, or a manual entry into the kdb shell. Inside the shell you can, for example: get a backtrace, inspect dmesg or ps, look at memory contents, and kill tasks.

This is a big improvement over the previous model of “something bad happens to your laptop while it’s in X, and the keyboard LEDs start blinking, and you hard-reboot and wonder what happened and wish your laptop had a serial port”.

Here’s a video of KDB+KMS in action — it’s from Jason Wessel at Wind River, who deserves massive kudos for having enough patience to get all of this debugging code merged into mainline Linux to everyone’s satisfaction:

I’ve written up a feature proposal on how we can use Btrfs snapshots to enable system rollbacks in Fedora 13, by gluing together the existing kernel code to do Btrfs snapshots, a UI for performing rollbacks, and a yum plugin to make snapshots automatically before each yum transaction. Lots of good comments so far, and LWN has written an article about it.

(Updated: The LWN link is no longer subscriber-only.)

It all started, as most things in the universe have, with a slight popping noise and a bad smell.

One of my coworkers sent me a message last weekend, while I was at home, pointing out that my desktop machine at work smelled like it was burning. After discussing the merits of turning off computers emitting burning smells before chatting with their owners about it, we had a look inside and found this:

The leftmost set of capacitors is fine, the set to the left of the fan is not. It’s actually a very well-controlled demolition; an electrolytic capacitor has boiled and blown out the top of its cap, which is perforated to enable exactly this failure method, and then once one cap is gone the rest are obliged to follow suit ‘cause the current on them increases in response. So, anyway, this left me in the market for a new graphics card.

The card in the picture is an nVidia 7300, which I bought two years ago because it was the only cheap card that could handle the dual-link DVI required for my 30” display with free drivers. Two months ago, though, Dave Airlie committed support for dual-link TMDS on ATI Avivo cards. These cards had historically been the worst of the worst for drivers—the free drivers were left nowhere without any specifications, and even the ATI binary driver for Linux took a long time to add support. Thanks largely to AMD’s latest NDA-less specification drops, though, we’re well on the way to a free accelerated driver for these cards, so I bought a X1600+ from buy.com.

When I booted X on it with the radeon driver, though, it looked craptastic. Since I like hacking on drivers, and since I didn’t have much of a choice, I thought I’d try to figure out what the problem was. The rest of this post is about what I tried during the subsequent bonding exercise between me and the registers on my new graphics card.

First, I asked around on IRC, and learned that it isn’t a known problem and that obtaining a register dump might be a good idea. You can get the “avivotool” that does this:

git clone git://people.freedesktop.org/~airlied/radeontool
git checkout -b avivo origin/avivo

I made the register dumps, but nothing was standing out. Then I lucked out: I tried the fglrx (proprietary) driver to see whether the problem happened there—it didn’t, and furthermore it didn’t happen on the radeon driver after fglrx had run. This suggests that fglrx is setting useful registers that radeon doesn’t even know enough about to reset when it starts.

The next step is to diff register dumps with broken-radeon and post-fglrx working-radeon. This got >100 differently-set registers. Dave Airlie gave me a basic explanation of the register layout: under 0×1000 and over 0×6000 are setup, and the rest are mostly the 2D acceleration space.

Manually setting the <0x1000 space to the working values didn’t come up with anything, though, and trying every register by hand was getting tedious. I wrote a perl script that takes two register dumps, assumes one is “good” and one “bad”, and offers to set each differently-set register to the “good” state, pausing for a second for you to inspect the screen output for a fix inbetween each.

And, eventually, it got it right—0×6590 (which radeon_reg.h knows to be AVIVO_D1SCL_SCALER_ENABLE ) and 0×6594 are both set when the card boots, but unset by the fglrx driver, and unsetting them while running radeon results in a fixed screen image. We can do this simply in the driver:

OUTREG(0x6590, 0);
OUTREG(0x6594, 0);

At this point, Alex Deucher (who works at AMD/ATI on open-source driver development—awesome!) stepped in and explained that the scaler code doesn’t work yet, and the driver should really turn it off when it isn’t being asked for. So now we do.

Here’s a photo of the working result. Planet Emacsers will be pleased to see a ridiculous amount of emacs in it.

GIT is immensely impressive. Sadly, Dominik Brodowski is even more impressive, and has a fix for the bug I was having fun bisecting sitting in his PCMCIA tree; note to self to next time mail the relevant maintainer and ask if they know anything about the bug you’re going to try and fix.

Here’s the git bisect visualisation letting me know which merge was responsible, which is where I decided to check out brodo’s tree. (Of course, I could also have continued the bisection down to the individual patch.)

Busy day. I’m pleased and impressed that 2/3 of the bugs I mentioned yesterday are fixed:

• S3 sleep works again after applying this patch from Hugh Dickins. I don’t get video or ethernet when I come back, but that’s taken care of by unloading my ethernet driver beforehand, and by killing/restarting X on resume. I should see if using vbetool differently (restorestate, perhaps) helps with that.

• A CVS commit to lvm claims to fix the problem I had, and the new package is in the buildqueue.

• Nothing new on my pccardctl eject oops. I’m going to try and git bisect this over the weekend — it’s a nice candidate for bisection, since it was working as recently as 2.6.16 and it’s not clear whether this is a locking problem (the PCMCIA code moved from semaphores to mutexes, but the patch looks sensible) or a netdev problem.

Keeping track of details for a few bugs I’m seeing in current Rawhide on the new laptop:

• S3 sleep is broken, possibly libata-wide since Jeremy is hitting the same symptoms as me (no disk access on resume) and he’s on AHCI while I’m on sd_mod and ata_piix. The symptoms are syslogd complaining that it can’t write out the journal, and lots of repeated:

sd 0:0:0:0: SCSI error: return code = 0x40000
end_request: I/O error, dev sda, sector 18800717

• pccardctl eject is giving me an oops, which is new to Rawhide; looks like this is upstream, since it’s the same oops as in this lkml post.
• The lvm(8) in my kernel-2141 initrd fails to boot, giving:

Volume group for uuid not found: (uuid)
0 logical volume(s) in volume group "group0" now active
mount: could not find filesystem /dev/root

I reverted bin/lvm in the initrd to the version found in 2136, which has the same version number but a different md5sum — that booted, so this looks like a new bug in lvm.

I’ve bought myself an Alienware m5500, which is a laptop based on the Uniwill 259EN3. I wanted a laptop with a high resolution screen (the m5500 does 1920×1200 on a 15″ LCD) and Intel graphics. (I don’t want to support either ATI or nVidia, and IBM and Dell make you choose an ATI/nVidia card if you go above their standard resolution screens). My machine has:

• Intel Pentium M 730 1.6GHz 2MB L2 Cache 533MHz FSB
• 512M RAM, 40G disk
• Alienware m5500 15.4″ WideUXGA 1920×1200 LCD
• Intel GMA900 (915GM/i810) Extreme Graphics
• Intel PRO/Wireless 2200 b/g Wireless Card

I have Fedora Core 5 running on it now, but I’m sad to say that it wasn’t without pain:

I booted from the FC5 install CD. The kernel booted, switched over to Anaconda, and I got a black screen and hung system. Tried again, this time using these instructions to start a VNC installer. That worked, and I soon found out that Fedora now has no NTFS support at all, which meant that the single NTFS partition taking up the entire disk would have to be deleted. (Which sounded fine at the time but is annoying me now — see the wifi section below.) Ubuntu’s installer can do NTFS resizing, but Fedora thinks that the kernel NTFS support violates patents and is not legally redistributable.

After the install, I got the same X crash on my first boot. Booting into runlevel 3 got me to a shell. I determined that the X failure was this bug, which requires applying this patch to Xorg to get the card working. Now that Xorg is modular, it’s (thankfully) sufficient to just download the latest xorg-x11-server.src.rpm, make the change, and rpmbuild -ba to have the specfile take care of building you a new RPM.

That got X working at 800×600; to get to 1920×1200 you need to use 915resolution and the following modeline:

ModeLine "1920x1200" 230 1920 1936 2096 2528 1200 1201 1204 1250

Now, on to wireless. I downloaded the ieee80211 stack, ipw2200 driver and ipw2000 firmware, and:

unity:cjb~ % sudo modprobe ipw2200
ipw2200: Detected Intel PRO/Wireless 2200BG Network Connection
ipw2200: Radio Frequency Kill Switch is On:
Kill switch must be turned off for wireless networking to work.
ipw2200: Detected geography ZZM (11 802.11bg channels, 0 802.11a channels)

There is a wifi button on the case, next to the power switch — which implies that it’s a hardware button, rather than a software button like the function keys on the keyboard — but pressing it turns on the “sleep” LED rather than the “wifi” LED, and ipw2200 continues to make the same complaint. I see that drivers exist to control the kill switch in software, but none I’ve seen work on this machine. I suspect that booting into Windows (which I can’t do now since I wasn’t able to keep/resize the partition) would get the wifi working in Linux either temporarily or permanently. I took a backup of the disk (dd if=/dev/sda of=nfs-mounted-backup) and booted the XP recovery CD that came with the machine, but it either hung or needed many minutes at “Setup is inspecting your computer’s hardware configuration”, and my patience for Windows is pretty limited. I’ll try again if I don’t get any other suggestions.

I posted some cries for help about this and dug out my very old, trusty Orinoco wireless PC Card and inserted it. And the machine hung immediately, and did so repeatably. I dug around until I found this linux-kernel thread which contains a safer /etc/pcmcia/config.opts, and that’s working fine. Update: Hmph. It works anytime after init has loaded (udev, perhaps?), but the machine hangs at the cs: memory probe if I try to boot with the card already inserted. Any ideas?

I set about installing AIGLX, and the performance is good, especially for an Intel chipset. The photo at the bottom shows AIGLX and true transparency in gnome-terminal, which is lovely to have (at last), especially on a laptop when you’re likely to want to be copying from a web page into a terminal, etc.

So, that’s about it for day one. To summarise:

ACPI

Well, sleep (suspend-to-ram; S3) seemed to be working out of the box, but it isn’t now. It looks like the IDE driver isn’t coming back; if I can keep an ssh session connected, I see repeated in dmesg:

sd 0:0:0:0: SCSI error: return code = 0x40000
end_request: I/O error, dev sda, sector 18800717

I’ll look into having the sd module removed and restarted — anyone know how I should do that on FC5/Rawhide? I also had to change resume_video() in /etc/pm/functions-intel to get video back:

#/usr/sbin/vbetool post
#/usr/sbin/vbetool vbestate restore < /var/run/vbestate
/usr/sbin/vbetool dpms on

Suspend to disk works, and doesn't even take that long, but kicks me back to the gdm prompt (!) from a logged in session. Update: Working now, after putting 915resolution in the resume path as well. Thanks, Jens!

The ACPI function keys don't do anything in software, not sure which driver they need.

CPU scaling

Working, switches between 750MHz/1.1GHz/1.6GHz automatically.

Graphics

Working after patching Xorg, acceleration works, AIGLX works.

Synaptics touchpad

Working, I disabled tap-to-click and vertical scrolling:

Option      "VertScrollDelta" "0"
Option      "HorizScrollDelta" "0"
Option      "MaxTapMove" "0"
Option      "MaxTapTime" "0"

USB, Sound (snd-intel-hda), SD card reader, DVD burning, Firewire.

Working.

VGA out

Works, after adding the following:

Option "MonitorLayout" "CRT,LFP"
Option "Clone" "true"
Option "SWCursor" "true"

I'll keep this post updated as I learn more about the machine. Here's the photo:

After writing my previous post about notifications (which is required reading for the rest of this one, I’m afraid), I was able to talk to Robert Love about the Yi Yang patch, and why he thinks it didn’t get a good response. He gave a few reasons:

• It’s lossy; you lose events from boot time, from before the userspace daemon starts, and if the daemon crashes.
• Requires root to listen on netlink, as opposed to the inotify_add_watch(2) syscall interface of inotify.
• For this purpose poll(2) is good, netlink is bad.

All of which are reasonable complaints. If that’s the wrong solution, though, what would the right one look like? Thankfully, Robert has ideas on that too (and I hope I explain them correctly):

---

To avoid the causes of lossiness above, the log should be an on-disk log maintained by the filesystem. Having it done by each filesystem separately isn’t necessarily awful for maintainability; the ext3 journalling layer is supposedly generic. There should be sequence points, so the (single) userspace daemon reading the log knows that it’s up to date as of sequence n, and can tell the kernel to clean the parts of the log up to n.
The on-disk log would be fixed in size and circular — so still lossy so far — but Robert has an idea (which Tridge and Rusty Russell are apparently also partly responsible for) to make sure the lossiness doesn’t hurt so bad. Here it is.

You do event “compression”; the log is stored in a tree of path names and events. If you have change events for a couple of hundred files in /home/foo/{bar,baz,etc}, you mark all of /home/foo as dirty and throw away the events inside it. Userspace has to go off and stat(2)-dance inside /home/foo to find which files have changed, but at least you’ve traded precision for accuracy and come out with a log that enables every change to be noticed. You’d keep reparenting as you run out of room in the log, so if you exhausted log space recording changes in /home/foo and /home/bar, all of /home gets marked dirty.
This is still root-only so far, but you can build security on it.

---

So! This is a lot more work than Yang’s elegant netlink patch, but I’ve decided that ridding the world of updatedb is a worthy goal, and so I’ll be starting work on Robert’s design next week. I’ve booked days off work to go to LinuxWorld Boston, and an interested friend is visiting from the UK as well. A further idea is to get the userspace daemon to export inotify-compatible events, so that programs like Beagle can use this new mechanism without requiring a rewrite. I’m assured that apps like F-spot and Leaftag are waiting for this kind of event notification too.

(Note: Firefox hung as I was writing this post, taking the unfinished blog entry with it, with strace hanging on a futex. I blame the flash on the LinuxWorld site. But! Getting a core file with gdb’s gcore and running strings on it gave me a perfectly-formatted blog post back. Yay!)

It took far too long to find this half-remembered linux-kernel thread on Google amidst all the results (mostly from distro installer guides) claiming that swap partitions are preferable to swap files. Here’s hoping the link below will help future searchers.

Swap files and swap partitions have the same performance:

In 2.6 [swap files and swap partitions] have the same reliability and they will have the same performance unless the swapfile is badly fragmented.

— Andrew Morton, on linux-kernel.

(There is a performance difference under Linux 2.4, though, as explained at the link above.)

Update: Tim comments that swsusp (the kernel software suspend-to-disk support) only works with swap partitions, so there is still one good reason to use a swap partition. Suspend2 doesn’t have this limitation.