Ticket #2135 (new enhancement)

Opened 6 years ago

Last modified 6 years ago

write kernel crash message somewhere where it can be retrieved after reboot?

Reported by: lindi Owned by: openmoko-kernel
Priority: high Milestone: stable-kernel-2009.1
Component: kernel Version: unspecified
Severity: normal Keywords:
Cc: Blocked By:
Blocking: Estimated Completion (week):
HasPatchForReview: no PatchReviewResult:
Reproducible:

Description

This is a wishlist bug. Would it be possible to write the kernel crash message somewhere where it can be retrieved after reboot? Would writing to nand (to dedicated partition) be safe? If uboot does not overwrite RAM contents could the error be retrieved from RAM after reboot?

If this fails it should be quite easy to send the contents of the panic message by blinking the AUX led, right? The only problem here is that there needs to be some automated way to capture it.

Audio would be easier to capture since people have soundcards to record it but it would probably be harder to control when the kernel has crashed (and it would annoy everyone around you while you are booting your laptop to capture the crash message :P).

Attachments

ramconsole-1.patch (2.3 KB) - added by lindi 6 years ago.
patch to log kernel messages to circular buffer that persists over reboot
ramconsole-2.patch (2.1 KB) - added by lindi 6 years ago.
new version, do not print contents of ramconsole on boot, let userland handle it

Change History

comment:1 Changed 6 years ago by werner

I would use RAM. It's easy to access and it's very likely to survive a
reset. Also, since the kernel itself may not know it is dead (e.g., if
the system just hangs and you have to reset it), you would want to write
all messages to that "safe" location, so access should be inexpensive.

For retrieval, you could either extend the boot loader, or, better, only
start writing to the "safe area" after you've checked for the presence
of old content. That way, crashes occurring before the retrieval could
not be recorded (if you use multiple buffers you rotate, you could even
handle some cases, but it may not be worth the effort), but you're
probably after those that are hard to reproduce and only happen after a
while.

comment:2 Changed 6 years ago by andy

For folks lucky enough to have a debug board, this isn't needed since you can just look in your terminal emulator and see the full backtrace or whatever. But I agree if you don't have that then it is very tough to know what the hell happened. It does not seem we can expect our devices to give access to the debug console in a user friendly way (ie, private USB connector for it) for foreseeable future unfortunately.

In the Linux x86 world there's a trick thing to store stuff in RTC RAM for debugging suspend for example, but it's very ugly.

Definitely this isn't the business of the bootloader to deal with. But I agree about using RAM.

We can use physical address at end of RAM, we could back off the 128MB we tell Linux about a little. Next Linux boot can display it, we can probably expose it to userspace and let initscript handle it, since we only target case where panic could not be seen on original boot due to X for example. If a panic is happening during boot shortly we will display it on LCM by default in Qi case.

It should be pretty certain that you can always boot into backup rootfs / kernel unless something awful has happened, even then you can nuke your SD Card and come back like that with high certainty.

comment:3 Changed 6 years ago by lindi

Have you tested kdump that uses kexec() to start a rescue kernel that can use its drivers to write user and kernel memory space to disk?

comment:4 Changed 6 years ago by andy

Nope. Project seems to be a bit dead?

comment:5 Changed 6 years ago by werner

For example, just a couple of months ago, Fedora considered it pretty
much alive:
http://fedoraproject.org/wiki/Kernel/kdump

The kernel-side functionality is old, so that's why you don't read much
about it these days. The magic is in the user space you use with this.

However, I wonder if kdump may not be a bit too much overkill for our
small devices. Also, a simple buffer in a reserved RAM area would be
more reliable for just keeping messages. kdump was designed for doing
real crash dumps on big servers, with kernel state analysis and all
the bells and whistles.

comment:6 Changed 6 years ago by andy

Hm on the main project page

http://lse.sourceforge.net/kdump/

the last patch they have for Fedora is FC4, the link to the kernel side git stuff is broken, the download dir

http://www.xmission.com/~ebiederm/files/kexec/

stops at support for 2.6.10-rc2...

I guess it means it got in and Fedora look after it, but it's fair to say it looks a bit dead.

comment:7 Changed 6 years ago by lindi

stable-tracking branch has kexec support and it's enabled by default in ./arch/arm/configs/gta02-moredrivers-defconfig so I built the userland package and tried

sudo kexec -l /boot/uImage2.bin --append="`cat /proc/cmdline`"
sudo kexec -e

but just got WSOD (first time I have ever seen WSOD btw!). lsusb does not show the phone as usb device so the phone did not just boot without display working. No leds are blinking or lit.

comment:8 Changed 6 years ago by andy

Wow nice try... WSOD is probably just because of reset action on LCM or Glamo and then nothing came and configured it, so it's not really related to WSOD that is spoken about lately.

I guess it crashed early in new kernel or on leaving old one or whatever, not much room to debug it without a debug board I would think.

comment:9 Changed 6 years ago by lindi

http://docs.openmoko.org/trac/ticket/930 seems to discuss kexec on openmoko and has been marked fixed 14 months ago. I reported what I tried and how it failed (but could not reopen the bug due to lack of permissions).

comment:10 Changed 6 years ago by lindi

In response to #930 mwester told me that I need to use zImage with kexec (tested and that works). This is nice since now I can test new kernels without having to change uboot configuration or touch /boot partition at all.

However, while kexec works on ARM it seems that CONFIG_CRASH_DUMP support is not available on ARM.

comment:11 Changed 6 years ago by lindi

I tried booting with mem=127M and incrementing a byte in memory on every boot

diff --git a/init/main.c b/init/main.c
index 7e117a2..fb2b531 100644
--- a/init/main.c
+++ b/init/main.c
@@ -693,6 +693,17 @@ asmlinkage void __init start_kernel(void)
 
        ftrace_init();
 
+        {
+            char *buf;
+            buf = ioremap(127<<20, 4096);
+            if (buf) {
+                printk("ioremap returned %08x\n", buf);
+                printk("%02x %02x %02x\n", buf[0], buf[1], buf[2]);
+                buf[0]++;
+                iounmap(buf);
+            }
+        }
+
        /* Do the rest non-__init'ed, we're now alive */
        rest_init();
 }

but every time I see zeroes. Any idea what is zeroing the memory?

comment:12 Changed 6 years ago by lindi

I tried mem=120M and writing to 123<<20 just in case but this did not make any difference.

I also tried changing the memory from uboot

GTA02v5 # base 0x30000000
Base Address: 0x30000000
GTA02v5 # md 0x087b00000 4
b7b00000: ffffffff ffffffff ffffffff ffffffff    ................
GTA02v5 # mw 0x087b00000 1 1
GTA02v5 # md 0x087b00000 4
b7b00000: 00000001 ffffffff ffffffff ffffffff    ................

but still only saw zeroes under linux. So either something is zeroing the memory during boot or I am writing/reading a completely wrong memory address?

comment:13 Changed 6 years ago by andy

md 0x087b00000 4

The address has too many zeros in it, maybe that's part of the problem.

comment:14 Changed 6 years ago by lindi

Thanks for pointing that out. I can see bytes written with

mw 0x37a00000 0x01234567 1

using

ioremap(0x30000000 + (122<<20), 4096);

... and it persists over "shutdown -r now"!

comment:15 Changed 6 years ago by lindi

In the spirit of "release early and release often" I'm attaching a prototype that logs all kernel messages to a circular buffer at 127M and dumps them as kernel messages after reboot. Use with mem=127M so that the last megabyte can be safely (ab)used as the circular buffer. I have tested this only on ARM.

Changed 6 years ago by lindi

patch to log kernel messages to circular buffer that persists over reboot

comment:16 Changed 6 years ago by andy

It's cool you got this working :-) I guess you already quite aware of it but we need to decrease the footprint in kernel.c and move all the magic addresses to mach-gta02.c.

It can do with a KConfig to enable it for compile. Rather than mem=, we can eventually screw with the ATAG for valid physical RAM extent in Qi so we don't have to burn a whole megabyte.

Also we could do with protecting at least the metadata struct info with a hash or crc so we know we can trust it on reboot, and some error checking there.

But I realize you only just got the whole cycle working :-) Nice work!

comment:17 Changed 6 years ago by shoragan

This seems to be similar to http://lwn.net/Articles/315943/. It may be easier (to get this code upstream) to use the same infrastructure.

comment:18 Changed 6 years ago by lindi

I upgraded to andy-tracking bc49de906ce5a6e39a4bc9a46ab33af09b93295f and noticed that now the deconfig file has
CONFIG_ANDROID_RAM_CONSOLE -- however, drivers/staging/android/ is not included so I don't know where the actual code is.

comment:19 Changed 6 years ago by arhuaco

  • Priority changed from normal to high
  • Milestone set to stable-kernel-2009.1

comment:20 follow-up: ↓ 21 Changed 6 years ago by lindi

The current shortcoming of ramconsole-1.patch are:
1) It needs mem=127M option
2) panic=20 and/or watchdog is needed so that user can recover the message after reboot
3) It prints contents of the ringbuffer to console on bootup. This can be very slow if the ringbuffer holds the full 1M of data.
4) Often after a crash there is an fsck and after that the system is rebooted. This means the contents of ringbuffer is actually printed twice to console.
5) Since contents of ringbuffer is printed to console using printk it is limited by the size of the printk buffer, sometimes fsck+reboot is enough to add so much data that when syslogd is finally started it can not read the original panic message from /proc/kmsg anymore.
6) Sometimes when the system is so stuck that I need to remove the battery to reboot it the contents of the ringbuffer starts to fade when no RAM refresh is done. This is usually harmless but if it happens to hit the magic word then ramconsole-1.patch will not print the contents of the ringbuffer and there is no way to access it from userland either (I couldn't do it via /dev/mem for some reason).

Advantages are:
1) It works and provides useful info
2) It is quite simple patch that does not modify kernel logging logic much, it merely adds its own ringbuffer in parallel to the normal logging which should reduce the likelyhood that it breaks something important.

comment:21 in reply to: ↑ 20 Changed 6 years ago by arhuaco

Replying to lindi:

The current shortcoming of ramconsole-1.patch are:
1) It needs mem=127M option
2) panic=20 and/or watchdog is needed so that user can recover the message after reboot

I read the thread Andy pointed to us ( http://lkml.org/lkml/2009/1/21/250 ). This is a feature that will help many people but I don't think we have the manpower to think of upstream for this right now. Thus I guess I can modify your patch and send it. We could have a boot parameter (om_ramconsole=N) and we could use 2N for the ramconsole size. It would be disabled by default. Since we would have to enable it manually and we would provide the documentation concern #1 and concern #2 are not really a problem.

comment:22 Changed 6 years ago by lindi

Inspired by werner's poke.c I wrote a userland tool to dump the contents of the ramconsole via /dev/mem -- http://iki.fi/lindi/openmoko/ramconsole-dump.c

I then modified the ramconsole patch so that it does not need to print the contents of the ramconsole on bootup. This fixes issues #3, #4, #5 and #6. (#6 is fixed since now the policy to discard corrupted ramconsole header is in userland and can be changed if somebody needs to recover partially corrupted data manually).

Changed 6 years ago by lindi

new version, do not print contents of ramconsole on boot, let userland handle it

Note: See TracTickets for help on using tickets.