Linux Kernel Panics on FW2B/FW4B with Coreboot <= 184.108.40.206
KB Article Updates
June 2, 2021
- Fixed typos stating that the issue exists for coreboot branch 4.12.0.X, when in fact the issue has been identified on the 4.9.0.X branch.
- New coreboot version 220.127.116.11 was released for the FW2B that fixes the HPET issue at the BIOS level. No update for FW4B at this time.
- Added new article section on how to update to 18.104.22.168 on the FW2B.
Some FW2B and FW4B CPUs have issues booting the Linux kernel from coreboot. This is due to how Linux tries to setup a high-precision internal timer and how coreboot exposes said timer. By passing a kernel parameter of
hpet=disable, you can instruct the Linux kernel to not try to use an HPET timer and default to an alternative.
Attempting to use some Linux kernels with an FW2B or FW4B device with coreboot 22.214.171.124 will result in the inability to boot due to a kernel panic when attempting to set up clock timing and IO-APIC. This is due to some inconsistencies seen in the Intel Braswell CPU platform employed in some Protectli devices.
This bug may affect devices that meet all of the following criteria
- The device is an FW2B or FW4B
- The device BIOS is coreboot version 126.96.36.199 or lower
- The kernel used is Linux-based (Debian, Ubuntu, etc.)
An affected unit is difficult to identify, as the origin of the issue is not entirely known. Protectli have seen this issue only with some Intel Braswell CPUs, and less than 10% of our tested devices suffer from this issue. To identify the issue, simply attempting to boot an OS using the Linux kernel will produce immediate failure like the following:
[ 1.042672] ACPI: Core revision 20180810
[ 1.046907] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 133484882848 ns
[ 1.056149] APIC: Switch to symmetric I/O mode setup
[ 1.062662] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[ 1.168761] ..MP-BIOS bug: 8254 timer not connected to IO-APIC
[ 1.174670] ...trying to set up timer (IRQ0) through the 8259A ...
[ 1.180923] ..... (found apic 0 pin 2) ...
[ 1.285109] ....... failed.
[ 1.287978] ...trying to set up timer as Virtual Wire IRQ...
[ 1.393720] ..... failed.
[ 1.396416] ...trying to set up timer as ExtINT IRQ...
[ 1.629867] ..... failed :(.
[ 1.632825] Kernel panic - not syncing: IO-APIC + timer doesn't work! Boot with apic=debug and send a report. Then try booting with the 'noapic' option.
[ 1.648228] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.19.0-16-amd64 #1 Debian 4.19.181-1
[ 1.656581] Hardware name: Protectli FW2B/FW2B, BIOS v188.8.131.52 03/25/2019
[ 1.663360] Call Trace:
[ 1.665897] dump_stack+0x66/0x81
[ 1.669290] panic+0xe7/0x24a
[ 1.672335] setup_IO_APIC+0x761/0x846
[ 1.676161] ? clear_IO_APIC_pin+0xb8/0x110
[ 1.680419] x86_late_time_init+0x17/0x1c
[ 1.684503] start_kernel+0x460/0x52a
[ 1.688244] secondary_startup_64+0xa4/0xb0
[ 1.692514] ---[ end Kernel panic - not syncing: IO-APIC + timer doesn't work! Boot with apic=debug and send a report. Then try booting with the 'noapic' option.
[ 1.692514] ]---
If you have an FW2B you may simply upgrade to coreboot 184.108.40.206 which fixes the HPET issue at the BIOS level. If you do not wish to upgrade to newer coreboot BIOS, then you may follow the below instructions to edit your Vault’s bootloader to instruct the Linux kernel to avoid using HPET for timing.
Option 1: Upgrade to coreboot 220.127.116.11 on the FW2B
This resolution was added on June 2, 2021.
Follow the instructions listed on our article How to Use Flashli and flash the coreboot 18.104.22.168 BIOS.
Option 2: Edit Kernel Command Line Options on the Bootloader
Resolving this issue involves editing your bootloader configuration to pass a parameter to the Linux kernel advising to not use
hpet for timing. This is achieved by passing the following parameter to the command line:
hpet=disable. Some examples follow for common bootloaders.
GRUB from the affected system
- At the boot screen, press
eto edit the console commands
- Add the following parameter to the end of the command line:
- Save and boot.
GRUB from a different system
If you had a stable experience with your Vault device, but then flashed an affected coreboot BIOS, you can remove the hard disk from an affected machine and mount it elsewhere to apply the fix.
- Unscrew the bottom plate of your Vault and remove the mSATA device.
- Place the mSATA device in a known, working system that also has Linux. For example, you can boot into a Live Ubuntu USB OS and attach the mSATA by a USB interface.
- Launch a terminal on the Live OS.
- Mount the disk if it is not already mounted. In this example we use
sdX1as the affected mSATA, but be sure to replace it with the proper device and partition.
sudo mkdir /mnt/msata
sudo mount /dev/sdX1 /mnt/msata
- Use chroot to change the apparent root directory to /mnt/msata
cd /mnt/msata && sudo chroot .
- Add the parameter to the boot file. This step may vary based on your operating system. If this fix does not work, it’s most likely because grub is managed differently for your system. This example is assuming Debian.
echo 'hpet=disable' | sudo tee -a /etc/default/grub
- Update the GRUB bootloader with your new configuration
- Exit chroot and unmount the device
sudo umount /mnt/msata
sudo rmdir /mnt/msata
- Remove the mSATA from the system and reassemble your Vault. Test your results by booting the machine.
We are working to have this issue fixed in a future coreboot release. When the new coreboot version is available, you may refer to our Knowledgebase again for instructions on how to flash the new BIOS. We do not anticipate needing to “undo” the fix explained in this article after flashing the new coreboot version.