Kernel Hacking + Device Bringup on NixOS
2024/11/16
one of my NixOS devices is a PinePhone Pro. at time of writing, it boots to a working display/touchscreen and even WiFi on stock NixOS, however other basic features like audio and battery readouts aren't available. actually, these features are available on distros like postmarketOS which ship kernel forks specifically for this. but maintaining a fork is at least a bit of work, and NixOS gives you lots better ways to tweak your kernel than simply building the whole thing from a different source tree.
so, how best to develop and deploy kernel-level changes on a NixOS system?
Kernel Pain Points
skip ahead if you're already familiar with the NixOS kernel options. the straightforward way to ship custom kernels on NixOS is something like:
{ pkgs, ... }:
{
boot.kernelPackages = pkgs.linuxPackagesFor (pkgs.buildLinux {
src = fetchgit {
# ...
};
# ...
});
}
or if your kernel patches are simple enough, then track the NixOS kernel and patch it instead:
{ ... }:
{
boot.kernelPatches = [
{
name = "my-kernel-patch";
patch = ./my-kernel-patch.patch;
}
];
}
but the crushing downside to this approach is that tweaking just a single line in my-kernel-patch.patch forces
a complete rebuild of the kernel: your iteration cycle time might well be 30 minutes.
Faster Kernel Iterations
until we get something like dynamic derivations, the path out of costly iteration is to ship whichever tweaks you're pursuing via some other mechanism than as part of the top-level kernel derivation. the kernel is fairly pluggable; consider:
- device tree files can be loaded at runtime.
- kernel modules can be loaded at runtime.
with the right config, either of these things can be freely modified without forcing a kernel build.
Shipping Device Tree Patches
Linux uses device tree files to determine which peripherals are available on your device and how to interface with them. it's more relevant for embedded devices than for traditional x86 machines.
the typical boot flow for an embedded device is that the bootloader (U-Boot) knows the name of
the device it's running on (perhaps because it was built specifically for that device) and communicates
this to the kernel when it hands over control of the device.
the kernel ships with device tree definitions for hundreds of different devices,
each with a name like samsung,exynos7-espresso or pine64,pinephone-pro,
and after learning which device it's actually running on it applies the appropriate device tree,
instantiating and configuring device drivers accordingly.
device trees are composable by design; the spec defines the concept of an "overlay",
such that the actual device tree to apply to a device should be the union of whatever's
defined in the kernel source tree plus any overlays that apply to the same device.
NixOS makes this feature available via the hardware.deviceTree.overlays option,
and a certain amount of our kernel changes can be expressed via this option instead of costly .patch files.
for example, the stock kernel misidentifies volume-down key presses as volume-up events on the PinePhone Pro. distros like postmarketOS fix this by patching the device tree file like so:
--- arch/arm64/boot/dts/rockchip/rk3399-pinephone-pro.dts
+++ arch/arm64/boot/dts/rockchip/rk3399-pinephone-pro.dts
@@ -48,11 +48,11 @@ / {
adc-keys {
compatible = "adc-keys";
io-channels = <&saradc 1>;
io-channel-names = "buttons";
keyup-threshold-microvolt = <1600000>;
poll-interval = <100>;
button-down {
label = "Volume Down";
linux,code = <KEY_VOLUMEDOWN>;
- press-threshold-microvolt = <600000>;
+ press-threshold-microvolt = <400000>;
};
this patch can alternately be represented as a Device Tree Overlay (DTO):
// file: rk3399-pinephone-pro-lradc-fix.dtso
// boilerplate required by the device tree compiler.
/dts-v1/;
/plugin/;
// instruct the system to only apply this overlay
// to PinePhone Pro, and not any other devices.
/ {
compatible = "pine64,pinephone-pro";
};
// address the parent node we want to patch,
// and then inject a new property value.
// this overrides `press-threshold-microvolt`
// while leaving all other properties unchanged.
&{/adc-keys/button-down} {
press-threshold-microvolt = <400000>;
};
and then we can ship it in NixOS like so:
{ ... }:
{
hardware.deviceTree.overlays = [
{
name = "rk3399-pinephone-pro-lradc-fix";
dtsFile = ./rk3399-pinephone-pro-lradc-fix.dtso;
}
];
}
after adding the above to a stock NixOS config, and deploying to a PinePhone Pro, the volume down button should now be fixed!
Shipping Custom Kernel Modules
one of the less trivial patches yet to be mainlined is support for
battery monitoring on the PinePhone Pro (i.e. viewing the charge level
and the charge/discharge rate). this feature is supported
in downstream kernels by shipping two new kernel modules:
rk818_battery and rk818_charger.
these modules are device drivers: they detect a rk818-battery
device somewhere on the system, and then run code in response
to that (namely, enable some voltage regulators, and create sysfs
nodes to let userspace interact with the device).
code wise, the typical driver is a single .c file with few if any direct dependencies on other drivers. most driver modules are standalone in the same sense that each nix package is standalone.
then, we can ship new kernel modules by:
- defining a nix derivation that compiles the kernel module.
- telling nix to deploy that on the kernel's module search path.
- instructing the kernel to load our module.
- shipping a device tree overlay to associate the relevant devices with the driver our module provides.
Building a Kernel Module
my kernel module has 3 files, which live in my nix repo inline:
these are compiled into rk818_battery.ko and rk818_charger.ko by a Makefile:
obj-m := rk818_battery.o rk818_charger.o
all:
$(MAKE) -C "$(KERNEL_DIR)" M="$(PWD)" modules
install:
install -Dm444 rk818_battery.ko \
$(INSTALL_MOD_PATH)/drivers/power/supply/rk818_battery.ko
install -Dm444 rk818_charger.ko \
$(INSTALL_MOD_PATH)/drivers/power/supply/rk818_charger.ko
this can be packaged for nix as so:
# file: pkgs/linux-packages/rk818-charger/default.nix
{
buildPackages,
kernel,
lib,
stdenv,
}: stdenv.mkDerivation {
pname = "rk818-charger";
version = "0-unstable-2024-10-01";
src = ./.;
hardeningDisable = [ "pic" ];
nativeBuildInputs = kernel.moduleBuildDependencies;
makeFlags = [
"KERNEL_DIR=${kernel.dev}/lib/modules/${kernel.modDirVersion}/build"
"INSTALL_MOD_PATH=$(out)/lib/modules/${kernel.modDirVersion}/kernel"
# from pkgs/os-specific/linux/kernel/manual-config.nix:
"O=$(buildRoot)"
"CC=${stdenv.cc}/bin/${stdenv.cc.targetPrefix}cc"
"HOSTCC=${buildPackages.stdenv.cc}/bin/${buildPackages.stdenv.cc.targetPrefix}cc"
"HOSTLD=${buildPackages.stdenv.cc.bintools}/bin/${buildPackages.stdenv.cc.targetPrefix}ld"
"ARCH=${stdenv.hostPlatform.linuxArch}"
] ++ lib.optionals (stdenv.hostPlatform != stdenv.buildPlatform) [
"CROSS_COMPILE=${stdenv.cc.targetPrefix}"
];
# NixOS kernel expects compressed kernel modules, so do that here.
postInstall = ''
find $out -name '*.ko' -exec xz {} \;
'';
}
then add this package to nixpkgs' linuxKernel package set:
{ ... }:
{
nixpkgs.overlays = [(self: super: {
linuxKernel = super.linuxKernel // {
packagesFor = kernel:
(
super.linuxKernel.packagesFor kernel
).extend (kFinal: kPrev: with kFinal; {
rk818-charger = callPackage ./pkgs/linux-packages/rk818-charger { };
});
};
})];
}
kernel modules must be built against the same kernel version
that they'll be loaded by: this linuxKernel package set looks
funny at first, but it exists to make that easier.
your kernel module can be built like this:
nix-build -A linuxPackages_6_11.rk818-chargernix-build -A linuxPackages_6_10.rk818-charger
and so on, depending on which kernel you're running.
Deploying a Kernel Module
add the following to your NixOS config:
{ config, ... }:
{
# this builds our modules against the specific kernel in use by the host
# and makes them available under /run/current-system/kernel-modules
# where they can be found by tools like `modprobe`.
boot.extraModulePackages = [
config.boot.kernelPackages.rk818-charger
];
# explicitly load the modules during stage-2 boot.
# if they're needed earlier in the boot process,
# then use `boot.initrd.kernelModules` instead.
boot.kernelModules = [
"rk818_battery"
"rk818_charger"
];
}
now we've recreated the same behavior as if the kernel module had been shipped in-tree, but we still have to associate the driver with a specific device if we want it to do anything.
Device Tree + Module Integration
add the following Device Tree Overlay to hardware.deviceTree.overlays,
as we did earlier with the battery DTO:
/dts-v1/;
/plugin/;
// apply the overlay only to PinePhone Pro; no other devices.
/ {
compatible = "pine64,pinephone-pro";
};
&{/} {
bat: battery {
compatible = "simple-battery";
voltage-min-design-microvolt = <3400000>;
voltage-max-design-microvolt = <4350000>;
};
};
&rk818 {
battery {
// this line associates this device with the rk818-battery module
// we just shipped
compatible = "rockchip,rk818-battery";
// constants were taken from the pine64-org kernel tree:
ocv_table = <3400 3675 3689 3716 3740 3756 3768 3780
3793 3807 3827 3853 3896 3937 3974 4007 4066
4110 4161 4217 4308>;
design_capacity = <2916>;
design_qmax = <2708>;
bat_res = <150>;
max_input_current = <3000>;
max_chrg_current = <2000>;
max_chrg_voltage = <4350>;
sleep_enter_current = <300>;
sleep_exit_current = <300>;
power_off_thresd = <3400>;
zero_algorithm_vol = <3950>;
fb_temperature = <105>;
sample_res = <10>;
max_soc_offset = <60>;
energy_mode = <0>;
monitor_sec = <5>;
virtual_power = <0>;
power_dc2otg = <0>;
otg5v_suspend_enable = <1>;
};
charger {
// this line associates this device with the rk818-charger module
// we just shipped
compatible = "rockchip,rk818-charger";
monitored-battery = <&bat>;
};
};
Patching an Upstream Kernel Module
this all works, however the PinePhone Pro battery integration
requires not just a new kernel module, but also to patch an existing
kernel module (rk8xx-i2c). since the kernel module is defined in-tree, the natural
way to do that would force a kernel rebuild every time we adjust the patches.
we can build our patched module out-of-tree using the same method
we used to build a wholly new module out-of-tree (write the derivation
as some patches atop src = linux-latest, or forget about tracking
patches and just copy the entire module source into our repo).
then we just configure the kernel to prefer our module over its in-tree module.
a first approach might be to configure the kernel with CONFIG_MFD_RK8XX_I2C=n,
then ship our module as above. this works:
{ lib, ... }:
{
boot.kernelPatches = [
{
name = "rk8xx-i2c-out-of-tree";
patch = null;
extraStructuredConfig = with lib.kernel; {
MFD_RK8XX_I2C = no;
};
}
];
}
this will result in one kernel rebuild, and then you can freely
edit your out-of-tree rk8xx-i2c module without costly rebuilds.
if you find yourself patching a lot of modules, then it may be
preferable to simply build all in-tree modules as dynamic modules,
and configure out-of-tree modules to take precedence over the in-tree ones.
remove the MFD_RK8XX_I2C = no patch and replace it with this:
{ ... }:
{
nixpkgs.config.hostPlatform.linux-kernel = {
# build every module known to the mainline kernel
autoModules = true;
# and build them as dynamically loaded modules (`=m`), not builtins (`=y`)
preferBuiltin = false;
};
# default nixos behavior is to error if a kernel module is provided by more
# than one package. but we're doing that intentionally, so patch the logic
# from <nixos/modules/system/boot/kernel.nix> (AKA pkgs.aggregateModules)
# to not complain.
system.modulesTree = lib.mkForce [(
(pkgs.aggregateModules
(config.boot.extraModulePackages ++ [ config.boot.kernelPackages.kernel ])
).overrideAttrs {
# when ignoring collisions, order becomes important:
# earlier items (config.boot.extraModulePackages) override later items
# (config.boot.kernelPackages.kernel).
ignoreCollisions = true;
}
)];
}
more examples for this method can be found on the nixos wiki.
Conclusion
and there you have it! several techniques for working with device trees and kernel modules without suffering lengthy builds. if you found anything here useful, then my request to you is to upstream your kernel work so that the next reader of this blog doesn't have to go through the same pain 😛
complete nix expressions for my PinePhone Pro system can be found here in case i missed some crucial detail in this writeup.