Linux driver development on a Raspberry Pi

I recently started exploring linux drivers more in-depth, and more specifically I got interested in the practice of cross-compiling them for a different platform (such as the Raspberry Pi). Even though I could develop drivers directly on the RPI, which would be easier since I would be directly compiling against the kernel that's currently running on the RPI, I think cross-compiling is more fun, and will help us learn a few things along the way.

A simple driver

First things first, let's write a basic "hello world" driver for our desktop computer, and build it against the desktop's kernel first:

hello_world.c

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

#include <linux/module.h>


static int __init hello_world_init (void)
{
    pr_info("Hello, World!\n");

    return 0;
}


static void __exit hello_world_exit (void)
{
    pr_info("Goodbye, World!\n");
}


module_init(hello_world_init);
module_exit(hello_world_exit);

MODULE_LICENSE("GPL");
MODULE_AUTHOR("Dimitri Kokkonis");
MODULE_DESCRIPTION("A simple hello world kernel module");

Here's a basic breakdown of this driver:

• line 1: we import the linux module header file, because this is a kernel module; any calls to userland functions (such as printf) make no sense in the kernel.
• lines 4-9: the initialization entry point. We have the static keyword because this function should only be defined and used in this file, and __init because it's good practice (even though this is a dynamic module so the __init directive will have no effect). We then just print a "hello world" message and return 0, like a classic main "hello world" function would.
• lines 12-15: the de-initialization entry point. Similarly, static and __exit are used; the latter is needed since dynamic modules can be unloaded/removed (but again it is good practice to always have these directives). Once more, we print a message using kernel functions.
• lines 18-19: assignment of the entry points using kernel macros module_init and module_exit.
• lines 21-23: various kernel module info (not really necessary in this case but might as well showcase them).

This driver should simply print "Hello, World!" when inserted into the kernel, and "Goodbye, World!" when removed from it.

Compiling for desktop

Let's first test this driver on the desktop (a x86 running Ubuntu in my case). In order to do so, we need to compile it against the desktop's running kernel; but before doing that, we need to add a Makefile in the same directory as our driver in order to mark the module as being dynamic:

Makefile

obj-m := hello_world.o

We can now compile it against the current kernel, to which we will point using /lib/modules/<kernel version>/build/. You can obtain your kernel version by running uname -r; in my case, that path is /lib/modules/5.4.0-52-generic/build:

$ sudo make -C /lib/modules/5.4.0-52-generic/build M=$PWD modules

The compilation should work correctly (if not, make sure both the Makefile as well as hello_world.c are in the same directory).

Now we can try to insert our module into the kernel:

$ sudo insmod hello_world.ko

If errors are produced during this step, you have most likely compiled against the wrong kernel version. If, on the other hand, there are no errors, you can verify that your module works by running dmesg:

$ dmesg
...
[ 7667.013583] Hello, World!

And to verify that the exit function of the module works as well, we can remove it and run dmesg again:

$ sudo rmmod hello_world.ko
$ dmesg
...
[ 7667.013583] Hello, World!
[ 7672.605206] Goodbye, World!

Finally, let's flesh out the Makefile to avoid compiling "by hand":

Makefile

obj-m := hello_world.o

all: x86_build

clean: x86_clean

x86_build: hello_world.c
    sudo make -C /lib/modules/5.4.0-52-generic/build M=$(PWD) modules

x86_clean:
    sudo make -C /lib/modules/5.4.0-52-generic/build M=$(PWD) clean


.PHONY: all clean x86_build x86_clean

Now we can just run make and make clean to build and cleanup.

Compiling for RPI

In order to cross-compile for the RPI, we need two things: the RPI's specific kernel to compile against, and the appropriate toolchain (compiler, linker etc). But first, we need to know the version of the kernel; to do that, we can connect to the RPI by ssh (I believe the easiest method to turn on ssh in a headless RPI is to just create an empty file called 'ssh' in the boot partition of the RPI). Here's a quick one-liner that allows you to find your RPI, assuming ssh is enabled:

$ sudo nmap -sS -p 22 <local_address>/24

The <local_address> can be found using ifconfig (mine is 192.168.1.20). You can then hopefully see your RPI:

$ sudo nmap -sS -p 22 192.168.1.20/24
...
Nmap scan report for 192.168.1.7
Host is up (0.10s latency).

PORT   STATE SERVICE
22/tcp open  ssh
MAC Address: XX:XX:XX:XX:XX:XX (Raspberry Pi Foundation)

You can then connect to it (there's probably a default pi user):

$ ssh pi@192.168.1.7

In order to get the kernel version, once again we'll run uname -r. After that, we need to go to the official RPI linux kernel repo and clone the appropriate branch, based on the kernel version.

In my case, I updated the kernel version of my RPI to the current "main" one of the repo (as of the time of writing), which is 5.4.y. So I just need to clone the repo and remain on the "main" branch:

$ git clone https://github.com/raspberrypi/linux --depth=1

Once that is done, we need to first compile the kernel in order to be able to cross-compile drivers using it. Also some dependencies are needed, so make sure you have them:

$ sudo apt install git bc bison flex libssl-dev make libc6-dev libncurses5-dev

Furthermore, we need a cross-compilation toolchain as mentioned above. My RPI is running on armv7 architecture, which is 32 bit; I thus need the 32-bit version of the toolchain:

$ sudo apt install crossbuild-essential-armhf

If you have a 64-bit architecture, you need the 64-bit version:

$ sudo apt install crossbuild-essential-arm64

Now, we can finally build the kernel. Let's cd into the source directory and clean up everything just in case:

$ cd linux/
$ make mrproper

Then we can configure the kernel for compilation (I have an RPI 3B+, so I'll run the appropriate command):

$ # ---- 32-BIT CONFIGS ----
$ #
$ # RPI 1, Zero, Zero W or Compute Module:
$ # make KERNEL=kernel ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- bcmrpi_defconfig
$ #
$ # RPI 2, 3, 3+ or Compute Module 3:
$ # make KERNEL=kernel7 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- bcm2709_defconfig
$ #
$ # RPI 4:
$ # make KERNEL=kernel7l ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- bcm2711_defconfig
$ #
$ #
$ # ---- 64-BIT CONFIGS ----
$ # RPI 2, 3, 3+ or Compute Module 3:
$ # make KERNEL=kernel8 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bcmrpi3_defconfig
$ #
$ # RPI 4:
$ # make KERNEL=kernel8 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bcm2711_defconfig
$ #
$ #
$ # I have an RPI 3B+, so I'll run this one:
$ make KERNEL=kernel7 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- bcm2709_defconfig
  HOSTCC  scripts/basic/fixdep
  HOSTCC  scripts/kconfig/conf.o
  HOSTCC  scripts/kconfig/confdata.o
  HOSTCC  scripts/kconfig/expr.o
  LEX     scripts/kconfig/lexer.lex.c
  YACC    scripts/kconfig/parser.tab.[ch]
  HOSTCC  scripts/kconfig/lexer.lex.o
  HOSTCC  scripts/kconfig/parser.tab.o
  HOSTCC  scripts/kconfig/preprocess.o
  HOSTCC  scripts/kconfig/symbol.o
  HOSTLD  scripts/kconfig/conf
#
# configuration written to .config
#

Once the config is generated, we can compile the kernel. It is highly recommended to add the -j n option, where n is roughly the number of processors in your computer. You can play around with it though and try greater values; it refers to the number of maximum concurrent jobs the compiler can launch. It is highly recommended because compiling the kernel takes a long time, so might as well parallelize as much as possible.

$ # 32-bit version:
$ # make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- zImage modules dtbs
$ #
$ # 64-bit version:
$ # make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- Image modules dtbs
$ #
$ # I'll run the first one since I have a 32-bit machine:
$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- zImage modules dtbs -j 5

Once the kernel is done compiling, we can finally build our module for the RPI! Let's add the rules needed in the Makefile we made previously in order to accomplish that:

Makefile

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

obj-m := hello_world.o

all: x86_build

clean: x86_clean RPI_clean

x86_build: hello_world.c
    sudo make -C /lib/modules/5.4.0-52-generic/build M=$(PWD) modules

x86_clean:
    sudo make -C /lib/modules/5.4.0-52-generic/build M=$(PWD) clean

RPI_build: hello_world.c
    sudo make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- -C $(PWD)/linux M=$(PWD) modules

RPI_clean:
    sudo make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- -C $(PWD)/linux M=$(PWD) clean


.PHONY: all clean x86_build x86_clean RPI_build RPI_clean

We should now be able to run make RPI_build to cross-compile our driver. If a hello_world.ko file is produced without errors, we can transfer it over to the RPI to test it on the actual machine:

$ scp hello_world.ko pi@192.168.1.7:~/

We can then ssh into the RPI and load/unload the module to check that it's working correctly:

$ ssh pi@192.168.1.7
$ sudo insmod hello_world.ko
$ sudo rmmod hello_world.ko
$ dmesg
...
[ 2063.017314] Hello, World!
[ 2070.706195] Goodbye, World!

And just like that, we've written, cross-compiled and tested a linux driver for an RPI!