Compiling the Linux kernel is fun! uClinux is a version of regular desktop Linux that’s designed for low-end embedded hardware, especially CPU’s that lack a memory management unit and can’t support virtual memory. It looks like a good fit for my 68000 single board computer project, so I’ve begun getting familiar with the uClinux source code and how to build it. My plan is to first build a uClinux image for some similar 68000-based system that’s already supported in the kernel tree, and then modify the source code as necessary to support my custom board.

In theory the process should go like this: download the uClinux source code, run a config program to choose the hardware target and kernel options that you’d like, then compile the code to produce a finished kernel image. And that’s true, but it skips over a huge number of details and gotchas, as I discovered along the way. The real challenge was finding an operating system, version of the uClinux source, version of the compiler toolchain, and version of the standard C library that all worked together. That proved to be much more challenging than I would have ever imagined!

 
TL;DNR

Here’s what finally worked for me for two existing 68K-based boards.

Arcturus uCsimm, 3.x kernel – Ubuntu 14.04 64-bit with libncurses5-dev package installed, 20140504 full source distribution of uClinux, 20130217 m68k-uclinux precompiled toolchain, uClibc selected as the C standard library.
Weiss SM2010, 2.0.x kernel – Ubuntu 12.04 32-bit with libncurses5-dev package installed, 20040218 full source distribution of uClinux, 20030314 m68k-elf precompiled toolchain, uC-libc selected as the C standard library.

If you don’t care much about build details, but are interested in how the compiled kernel boots on 68K hardware, skip the sections below and jump to the end.

 
Host OS

Step 1 is setting up a build computer with a Linux operating system. I chose to use Ubuntu, a popular and well-supported Linux distribution. But what version of Ubuntu? As I later discovered, the Ubuntu version number wasn’t very important, but the choice of 32-bit or 64-bit Ubuntu was critical. If you anticipate using pre-compiled 64-bit toolchain binaries (described further below), you must use 64-bit Ubuntu. If you anticipate using pre-compiled 32-bit toolchain binaries, you should use 32-bit Ubuntu. All but one of the precompiled toolchains I found were 32-bit binaries, so I recommend using 32-bit Ubuntu. This is sometimes also called x86, and 64-bit is sometimes called x86_64 or amd64.

Supposedly it’s possible to run 32-bit binaries on 64-bit Ubuntu, but I was never able to get it to work. The way it fails is almost comical, too. Say you’re running 64-bit Ubuntu, and you’ve got a program binary called tool in your current directory. Unknown to you, the program was compiled as a 32-bit binary. When you attempt to run the program at the command prompt, you’ll see this:

user@ubuntu$ ./tool
bash: ./tool: No such file or directory

What do you mean, no such file or directory? It’s right there!

Attempting to run a 64-bit binary on 32-bit Ubuntu generates a slightly more useful error message “cannot execute binary file”. When in doubt, the file command (type file <filename>) will tell you if a program binary is 32-bit or 64-bit.

If you don’t have a spare PC ready to dedicate to an Ubuntu install, VirtualBox is a convenient way to run Ubuntu as a virtual machine within a Windows environment. virtualboxes.org has many pre-built OS images for Virtual Box, including many for Ubuntu, which will save you the trouble of going through the Ubuntu install process in the VM.

 
uClinux Source Code

The source code for uClinux can be downloaded here. Should you just grab the newest version? Maybe not, if you’re targeting a specific board or a custom board with very limited resources. The latest version of the source (currently 20140504) has dropped support for building 2.0.x kernels, and only 2.4.x or 3.x kernels are supported. The configs for some useful older boards (useful to 68K hackers anyway) like the Weiss SM2010 were never updated after kernel 2.0.x, so kernels for those boards can’t be built from the latest uClinux source.

Even if the board you’re interested in is supported in newer kernels, you may wish to build an older kernel to reduce ROM and RAM requirements. Using the latest uClinux source, an image for the Arcturus uCsimm board using the 3.x kernel was 3 MB, but an image for the same board using the 2.4.x kernel was only 1.7 MB. That’s a huge savings. And using a 10 year old version of the uClinux source, an image for the Weiss SM2010 using the 2.0.x kernel was only 870K! Hello, bloat.

There’s a hidden gotcha in the uClinux source distributions: the included user applications (telnet, busybox, etc) and standard C libraries continue to evolve, while the actual kernel source code and board configs don’t. Over time this can break things without anyone noticing. For example, I tried to build the Weiss SM2010 2.0.x kernel image, using the 20070717 source distribution of uClinux. This failed during compilation of busybox, with a “dereferencing pointer to incomplete type” error in ping.c. This was probably due to me using an older toolchain from 2003, which was necessary to compile the 2001-vintage kernel 2.0.x source. When I switched to the 20040218 source distribution of uClinux, the ping.c error disappeared.

For best results, the source distribution of uClinux and the precompiled toolchain should be from the same year as the kernel source you’re building, and the board config files. If you want to build 2.0.x kernels, then use old uClinux source distributions from back when 2.0.x was semi-current – 2001 to 2004 or thereabouts. Failure to do this can lead to all sorts of crazy compile errors, as I discovered.

The uClinux source distributions come as .gz or .bz2 files. To uncompress them, copy them to your Ubuntu home directory, and then type

tar xvf uClinux-dist-YYYYMMDD.tar.bz2

or

tar xzvf uClinux-dist-YYYYMMDD.tar.gz

 
Toolchain

The toolchain contains required tools like the 68K cross-compiler, and elf2flt conversion tool. You might think these would be included in source form with the uClinux source distribution, but that’s not how it works: you need to provide your own toolchain. This page has links to many precompiled toolchain versions for uClinux with 68K targets. If possible, use a toolchain from the same year as the source code you’re compiling.

Somewhere along the way, the uClinux 68K tools appear to have all been renamed from m68k-elf-* to m68k-uclinux-*. For example, the compiler in newer toolchains is m68k-uclinux-gcc, and in older ones it’s m68k-elf-gcc. If you installed the toolchain, but your kernel compile fails because it can’t even find the compiler, this is probably the reason. Install a newer or older version of the toolchain to get a compiler with the right name, and avoid a pile of confusing compilation errors.

As mentioned in the introduction, you also need to make sure the precompiled binaries in the toolchain match the requirements of your OS. 32-bit Ubuntu needs 32-bit toolchain binaries. It should be possible to compile the toolchain itself from its source code, but I found very little documentation about this and didn’t try it.

Precompiled toolchains are typically distributed as shell files with a compressed payload appended. To install them, you copy them to the Ubuntu / (root) directory, and then type:

sudo sh ./m68k-uclinux-tools-YYYYMMDD.sh

This installs the tools under /usr/local. Most toolchains were easy to install, but I ran into a problem with the 20030314 version of the toolchain, and had to do a little script hacking. The script uses tail to pipe its compressed payload to gunzip, but it appears to have been written for an older/different version of tail than what’s included in Ubuntu 12.04. It’s missing the -n flag needed to tell tail how many lines to skip, so it interprets the line count as the filename and complains:

tail: cannot open '+43' for reading: No such file or directory

I tried to edit the script, but the editor screwed up the binary data in the payload. Finally I just unpacked the data myself without running the script at all:

user@ubuntu$ tail -n +43 m68k-elf-tools-20030314.sh | gunzip > tools.tar
user@ubuntu$ tar xvf tools.tar

 
Standard C Library

The standard C library is a small bit of code that implements functions like strcpy(). Under uClinux, there are three choices for the standard C library available in the config menu: uClibc, uC-libc, and glibc. Yes, there are two entirely different libraries whose names differ only by a hyphen. As I understand it, uC-libc was the original standard C library for uClinux, but has now been mostly replaced by uClibc.

If building a modern kernel, you most likely want to select uClibc, but for older kernels it’s unclear. I tried building the Weiss SM2010 2.0.x kernel with uClibc, but it eventually failed during compilation of __assert.c:

cc1: error: unrecognized command line option "-mlittle-endian"

All the references I could find to this error on the web were related to ARM compilers. After some digging, I discovered that the Weiss SM2010 configs were just totally broken with respect to uClibc: vendors/Weiss/SM2010/config.uClibc was set up for an ARM target, not 68K! The original authors clearly never used uClibc then, and that file was probably just copied from some other board config. I switched to using uC-libc, and everything built without errors.

 
Building It All

Once you’ve got the uClinux source and the toolchain installed, it’s time to configure the build options. Cd into the directory containing uClinux source and type make menuconfig. If you’re lucky, after a few moments you’ll see a terminal-based menu program. If you’re not lucky, make menuconfig will fail with a cryptic-looking error.

The most likely failure is an error about curses.h not being found. This is because you’re missing the curses library, needed for fancy terminal graphics and cursor control. To install it, type sudo apt-get install libncurses5-dev. If you’re building xconfig instead of menuconfig, you may also be missing gtk+ (sudo apt get install gtk+2.0-dev) and libglade (sudo apt-get install libglade2-dev).

If you’re using the latest version the uClinux source, and running on 32-bit Ubuntu, you’ll also fail with a linker error about 64-bit incompatible linkage types:

/usr/bin/ld: i386:x86-64 architecture of input file 'zconf.tab.o' is incompatible with i386 output

This is because the person who prepared the uClinux source distribution accidentally left in some 64-bit .o files in the config directory. Delete the offending .o files in config/kconfig, type make menuconfig again, and this time it should work.

The config menu is used to select the vendor and board to target, the C standard library to use, the user space programs to compile, and other options. Make your choices here, then exit. This will save a config file to be used during kernel compilation.

If you’re targeting a pre-3.x kernel, now type make dep. This isn’t necessary for 3.x kernels.

Finally, type make to start the build process. This will run for a long time, filling your screen with thousands of lines of compiler messages. If you’re successful, an image file named something like image.bin will be produced in the images/ subdirectory – this is the actual file that should be downloaded/bootloaded/flashed to the target board. If you’re unsuccessful, compilation will fail somewhere, and you’ll spend hours troubleshooting your source distribution, toolchain, and build settings.

The first target I tried was the Arcturus uCsimm board with a 3.x kernel. The build always failed with a linker error while linking the telnet daemon, telnetd:

/usr/local/m68k-uclinux/bin/ld.real: cannot find -lutil

I think this has something to do with the standard C library, but I never figured it out. I disabled compilation of telnetd using menuconfig. I also got linker errors related to something called libmpc:

/usr/local/libexec/gcc/m68k-uclinux/4.5.1/cc1: error while loading shared libraries: libmpc.so.2:

I thought this was due to a missing package. However, there is no libmpc2 package available for Ubuntu 64-bit. I was finally successful by installing libmpc3, and creating a symlink from libmpc.so.2 to libmpc.so.3. That feels evil.

The uCsimm build always concludes with the error:

cp: cannot create regular file '/tftpboot'

Looking at the Makefile, this error can be safely ignored.

The second target I tried was the Weiss SM2010 with a 2.0.x kernel. This one took a while to find the right combination of OS, uClinux source, toolchain, and C standard library as described above. But once those were in place, it built cleanly with no errors.

 
Deconstructing image.bin

So now that you’ve got an image.bin file (or sm2010.rom in the case of the Weiss board), what do you do with it? What is this file anyway? That’s what I wondered, so I opened it with a hex editor and started looking around. It didn’t take long to discover that it’s just a regular 68000 program, with a lot of extra stuff attached, including a filesystem image.

The uCsimm image begins with a very recognizable 256-entry 68000 vector table, which is supposed to appear at address 0 in the 68000’s memory space. The table specifies a reset vector of 0x10C10400, so presumably ROM is mapped to address 0x10C10000 on that board, but is also temporarily mapped to 0x00000000 during startup – a common 68000 trick also employed by the Macintosh. 0x400 is the size of the vector table that must be skipped to get to the code entry point, so 0x10C10000 ROM base plus 0x400 vector table size equals the 0x10C10400 reset vector.

I was later able to confirm my guess about the uCsimm’s ROM memory map: vendor/Arcturus/uCsimm/config.linux defines CONFIG_ROMBASE as 0x10C10000 and CONFIG_ROMSTART as 0x10C10400.

The SM2010 image doesn’t contain a vector table, and the code entry point looks to be the first byte in the file. Presumably its vector table is hard-wired, or is updated through some other means.

OK, so what code is at the entry point? What does it do? By copying the bytes from the image file into this awesome online disassembler, I found the answer. The boards have similar init code, though they’re not identical. Here are the first instructions for the SM2010 board:

4e71         nop	
46fc2700     movew #0x2700,%sr	
2e7c000ffffc moveal #0x000ffffc,%sp	
207c00e62bb8 moveal #0x00e62bb8,%a0	
227c00000400 moveal #0x00000400,%a1	
247c00008f80 moveal #0x00008f80,%a2	
2018         movel %a0@+,%d0	
22c0         movel %d0,%a1@+	
b5c9         cmpal %a1,%a2	
6200fff8     bhiw 0x0000001e	
207c00008f80 moveal #0x00008f80,%a0	
227c000239f0 moveal #0x000239f0,%a1	
20fc00000000 movel #0,%a0@+	
b3c8         cmpal %a0,%a1	
6200fff6     bhiw 0x00000034	
48780000     pea 0x00000000	
487900000400 pea 0x00000400	
486f0004     pea %sp@(4)	
48780000     pea 0x00000000	
4eb900e00570 jsr 0x00e00570

So let’s see. Moving 0x2700 to the status register, that will disable interrupts on the 68000. Then it’s initializing the stack pointer to 0x000ffffc, which is presumably the top of RAM. After that there’s a copy loop, and about 35K bytes of data are copied from 0x00e62bb8 to 0x00000400. This must mean the former address is part of ROM, and the latter is RAM. Next it fills memory with zeroes from 0x00008f80 to 0x000239f0. It concludes by pushing some hard-coded values on the stack, and jumping to 0x00e00570, another ROM address.

After some extended sleuthing, I finally found that this code comes from /linux-2.0.x/arch/m68knommu/platform/68000/SM2010/crt0_rom.S. Here’s the commented source version:

	nop
	movew	#0x2700, %sr		/* disable interrupts: */
	moveal	#_boot_stack, %sp	/* set up stack at the end of RAM: */

	/* Copy data segment from ROM to RAM */
	moveal	#__data_rom_start, %a0
	moveal	#__data_start, %a1
	moveal	#__data_end, %a2

	/* Copy %a0 to %a1 until %a1 == %a2 */
LD1:
	movel	%a0@+, %d0
	movel	%d0, %a1@+
	cmpal	%a1, %a2
	bhi	LD1

	moveal	#__bss_start, %a0
	moveal	#end, %a1

	/* Copy 0 to %a0 until %a0 == %a1 */
L1:
	movel	#0, %a0@+
	cmpal	%a0, %a1
	bhi	L1

	pea	0
	pea	env
	pea	%sp@(4)
	pea	0
        jsr start_kernel

Aha! So this code looks like it’s designed to run directly from ROM, and it begins by setting up the RAM-based .data and .bss segments.

I found the code for start_kernel, and that’s where things began to look complicated. From what I can tell, the kernel isn’t actually meant to be launched directly, but rather it expects to be passed command line arguments from a bootloader. I’m guessing those pea instructions are the SM2010’s way of passing an empty argument list to the kernel.

 
What’s Next

That’s as far as I’ve gotten with deconstructing uClinux. I’m still a long way from having a Linux image I can try programming to my custom 68000 board, either the current breadboard version or the planned PCB version with extra memory and peripherals. But I feel like I’ve taken some important steps in understanding how uClinux is put together, how to build it, and how it boots. Now I’ll focus on creating a new uClinux target for my custom board, with the right memory map and set of peripherals. I don’t know what that involves, but I’ll find out!

Read 4 comments and join the conversation