Labels

Thursday 16 October 2014

Debian on Nexus 7: Booting a Ramdisk

Now that I have a custom kernel image, I'll look at creating a ramdisk image to accompany this. The ramdisk image shall contain a minimal Linux (not Android!) user space with a suite of basic utilities. The kernel and ramdisk images shall be combined into a boot image, in the format that the Nexus 7's bootloader expects.

The Android Factory Image Ramdisk


Booting a minimal Linux user space is an important step in this project, as it is the first attempt at running a non-Android Linux on the tablet. But, the user space won't be capable of interacting with much of the tablet's hardware, such as the screen or WiFi adapter (this will be the focus of subsequent posts!). I need to make sure that I can make the user space do something to show that it booted successfully. And, ideally, I want some kind of shell access too!

Its time to return to the boot image file from the Android KTU84P factory image, which I unpacked in my first post. From this blog post (which, incidentally, is where I found unmkbootimg), I see that the ramdisk CPIO archive can be unpacked with:
$ mkdir ramdisk
$ cd ramdisk
$ gunzip -c ../ramdisk.cpio.gz | cpio -iu
$ ls
charger          init.flo.rc        sbin
data             init.flo.usb.rc    seapp_contexts
default.prop     init.rc            sepolicy
dev              init.trace.rc      sys
file_contexts    init.usb.rc        system
fstab.flo        proc               ueventd.flo.rc
init             property_contexts  ueventd.rc
init.environ.rc  res
and repacked with:
$ find . | cpio -o -H newc | gzip > ../ramdisk.cpio.gz
What is of most interest here are the init.*.rc init scripts. Android provides its own unique init, which acts on instructions from these scripts when certain events occur. The most interesting scripts are init.flo.rc and init.flo.usb.rc (which, of course, are both available in the AOSP sources). This is an interesting segment from init.flo.rc:
# White LED
chown system system /sys/class/leds/white/device/lock
chown system system /sys/class/leds/white/brightness
chown system system /sys/class/leds/white/device/grpfreq
chown system system /sys/class/leds/white/device/grppwm
chown system system /sys/class/leds/white/device/blink
This suggests that the tablet has a white LED, which can be controlled from user space. The kernel driver which controls this hardware exposes some kind of interface through sysfs (one of the special filesystems that the kernel provides, and which is usually mounted on /sys). A quick Google yields this forum post, which suggests that the following might be enough to switch the LED on and into a blinking state:
$ echo 255 > /sys/class/leds/white/brightness
$ echo 1 > /sys/class/leds/white/device/blink
This is enough to provide an indication that the minimal user space is booting, or has booted. The shell commands above can be used in the ramdisk's own init boot script. If the LED comes on, then that boot script is getting executed!

The next interesting segments come from init.flo.usb.rc:
on init
    write /sys/class/android_usb/android0/f_rndis/manufacturer LGE
    write /sys/class/android_usb/android0/f_rndis/vendorID 18D1
    write /sys/class/android_usb/android0/f_rndis/wceis 1

# rndis
on property:sys.usb.config=rndis
    stop adbd
    write /sys/class/android_usb/android0/enable 0
    write /sys/class/android_usb/android0/idVendor 18D1
    write /sys/class/android_usb/android0/idProduct 4EE3
    write /sys/class/android_usb/android0/bDeviceClass 239
    write /sys/class/android_usb/android0/bDeviceSubClass 2
    write /sys/class/android_usb/android0/bDeviceProtocol 1
    write /sys/class/android_usb/android0/functions rndis
    write /sys/class/android_usb/android0/enable 1
    setprop sys.usb.state ${sys.usb.config}
This is again using sysfs to interact with the Android USB gadget driver, which I deliberately kept in when building my custom kernel in the previous post. This gadget driver provides the various USB personalities that an Android device may assume, including amongst others CDC-ACM, RNDIS and the USB interface used by adb. RNDIS (a proprietary Microsoft protocol) is used when an Android device enters its USB tethering mode. It provides an IP network connection between a PC and the Android device over the USB cable. These segments appear to be the steps necessary to instruct the Android USB gadget driver to activate its RNDIS mode. It would be great to utilise this to get a remote shell from the minimal user space over, say, telnet!

As an aside, I did experiment with trying to get the CDC-ACM mode of the Android USB gadget driver working. CDC-ACM allows serial over USB, and might have been useful in setting up a virtual serial console (allowing boot messages and the like to be viewed). Unfortunately, despite trying all sorts of port settings on both sides, I couldn't achieve a stable connection. I could get some text through it (which turned out to be invaluable in grabbing small excerpts of the kernel log), but after a short while the serial connection would unfortunately hang (and no longer be usable until the next boot). I therefore tried the RNDIS mode instead, which did work reliably.

Creating the Minimal User Space


The first thing to do is to set up the basic filesystem structure of the ramdisk. I'll do this as root, so that the user and group IDs used in the filesystem are also set to root.
$ mkdir initrd
$ sudo mkdir initrd/rootfs
$ cd initrd/rootfs
$ sudo mkdir bin dev etc lib mnt proc root sbin sys tmp var
I also need some important static device nodes:
$ cd dev
$ sudo mknod -m 622 console c 5 1
$ sudo mknod -m 666 null c 1 3
Since I want a minimal user space, the obvious choice for the user space tools is BusyBox. This comes as a single binary, and a large collection of symbolic links. The binary provides the implementation of all the user space tools, and the specific tool that gets started depends on the symbolic link that we invoke it through. BusyBox is widely used in ramdisk images and embedded systems which require a minimal user space. So, I'll fetch the source for BusyBox 1.22.1:
$ cd initrd
$ mkdir busybox
$ cd busybox
$ wget http://busybox.net/downloads/busybox-1.22.1.tar.bz2
$ tar xjf busybox-1.22.1.tar.bz2
$ cd busybox-1.22.1
As with the kernel build, this will use the ARM cross compiler toolchain that I installed in my previous post. So, I'll again need to use the ARCH, SUBARCH and CROSS_COMPILE environment variables. I'll set up the default configuration, and enter the menuconfig utility (which works in the same way as the Linux kernel's own menuconfig tool):
$ ARCH=arm SUBARCH=arm CROSS_COMPILE=arm-linux-gnueabi- make defconfig
$ ARCH=arm SUBARCH=arm CROSS_COMPILE=arm-linux-gnueabi- make menuconfig
I'll first make the following configuration changes:
CONFIG_DESKTOP=n
CONFIG_INSTALL_NO_USR=y
CONFIG_STATIC=y
CONFIG_PREFIX=/home/simon/Android/Nexus7-Debian/initrd/rootfs
CONFIG_FEATURE_SYSTEMD=n
This isn't a desktop installation, and I don't want to use /usr in my ramdisk. I also want to force BusyBox to statically link against the C library, so that I don't have to worry about having to also deploy that to the ramdisk (and as we're only using a single binary, there isn't really a need for a shared C library). /home/simon/Android/Nexus7-Debian/initrd/rootfs is the full path to initrd/rootfs directory on my system. I don't want systemd, as this really will be a very minimal user space.

I also need to ensure that I have proper support for user accounts, and a DHCP and telnet server:
CONFIG_USE_BB_PWD_GRP=y
CONFIG_USE_BB_SHADOW=y
CONFIG_UDHCPD=y
CONFIG_TELNETD=y
CONFIG_FEATURE_TELNETD_STANDALONE=y
And to reduce the size a bit, I can remove some tools that I don't think I'll need (this is in no way an optimal list):
CONFIG_FEATURE_HWIB=n
CONFIG_RPM=n
CONFIG_RPM2CPIO=n
CONFIG_RUN_PARTS=n
CONFIG_START_STOP_DAEMON=n
CONFIG_LPD=n
CONFIG_LPR=n
CONFIG_LPQ=n
CONFIG_MAKEMIME=n
CONFIG_POPMAILDIR=n
CONFIG_REFORMIME=n
CONFIG_SENDMAIL=n
I'll now build and install BusyBox to the ramdisk filesystem:
$ ARCH=arm SUBARCH=arm CROSS_COMPILE=arm-linux-gnueabi- make -j8
$ sudo bash -c "PATH=$PATH ARCH=arm SUBARCH=arm CROSS_COMPILE=arm-linux-gnueabi- make install"
There are a couple more tweaks required. Firstly, BusyBox recommends that its binary has setuid set. This is done for completeness (although isn't critical as I'll be running as root anyway). Also, the kernel will expect to find init as /init (rather than /sbin/init). So I need to fulfil that requirement with a symbolic link:
$ sudo chmod 4755 ~/Android/Nexus7-Debian/initrd/rootfs/bin/busybox
$ sudo ln -s sbin/init ~/Android/Nexus7-Debian/initrd/rootfs/init
The minimal user space is now populated. It now needs to be configured.

Configuring the Minimal User Space


I'll lay down a basic configuration in the ramdisk's /etc directory. The paths below are relative to the root of the ramdisk filesystem (which on my system is ~/Android/Nexus7-Debian/initrd/rootfs). All files need to be owned by root, so I'll run my editor with sudo.

This sets up the root user and group, and permits remote logins as root (obviously, security is not a concern in this image).

/etc/passwd:

root::0:0:root:/root:/bin/sh

/etc/group:

root:x:0:

/etc/securetty:

pts/0

And this configures the DHCP server to operate over the RNDIS network connection:

/etc/udhcpd.conf:

# The IP lease block
start 192.168.1.20
end 192.168.1.254

# The network interface to use
interface usb0
option subnet 255.255.255.0
Finally, I need an appropriate boot script to start up the minimal user space, and the services it requires. I'll also try to turn on the tablet's white LED light, so I can see that the boot is actually happening. We're using BusyBox's init, so the boot script shall be a simple init script:

/etc/init.d/rcS:

#!/bin/sh

# Mount core filesystems
mount -t devtmpfs none /dev
mkdir /dev/pts
mount -t devpts none /dev/pts
mount -t proc none /proc
mount -t sysfs none /sys  
mount -t tmpfs none /tmp
mount -t tmpfs none /var

# Blink the white LED to show that we're booting
echo 255 > /sys/class/leds/white/brightness
echo 1 > /sys/class/leds/white/device/blink

# Setup logging
mkdir /var/log
syslogd

# Set a hostname
/bin/hostname nexus-7-initrd

# Disable the Android composite USB gadget driver
echo 0 > /sys/class/android_usb/android0/enable

# Setup USB device properties the driver should advertise
echo 18d1 > /sys/class/android_usb/android0/idVendor
echo 2d04 > /sys/class/android_usb/android0/idProduct
echo 239 > /sys/class/android_usb/android0/bDeviceClass
echo 2 > /sys/class/android_usb/android0/bDeviceSubClass
echo 1 > /sys/class/android_usb/android0/bDeviceProtocol

# Setup RNDIS-specific properties that the driver should advertise
echo LGE > /sys/class/android_usb/android0/f_rndis/manufacturer
echo 18D1 > /sys/class/android_usb/android0/f_rndis/vendorID
echo 1 > /sys/class/android_usb/android0/f_rndis/wceis

# Select the RNDIS function of the gadget driver
echo rndis > /sys/class/android_usb/android0/functions

# Re-enable the gadget driver, using the updated configuration
echo 1 > /sys/class/android_usb/android0/enable

# Wait a few seconds
sleep 5

# Bring up usb0, set a static IP and start udhcpd
/sbin/ifconfig usb0 up
/sbin/ifconfig usb0 192.168.1.1 netmask 255.255.255.0
/sbin/udhcpd /etc/udhcpd.conf

# Show a solid white LED to indicate that we're ready
echo 0 > /sys/class/leds/white/device/blink

# Run telnetd
while true
do
  /sbin/telnetd -l /bin/login -F -K
done

By mounting devtmpfs on /dev, I'm allowing the kernel to provide the appropriate device nodes. I won't get notifications of devices coming and going, and can't enforce any customisations (such as renames or particular permissions), but its good enough. I also ensure that I mount the other special kernel filesystems, and use two further tmpfs ramdisks for /tmp and /var. The configuration of the white LED and the Android USB gadget driver use the information from earlier in this blog post. As I'm keeping this very simple, I also start the services I require (namely the system logger (syslogd) and the DHCP server (udhcpd)) and continuously run the telnet server (telnetd) to permit remote logins. The five second sleep is there to "play it safe" (ensuring that the Android USB gadget driver is definitely ready before trying to configure networking).

The final touch is to ensure that the init script is executable:
$ sudo chmod 755 ~/Android/Nexus7-Debian/initrd/rootfs/etc/init.d/rcS
The ramdisk filesystem, with the minimal user space, is now ready. I can package this into the required gzipped CPIO archive (using the command mentioned earlier in this post):
$ cd ~/Android/Nexus7-Debian/initrd/rootfs
$ sudo bash -c "find . | cpio -o -H newc | gzip > ../initrd-rootfs.cpio.gz"

Creating the Boot Image


I now need to package up both the kernel from the previous post and the ramdisk CPIO archive into a boot image for the tablet. On my system, the kernel image is located at ~/Android/Nexus7-Debian/kernel/msm/arch/arm/boot/zImage and the ramdisk CPIO archive is at ~/Android/Nexus7-Debian/initrd/initrd-rootfs.cpio.gz.

I'll need mkbootimg, which I built in my first post. Recalling the parameters that unmkbootimg suggested when it unpacked the boot.img from the KTU84P factory image, I'll build my boot image as follows:
$ cd ~/Android/Nexus7-Debian
$ mkdir out
$ cd out
$ mkbootimg --base 0 --pagesize 2048 --kernel_offset 0x80208000 --ramdisk_offset 0x82200000 \
      --second_offset 0x81100000 --tags_offset 0x80200100 \
      --cmdline 'console=ttyHSL0,115200,n8 msm_rtb.filter=0x3F ehci-hcd.park=3' \
      --kernel ../kernel/msm/arch/arm/boot/zImage \
      --ramdisk ../initrd/initrd-rootfs.cpio.gz -o boot.img
$ ls
boot.img
It worked! Now to see if it'll actually boot.

Booting the Tablet


With the tablet off, I'll power it on with the volume down key held down to enter the bootloader menu (which depicts an Android robot lying on its back). I'll also connect the tablet to my PC with its USB cable. I also need the fastboot utility, which I've taken from an Android SDK. I'll first check that fastboot sees the tablet:
$ fastboot devices
01234567 fastboot
My tablet is already unlocked, as the bootloader menu states:
LOCK STATE - unlocked
It it weren't unlocked, I'd need to issue the fastboot command given below and follow the instructions on the tablet screen to confirm. Note that, for security reasons, unlocking the tablet will wipe all user data from it (effectively leaving you with a factory reset device).
$ fastboot oem unlock
I can now perform a one-time boot of my boot image, without flashing anything to the tablet. If anything went wrong, the tablet would be fine (powering it off and on again would return it to Android). I'll kick off the boot with:
$ fastboot boot boot.img
downloading 'boot.img'...
OKAY [  0.245s]
booting...
OKAY [  0.025s]
finished. total time: 0.270s
And now the moment of truth! After a brief pause, I see the white LED light begin to flash. Hurrah, this means that the kernel has booted and my init boot script is running! The tablet's screen still shows the bootloader menu, because my minimal user space environment has not tried to display anything (and the bootloader doesn't bother clearing the screen first). After a little while longer, the white LED stops blinking and remains constantly on. I also notice that my PC has a new network interface (usb0) with the IP address 192.168.1.20. This indicates that the RNDIS network connection is active, and that the tablet assigned my PC an IP address. I should now be able to establish a telnet connection to get a shell:
$ telnet 192.168.1.1
Trying 192.168.1.1...
Connected to 192.168.1.1.
Escape character is '^]'.

nexus-7-initrd login: root
~ #
Success!! I've booted the tablet with a minimal user space, and got access to a shell! For good measure:
~ # cat /proc/cpuinfo 
Processor       : ARMv7 Processor rev 0 (v7l)
processor       : 0
BogoMIPS        : 13.53

processor       : 1
BogoMIPS        : 13.53

processor       : 2
BogoMIPS        : 13.53

processor       : 3
BogoMIPS        : 13.53

Features        : swp half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 
CPU implementer : 0x51
CPU architecture: 7
CPU variant     : 0x1
CPU part        : 0x06f
CPU revision    : 0

Hardware        : QCT APQ8064 FLO
Revision        : 0000
Serial          : 0000000000000000
I'm now off to explore this a little! Obviously, once done I can power off the tablet in the usual way (reboot also works):
~ # poweroff
~ # Connection closed by foreign host.

Next Time...


... I'm going to see if I can get the screen to work!

Saturday 11 October 2014

Debian on Nexus 7: Building a Kernel

In this post, I'll look at building a custom Linux kernel image for the Nexus 7 which can be used later on to boot a ramdisk containing a minimal user space environment. I've decided to keep this as its own post, to keep it manageable. I'll cover the creation and booting of the ramdisk in my next post.

Cross Compiler Toolchain


As this is going to involve compiling C source code to run natively on the tablet, I'm going to need to get a C cross compiler toolchain. I could use Android's own ARM toolchain for this but, as it turns out, Ubuntu 14.04 also provides a suitable toolchain in the gcc-arm-linux-gnueabi package. This targets the armel Embedded Application Binary Interface (EABI), which Debian has supported from Lenny onwards. It would also be worth looking into targeting the armhf ABI in future, as this allows use of the hardware floating point unit.

To fetch the toolchain, along with a few other useful packages:
$ sudo apt-get install build-essential libncurses5-dev gcc-arm-linux-gnueabi
As the name suggests, build-essential provides some essential packages for software building. libncurses5-dev is required by the kernel's menuconfig configuration tool, to provide an ANSI-based user interface in a terminal.

Building the Kernel Image


And so now onto the main challenge of building the custom Linux kernel image. Unfortunately, this is one area where I can't avoid the Android sources. The branch of the kernel source provided by the AOSP for this tablet contains important drivers, which are not currently available in the mainline kernel. Whilst it would be entirely possible to port these drivers from the Android kernel source to a copy of the mainline source, this is not something I currently have the time or inclination to do. So, my kernel source tree shall be taken from the AOSP.

The starting point then is Google's building kernels guide. This guide lists the various kernel source trees provided by the AOSP, and how they match up to devices (listed by codename). As mentioned in the previous post, the Nexus 7 2013 WiFi-Only device is known by the codenames flo and razor. So, the guide indicates that the kernel/msm tree should be used, with the build configuration flo_defconfig. The guide also describes how the kernel version can be extracted from a kernel image file. Helpfully, my Nexus 7 displays the kernel version in its "About device" settings page. I'm using the KTU84P factory image, and the kernel version in this image is 3.4.0-g03485a6. The g03485a6 is the important bit, as this describes the git commit that the kernel source tree was checked out from when the kernel image was built. The leading g is dropped, giving the commit ID 03485a6. Looking through the web interface for the kernel/msm tree, I can see that this commit corresponds to the head of the android-msm-flo-3.4-kitkat-mr2 branch. Therefore, currently, simply checking out the head of this branch should give me what I need:
$ mkdir kernel
$ cd kernel
$ git clone https://android.googlesource.com/kernel/msm.git
$ cd msm
$ git checkout android-msm-flo-3.4-kitkat-mr2
$ ls
android           Documentation  Kbuild             mm              sound
AndroidKernel.mk  drivers        Kconfig            net             tools
arch              firmware       kernel             README          usr
block             fs             lib                REPORTING-BUGS  virt
COPYING           include        MAINTAINERS        samples
CREDITS           init           make_defconfig.sh  scripts
crypto            ipc            Makefile           security
There appears the nice and familiar Linux kernel source tree. Next, I need to select the appropriate kernel configuration for this tablet. This is achieved with:
$ ARCH=arm SUBARCH=arm CROSS_COMPILE=arm-linux-gnueabi- make flo_defconfig
The ARCH, SUBARCH and CROSS_COMPILE environment variables indicate to the kernel build system the target architecture and compiler toolchain that it should use. Instead of specifying them on each call to make, I could also simply export them in the current shell but, for clarity, I'll continue to use them with each command. flo_defconfig is the name of the build configuration (as mentioned above).

Now I have a configuration which would produce a fully functional Android kernel for my tablet. But I want to customise it some more, to give me a kernel more suited for non-Android use. So, I'll enter the kernel's menuconfig configuration tool and make some tweaks:
$ ARCH=arm SUBARCH=arm CROSS_COMPILE=arm-linux-gnueabi- make menuconfig
In the current configuration, certain important features have been disabled. I enabled the configuration options with the parameter names listed below:
CONFIG_DEVTMPFS=y
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
Typing / at any time in menuconfig will allow you to search for a configuration option by its parameter name. Once an option has been located, the spacebar toggles between unselected (< >), built-in (<*>) or module (<M>). In the notation used above (which is that of the configuration file itself), unselected is n, built-in is y and module is m. Sometimes, built-in or module will be unavailable for an option, depending on what the rest of the configuration looks like.

There's also a few options I don't need for now. Such as SELinux:
CONFIG_SECURITY_SELINUX=n
CONFIG_EXT4_FS_SECURITY=n
It is also possible to remove quite a few Android-specific drivers that make a Linux kernel capable of supporting the Android user space. This includes things such as Android's Binder IPC driver, and the Anonymous Shared Memory (ASH) subsystem. Unfortunately, disabling the Android-specific drivers did cause a couple of build errors:
  • The function kgsl_get_vma_from_start_addr() in drivers/gpu/msm/kgsl.c was no longer used because CONFIG_ASHMEM was unselected, leading to a fatal warning. Enclosing the function in a #ifdef CONFIG_ASHMEM...#endif block appeared to fix this issue.
  • The file drivers/staging/prima/CORE/WDI/TRP/CTS/src/wlan_qct_wdi_cts.c erroneously switched from including mach/msm_snd.h to including msm_snd.h because CONFIG_ANDROID was unselected. Simply removing this conditional behaviour appeared to fix this issue.
After fixing the source files above, these Android-specific drivers could be disabled by setting:
CONFIG_ANDROID=n
It probably wouldn't have done any harm to leave in these Android-specific drivers, I'm just curious to see if I can boot the kernel without them. However, one Android driver that I do not want to disable is the Android USB gadget driver (CONFIG_USB_G_ANDROID). As I'll describe later on, this will be fundamental in establishing basic communication (as in access to a shell) with the minimal user space environment.

Now I'm finished making my configuration changes, I'll exit menuconfig (saving the changes) and kick off the kernel build with:
$ ARCH=arm SUBARCH=arm CROSS_COMPILE=arm-linux-gnueabi- make -j8
The -j argument governs the number of tasks that the build will perform in parallel. I tend to set this to twice the number of (real) CPU cores on my system (I want the build to finish ASAP!). My machine has an Intel Core i7-4770 processor which is quad-core, so I use -j8. The build still takes a good few minutes, however. The build then finishes with the usual notification:
  Kernel: arch/arm/boot/zImage is ready
And sure enough:
$ file arch/arm/boot/zImage
arch/arm/boot/zImage: Linux kernel ARM boot executable zImage (little-endian)
I have a kernel image! I'm now ready to create a ramdisk image to accompany it.

Next Time...


... I will create the ramdisk image and boot the tablet into a minimal Linux userspace environment.