Friday, November 21, 2014

Tutorials on how to build, edit and debug Android Kernel ARM64 (aarch64)

Lot of people connected to me look interested to the topic, here I decided to create a short tutorial on how download and build the kernel for aarch64 platform, make it run on QEMU (see previous article) and finally, navigate it in Eclipse, or edit it up to debugging the running kernel via GDB or in Eclipse through the IDE.

Yeahh, I know, I real kernel developer uses Vim or Emacs to play with the kernel code; unfortunately I am not a kernel developer, don't blame me!

Here the goal is not to write kernel code but to understand what happens, read the code with the help of Eclipse indexing and if you are brave, debug it!


1) need a proper toolchain for aarch64 installed (check ubuntu development essential for aarch64 package or just get what you need via single hit apt-get install). Ubuntu 14.04 has a repo with a good Linaro aarch64 toolchain 4.8 which works great!
2) optionally, get NDK from Google. The version 4.9 supports aarch64.
3) Eclipse Luna get it from, of course you need the package for C/C++ developers which includes CDT
4) Optionally, if you don't like the Eclipse base Cross GCC plugin, I would recommend to get ARM Cross GCC which ships magically with a Linaro aarch64-linu-gnu- predefined toolchain and is a lot less work in configuration and pain getting compiler macros.

Get the code and compile it, here is the video

Configure Eclipse to properly deal with Kernel Code (thanks here to the forum article)

Play with debugging

Just heard some comments, and adding this. Eclipse could, actually will, run out of memory during the re-indexing of such a big codebase. You need to tune up memory management in eclipse.ini. I attack here my eclipse.ini, the trick is to give more max heap memory -xmx and start with a bigger base minim memory -xms. Yes, I know, at the time I was looking into Java (version 6) the maximum cannot go over 1024Kb; had no time to investigate if this is still a valid limit. Set it to 2048 :)

Saturday, November 8, 2014

Ranchu where are you, kernel and emulator aarch64 (arm64)

With the release of lollipop Android L, few things have changes. First emulator64-arm64 is not included in the AOSP any more, it's a project on its own.

Find it here, git clone and compile:

As soon as a dedicated team of engineer will perform proper testing, one day or another expect to see it back. For the time being, it's kept in a different source repo.

Once you get it, you may wonder how to compile proper kernel, where is ranchu (beside the version you have on linaro), guess where it is? ranchu as natural successor of goldfish, Linux Kernel 3.10 is available (do not expect to see older versions)

1) get a proper toolchain (4.9 at least to have support for arm64, the one that ships in ubuntu repo aarch64-linux-gnu- should work fine too)

2) get the kernel source (you need this specific repo)

git clone -b android-goldfish-3.10

3) compile

make ARCH=arm64 ranchu_defconfig
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -j8
(prerequisite install arm64 compiler toolchain from your repos or from google, NDK with compiler 4.9)

A working kernel is built and available as explained in my previous post

Second the build for arm64 is kind of screwed on AOSP 5.0 r2.
The vendorsetup for a full aosp 64 bit ranchu is not there, but funny enough the make script is!
Another good useful point is the mini_emulator which allows to have a running shell only mini version of android running on qemu. If your goal is to work with low level os stuff, this is your way.

mini emulator arm64

makefile are literally wrong, they point to armv7 build, thus you build the mini_emulator_arm64 and get out the armv7 32 bit version... woaahhh

so I fixed the files, you need also to include proper init.rc for ranchu, fstab and other bits.
this file contains my fixed scripts.

- just untar it in the AOSP/device/generic
on AOSP root folder, source the build env

source build/
lunch mini_emulator_arm64-userdebug 
m -j8

wait... you'll get out/products/mini-emulator-arm64 with *.img files to run the emulator like described in my previous article, it works!

full AOSP arm64 on qemu

here it's even easier, google was a bit more kind and left the make files there, you just miss the vendorsetup to get the lunch bundle. in the file above, download it, you have it. it extracts the file in the device/generic/qemu folder. just do it and select the new entry:

source build/
lunch ranchu_arm64-eng
m -j8

Tuesday, November 4, 2014

Android L AOSP(preview) under ARM64 (aarch64) QEMU emulator

As many others, I can't wait to put my hands on Android running on aarch64 (arm-v8) and as many other, it's difficult to afford such an expensive Juno board (the only development board arm-v8 available which I am aware of) If you Google a bit you'll bump into this cool article:

The Linaro team put together ahead of time, around may 2014, a version of the Kernel for aarch64 called ranchu (forked from 3.10) and patched/worked together with the qemu team to create a machine model ranchu capable of emulating aarch64. Everything is well explained in the article above. 

However, by compiling the latest Android L preview 2 AOSP, I ended up with serious issues during the zygote startup. It looks there is something seriously different in how is made, thus passing it to patchoat (at first time boot) something goes very wrong. Not going into details of the hell issue.

Sometimes, when things goes wrong, better to restart from scratch. It's the old rule learned since the old Windows 95 time, hit ctrl-alt-delete. 

While compiling the AOSP, I noticed that into the prebuilt folder you have something called emultator64-arm64 which sounds pretty much what I wanted; who said that Google is not providing amr64 emulation yet? :) Yes, it's not official, emulator images are not provided with the latest SDKv19 but I happily found another piece of gold. In the folder qemu-kernel/arm64/kernel-qemu this file sounds very much like the kernel prebuilt for the arm64 emulator. Why not trying compiling AOSP for generic arm64 support? in the end the device reference is there in the AOSP without any patching... and voila...

Here are the steps to follow:

1) get Android L developer preview AOSP

cd /data/src
mkdir AOSP 
repo init -u -b android-l-preview_r2
repo sync

wait a considerable amount of time to sync, almost 2 hours, prepare almost 100GB on your drive
note: you need repo command from google configured on your system.

2) have fun and compile (takes almost 1h on a decent machine)

source build/
lunch aosp_arm64-eng
m -j8
here is the lunch output

3) move to the prebuilt emulator folder, in my case I am under linux-x64, chose your arch

cd /data/src/AOSP/prebuilts/android-emulator/linux-x86_64

4) to run the emulator you need anyhow an AVD configuration or start it manually from the qemu binary without using the google emulator wrapper. I prefer to use the wrapper here as the command is cleaner.

create an AVD using your ANDROID SDK, launch 

./android avd

(from platform tools, you can create one via command line ./android create)

or if you have already defined AVDs, just use one of them, settings will be overwritten by the command, so whatever you have in your AVD you are just using the config.ini file (stored in the .android folder). Preferably, set your AVD with more then 1024 MB ram. 

5) now you are ready to launch the emulator, here is the command (all one line). 

testL is the name of the AVD prepared. Wait, wait wait for android to finish booting, remember you are on a slow emulation, it takes time. If you are curios to see what's going on, just run "logcat" at the prompt to see all the mess in real time. 

./emulator64-arm64 -kernel ../../qemu-kernel/arm64/kernel-qemu -data /data/src/AOSP/out/target/product/generic_arm64/system.img -system /data/src/AOSP/out/target/product/generic_arm64/system.img -cache /data/src/AOSP/out/target/product/generic_arm64/cache.img -ramdisk /data/src/AOSP/out/target/product/generic_arm64/ramdisk.img -avd testL

Output, kernel boot and android prompt:

console on port 5554, ADB on port 5555
Initializing cgroup subsys cpu
Linux version 3.10.0+ ( (gcc version 4.8 (GCC) ) #12 SMP Tue Sep 16 22:36:19 CEST 2014
CPU: AArch64 Processor [411fd070] revision 0
Machine: ranchu
debug: skip boot console de-registration.
Unknown earlyprintk arguments: ttyAMA0
PERCPU: Embedded 10 pages/cpu @ffffffc07ffdf000 s11456 r8192 d21312 u40960
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 517120
Kernel command line: console=ttyAMA0,38400 keep_bootcon earlyprintk=ttyAMA0
PID hash table entries: 4096 (order: 3, 32768 bytes)
Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)
Memory: 2048MB = 2048MB total
Memory: 2058560k/2058560k available, 38592k reserved
Virtual kernel memory layout:
    vmalloc : 0xffffff8000000000 - 0xffffffbbffff0000   (245759 MB)
    vmemmap : 0xffffffbc00e00000 - 0xffffffbc02a00000   (    28 MB)
    modules : 0xffffffbffc000000 - 0xffffffc000000000   (    64 MB)
    memory  : 0xffffffc000000000 - 0xffffffc080000000   (  2048 MB)
      .init : 0xffffffc00057a000 - 0xffffffc0005a8cc0   (   188 kB)
      .text : 0xffffffc000080000 - 0xffffffc000579364   (  5093 kB)
      .data : 0xffffffc0005a9000 - 0xffffffc0005e7200   (   249 kB)
SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Hierarchical RCU implementation.
 RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=1.
NR_IRQS:64 nr_irqs:64 0
GIC CPU mask not found - kernel will fail to boot.
GIC CPU mask not found - kernel will fail to boot.
Architected local timer running at 62.50MHz (virt).
Console: colour dummy device 80x25
Calibrating delay loop (skipped), value calculated using timer frequency.. 125.00 BogoMIPS (lpj=625000)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 256
/cpus/cpu@0: Unknown CPU type
hw perfevents: no hardware support available
Brought up 1 CPUs
SMP: Total of 1 processors activated (125.00 BogoMIPS).
atomic64 test passed
NET: Registered protocol family 16
vdso: 2 pages (1 code, 1 data) at base ffffffc0005b1000
hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
software IO TLB [mem 0xbd400000-0xbd800000] (4MB) mapped at [ffffffc07d400000-ffffffc07d7fffff]
Serial: AMBA PL011 UART driver
9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 33) is a PL011 rev1
console [ttyAMA0] enabled
bio: create slab  at 0
SCSI subsystem initialized
Switching to clocksource arch_sys_counter
NET: Registered protocol family 2
TCP established hash table entries: 16384 (order: 6, 262144 bytes)
TCP bind hash table entries: 16384 (order: 6, 262144 bytes)
TCP: Hash tables configured (established 16384 bind 16384)
TCP: reno registered
UDP hash table entries: 1024 (order: 3, 32768 bytes)
UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes)
NET: Registered protocol family 1
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
Unpacking initramfs...
Freeing initrd memory: 816K (ffffffc008000000 - ffffffc0080cc000)
fuse init (API version 7.22)
msgmni has been set to 4022
io scheduler noop registered
io scheduler cfq registered (default)
loop: module loaded
 vda: unknown partition table
 vdb: unknown partition table
 vdc: unknown partition table
tun: Universal TUN/TAP device driver, 1.6
tun: (C) 1999-2004 Max Krasnyansky 
mousedev: PS/2 mouse device common for all mice
input: qwerty2 as /devices/9040000.goldfish-events/input/input0
ashmem: initialized
logger: created 256K log 'log_main'
logger: created 256K log 'log_events'
logger: created 256K log 'log_radio'
logger: created 256K log 'log_system'
ip_tables: (C) 2000-2006 Netfilter Core Team
arp_tables: (C) 2002 David S. Miller
TCP: cubic registered
NET: Registered protocol family 10
ip6_tables: (C) 2000-2006 Netfilter Core Team
sit: IPv6 over IPv4 tunneling driver
NET: Registered protocol family 17
Bridge firewalling registered
Registering SWP/SWPB emulation handler
drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
Freeing unused kernel memory: 184K (ffffffc00057a000 - ffffffc0005a8000)
init (1): /proc/1/oom_adj is deprecated, please use /proc/1/oom_score_adj instead.
init: /dev/hw_random not found
init: /dev/hw_random not found
EXT4-fs (vda): mounted filesystem with ordered data mode. Opts: (null)
EXT4-fs (vdb): Ignoring removed nomblk_io_submit option
EXT4-fs (vdb): mounted filesystem with ordered data mode. Opts: nomblk_io_submit,errors=panic
EXT4-fs (vdc): Ignoring removed nomblk_io_submit option
EXT4-fs (vdc): mounted filesystem with ordered data mode. Opts: nomblk_io_submit,errors=panic
init: cannot find '/system/etc/', disabling 'flash_recovery'
healthd: BatteryVoltagePath not found
healthd: BatteryTemperaturePath not found
binder: 774:774 transaction failed 29189, size 0-0
logd.auditd: start
shell@generic_arm64:/ $