Thursday, December 4, 2014

The truth about mobile security

Here is the deck I gave during the Solution Connect 2014 event. Usually I don't make public posts with decks but since they did it, I will do the same :)


Tuesday, December 2, 2014

kprobes on Android arm64, the story

I am trying to bring kprobes support into Android (arm64); actually don't need the full kprobes stack at the moment but something light like jprobe would be enough. Of course having the full instrumentation capabilities... wow

Sandeep Prabhu first and later David (Dave) Long from Linaro are working on a similar task. I wrote and talked to both of them.

Here is where I stand:

I tried first the patch from Dave https://patches.linaro.org/40993/ getting out a lot of problems, and finally I moved back to the original patch from Sandeep https://git.linaro.org/people/sandeepa.prabhu/linux-aarch64.git/shortlog/refs/heads/arm64_kprobes_3.14-rc3 getting almost the same results.

- compiled the simple sample/kprobes/jprobes_example.c module and trying to load it, it's a simple instrumentation of do_fork showing params passed.
- I am working on two different kernels: goldfish android kernel (google repo https://android.googlesource.com/kernel/goldfish branch  android-goldfish-3.10 ) 3.10 AND linaro-stable-kernel android 3.14 (origin https://git.linaro.org/kernel/linux-linaro-stable.git branch linux-linaro-lsk-v3.14-android). 
Sandeep patch was initially wrote for 3.14.
- to test, I am using qemu (arm64 not yet available in google emulators), latest version google forked of qemu include ranchu (aarch64 armv8 vm), it's possible to run android on it. I wrote more, see my previous two articles: http://restart-thinking.vitorallo.com 

The jprobe is implanted correctly but when do_fork happens: Bad mode in Synchronous Abort handler detected

(#define in asm/ptrace.h patched - see below)

insmod jprobe_example.ko                                                       <
Planted jprobe at ffffffc000096800, handler addr ffffffbffc000000
shell@mini-emulator-arm64:/system/vito # ls
Bad mode in Synchronous Abort handler detected, code 0x84000005
CPU: 0 PID: 1068 Comm: sh Tainted: G           O 3.14.25+ #5
task: ffffffc03ee93d80 ti: ffffffc03dc58000 task.ti: ffffffc03dc58000
PC is at 0x0
LR is at 0x10001
pc : [<0000000000000000>] lr : [<0000000000010001>] pstate: 00100145
sp : ffffffc03dc5c040
x29: 0000000000000000 x28: 0000000000000000
x27: 0000000000000000 x26: 0000000000000000
x25: 00000000000000dc x24: 0000000001200011
x23: 0000000060000000 x22: 0000007fa4a765a0
x21: 0000007ff043e270 x20: 0000007fa4a78498
x19: 0000007ff043e270 x18: 0000005574556000
x17: 0000000000000000 x16: 00000000ffffffff
x15: 0000007ff043e318 x14: 0000005574557620
x13: 0000007fa471e2a8 x12: 0000007fa4775108
x11: 000000000000042c x10: 0000000000000001
x9 : 0000007fa4b580e8 x8 : 0000000000000000
x7 : 0000007fa4a76580 x6 : 0000007fa4b1a180
x5 : 0000000000000000 x4 : 0000007fa4b28000
x3 : 0000007fa4b1a000 x2 : 000000000000000c
x1 : 0000007fa4700ac8 x0 : 000000000000042f

Internal error: Oops - bad mode: 0 [#1] SMP
Modules linked in: jprobe_example(O)
CPU: 0 PID: 1068 Comm: sh Tainted: G           O 3.14.25+ #5
task: ffffffc03ee93d80 ti: ffffffc03dc58000 task.ti: ffffffc03dc58000
PC is at 0x0
LR is at 0x10001
pc : [<0000000000000000>] lr : [<0000000000010001>] pstate: 00100145
sp : ffffffc03dc5c040
x29: 0000000000000000 x28: 0000000000000000
x27: 0000000000000000 x26: 0000000000000000
x25: 00000000000000dc x24: 0000000001200011
x23: 0000000060000000 x22: 0000007fa4a765a0
x21: 0000007ff043e270 x20: 0000007fa4a78498
x19: 0000007ff043e270 x18: 0000005574556000
x17: 0000000000000000 x16: 00000000ffffffff
x15: 0000007ff043e318 x14: 0000005574557620
x13: 0000007fa471e2a8 x12: 0000007fa4775108
x11: 000000000000042c x10: 0000000000000001
x9 : 0000007fa4b580e8 x8 : 0000000000000000
x7 : 0000007fa4a76580 x6 : 0000007fa4b1a180
x5 : 0000000000000000 x4 : 0000007fa4b28000
x3 : 0000007fa4b1a000 x2 : 000000000000000c
x1 : 0000007fa4700ac8 x0 : 000000000000042f

Process sh (pid: 1068, stack limit = 0xffffffc03dc58058)
Stack: (0xffffffc03dc5c040 to 0xffffffc03dc5c000)
Call trace:
Code: bad PC value
---[ end trace 77f3ec18f6ecc57c ]---
Kernel panic - not syncing: Fatal exception

I kind of figure out something bad. PC is zero, :O who did that.. it must be something wrong with who is handling PC.
In the patched tree, ptrace.h is kind of "old":

original in my kernel, android goldfish: 
#define instruction_pointer(regs)       ((unsigned long)(regs)->pc)
in the patch sandeepa's tree: 
#define instruction_pointer(regs)       (regs)->pc
or dave's tree:
#define instruction_pointer(regs)       ((regs)->pc)

This change was introduced back in time by commit 27aa55c5e5

git diff 27aa55c5e5123fa8b8ad0156559d34d7edff58ca arch/arm64/include/asm/ptrace.h

...
  * Are the current registers suitable for user mode? (used to maintain
@@ -190,7 +183,7 @@ static inline int valid_user_regs(struct user_pt_regs *regs)
        return 0;
 }

-#define instruction_pointer(regs)      (regs)->pc
+#define instruction_pointer(regs)      ((unsigned long)(regs)->pc)
... 

so if I finally set back the original value casted with (unsigned long) in the define, the compilation obviously fails like this:

errors with ktrace.c

arch/arm64/kernel/kprobes.c: In function ‘skip_singlestep_missed’:
arch/arm64/kernel/kprobes.c:201:28: error: lvalue required as left operand of assignment
  instruction_pointer(regs) += sizeof(kprobe_opcode_t);
                            ^
arch/arm64/kernel/kprobes.c: In function ‘setup_singlestep’:
arch/arm64/kernel/kprobes.c:241:29: error: lvalue required as left operand of assignment
   instruction_pointer(regs) = slot;
                             ^
arch/arm64/kernel/kprobes.c: In function ‘post_kprobe_handler’:
arch/arm64/kernel/kprobes.c:286:29: error: lvalue required as left operand of assignment
   instruction_pointer(regs) = cur->ainsn.restore.addr;
                             ^
arch/arm64/kernel/kprobes.c: In function ‘kprobe_fault_handler’:
arch/arm64/kernel/kprobes.c:325:29: error: lvalue required as left operand of assignment
   instruction_pointer(regs) = (unsigned long)cur->addr;
                             ^
arch/arm64/kernel/kprobes.c: In function ‘setjmp_pre_handler’:
arch/arm64/kernel/kprobes.c:495:28: error: lvalue required as left operand of assignment
  instruction_pointer(regs) = (long)jp->entry;
                            ^
At this point I can use the patched ptrace.h in Dave's tree //like less// or patch ktrace.c to fix the assignments like this //which I believe is not wrong//:

- instruction_pointer(regs) = slot;
+ (regs)->pc = slot;

I always end up with the same results:

(WITH kprobes.c FILE PATCHED and #define in .h left as original)

shell@mini-emulator-arm64:/data/local/tmp # 
init[1]: unhandled level 2 permission fault (11) at 0xffffffc0000a75cc, esr 0x8100000e
pgd = ffffffc03ddc4000
[ffffffc0000a75cc] *pgd=0000000000000000

CPU: 0 PID: 1 Comm: init Tainted: G           O 3.10.0+ #12
task: ffffffc03ec58000 ti: ffffffc03ec60000 task.ti: ffffffc03ec60000
PC is at do_no_restart_syscall+0x0/0x8
LR is at 0xffffffc03ec58940
pc : [] lr : [] pstate: 00000000
sp : ffffffc0005c0d90
x29: ffffffffffffffff x28: 0000000000000000
x27: 0000000000000000 x26: 0000000000000000
x25: 00000000000000dc x24: 0000000001200011
x23: 0000000060000000 x22: 00000000004173b4
x21: 0000007fce10e430 x20: 0000000000418104
x19: 0000007fce10e430 x18: 0000000000404298
x17: 0000000000486000 x16: 0000000000000001
x15: 0000000000000000 x14: 0000000000000000
x13: 0000007fce10e8f8 x12: 0000007fce10e500
x11: 0000000000000001 x10: 0000000000484000
x9 : 00000000004884e0 x8 : 0000000000000000
x7 : 0000000000000001 x6 : 0000000000022ab0
x5 : 15787f021d0f0000 x4 : 000000000048a000
x3 : 0000000000484000 x2 : 0000000000000014
x1 : 00000000ffffff80 x0 : 0000000000000b19

Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b

CPU: 0 PID: 1 Comm: init Tainted: G           O 3.10.0+ #12
Call trace:
[] dump_backtrace+0x0/0x12c
[] show_stack+0x10/0x1c
[] dump_stack+0x1c/0x28
[] panic+0xe4/0x208
[] do_exit+0x880/0x948
[] do_group_exit+0x3c/0xd4
[] get_signal_to_deliver+0x158/0x4cc
[] do_signal+0x544/0x570
[] do_notify_resume+0x5c/0x68


I am kind of running out of ideas, I think it's a memory issue from mm.c or kind of paging issue (protected level 2 makes me thinking of) even attaching a debugger I don't believe it helps a lot!