Android kernel module for accessing Cortex A9's cache memory

2015-02-16T00:00:00+01:00

Objectives

The aim of this module was to be able to:

enable the access to the Performance Monitor Unit (cf. PMU) control registers in user-mode
read the Cycle Counter register
trigger the cache flush operation from the external world (outside the kernel world)

Get access to the Performance Monitor Unit

According to ARM’s documentation: “The system control coprocessor, CP15, controls […] cache configuration and management.” (cf. About system control)

If we check the Registers summary one can notice that we can access to the CP15 system control registers using the instructions “MCR” and “MRC”.

“MRC” stands for “Move from ARM to Co-processor” and “MCR” for “Move from Co-Processor to ARM”.

MRC{cond} <cp#>,<op>,<ARM srce>,<lhs>,<rhs>,{info}
MCR{cond} <cp#>,<op>,<ARM dest>,<lhs>,<rhs>,{info}

where

- <cp#> is the co-processor number (0-15)
- <op> is the operation code required (0-7)
- <ARM srce>/<ARM dest> is the ARM source/destination register (0-15)
- <lhs> and <rhs> are co-processor register numbers (0-15)
- {info} is optional extra information (0-7)

(source)

Still from the Registers summary page, one can notice that one of the main functionalities of the CP15 is to provide “Performance Monitor Registers”.

From there one can see that while the CRn = C9, Op1 = 0, CRm = c13, Op2 = 0 we can access to the “Cycle Count Register”.

uint32_t r = 0;

                  <cp#>   |<op> | <ARM Reg. dest.> | <lhs>|   <rhs> |  <op2>    (1)*
                --------------------------------------------------------------------
asm volatile("MRC p15,    |  0, |  %0,             |  c9, |   c13,  |  0",      "=r"(r)); 

return r;

(*(1): 6.43.2 Extended Asm […])

However, so far, the Cycle count is not incremented and furthermore it is not accessible from the user-mode. Therefore, iwith the help of the manual regarding “4.2.24. Performance Monitor Registers” one should:

Enable the Performance Monitor Control Regisers (PMCRs) in user-mode
Enable all counters by setting to “1” the first bit of the PMC Register (enable all counters)
Enable the Cycle Count Register so it can be incremented and read

As I had the kernel source, after some grep I found out in the file “arch/arm/kernel/perf_event_v7.c” the functions that were doing what I was looking for.

I just reused some part of the code that interested me and I’ve writtern the following functions:

#define ARMV7_CCNT      31  /* Cycle counter */
#define ENABLE_CNTR (1 << ARMV7_CCNT)
#define ARMV7_PMNC_E        (1 << 0) /* Enable all counters */
#define ARMV7_CNTENS_C (1 << ARMV7_CCNT)

static void enable_cpu_counters(void* data)
{
    /* Enable user-mode access to counters. */
    asm volatile("mcr p15, 0, %0, c9, c14, 0" :: "r"(1));
    /* Enable all counters */
    asm volatile("mcr p15, 0, %0, c9, c12, 0" : : "r"(ARMV7_PMNC_E));
    asm volatile("mcr p15, 0, %0, c9, c12, 1" : : "r" (ARMV7_CNTENS_C));
}

static void disable_cpu_counters(void* data)
{
    /* disable all counters */
    asm volatile("mcr p15, 0, %0, c9, c12, 0" : : "r" (0));
    asm volatile("mcr p15, 0, %0, c9, c12, 2" : : "r" (ARMV7_CNTENS_C));
    /* Disable user-mode access to counters. */
    asm volatile("mcr p15, 0, %0, c9, c14, 0" :: "r"(0));
}

...

//then later when I want to enable the counters...
on_each_cpu(enable_cpu_counters, NULL, 1);

...

//or disable them
on_each_cpu(disable_cpu_counters, NULL, 1);

I embedded this code within a kernel module in order to enable the CPU cycle counter after inserting the module using the command “insmod”. The kernel module source code is as follows:

#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/version.h>


#define ARMV7_CCNT      31  /* Cycle counter */
#define ENABLE_CNTR (1 << ARMV7_CCNT)
#define ARMV7_PMNC_E        (1 << 0) /* Enable all counters */
#define ARMV7_CNTENS_C (1 << ARMV7_CCNT)

static void enable_cpu_counters(void* data)
{
    printk("[+]" KERN_INFO "[" DRVR_NAME "] enabling user-mode PMU access on CPU #%d", smp_processor_id());
    /* Enable user-mode access to counters. */
    asm volatile("mcr p15, 0, %0, c9, c14, 0" :: "r"(1));
    /* Enable all counters */
    asm volatile("mcr p15, 0, %0, c9, c12, 0" : : "r"(ARMV7_PMNC_E));
    asm volatile("mcr p15, 0, %0, c9, c12, 1" : : "r" (ARMV7_CNTENS_C));
}

static void disable_cpu_counters(void* data)
{
    printk(KERN_INFO "[" DRVR_NAME "] disabling user-mode PMU access on CPU #%d",smp_processor_id());
    /* Program PMU and disable all counters */
    asm volatile("mcr p15, 0, %0, c9, c12, 0" : : "r" (0));
    asm volatile("mcr p15, 0, %0, c9, c12, 2" : : "r" (ARMV7_CNTENS_C));
    /* Disable user-mode access to counters. */
    asm volatile("mcr p15, 0, %0, c9, c14, 0" :: "r"(0));
}

int init_module(void)
{
    printk("[+] " KERN_INFO "[" DRVR_NAME "] Module initialisation\n");
    on_each_cpu(enable_cpu_counters, NULL, 1);
    return 0;
}

void cleanup_module(void)
{
    printk("[+]" KERN_INFO "[" DRVR_NAME "] Module unloading\n");
    on_each_cpu(disable_cpu_counters, NULL, 1);
}

MODULE_AUTHOR("razaina.");
MODULE_LICENSE("GPL");
MODULE_VERSION("1");
MODULE_DESCRIPTION("Enables user-mode access to ARMv7 PMU counters.);

Therefore on a higher level, from my Android native project (in user-mode) I can read the Cycle Count register.

Read the Cycle Count Register

Earlier in the previous section, I already showed the assembler instruction to execute in order to get the Cycle Count Register.

Hereafter the function I use within my native project:

static inline uint32_t
get_cycle_count(void)
{
        uint32_t r = 0;
        asm volatile("mrc p15, 0, %0, c9, c13, 0" : "=r"(r) );
        return r;
}

Trigger the flush operation

After grepping through the kernel sources, i found a file “arch/arm/include/asm/cacheflush.h” which ccontains the following functions:

flush_icache_all() //Clean entire Instruction cache
flush_kern_all() // Clean the entire cache (only L1 I guess..)
flush_user_all() //Clean user space cache entries
etc....

There was a particular function which was “flush_all_cpu_caches()” which refers to “flush_cache_all()”. The latter actually refers to “__cpuc_flush_kern_all” which refers to the “flush_kern_all()” function.

In the header of the current source file, there was the following description:

 *  The arch/arm/mm/cache-*.S and arch/arm/mm/proc-*.S files
 *  implement these methods.

In my case the corresponding files that contain the implementation (assembler) of these different functions was “arch/arm/mm/proc-arm926.S” and “arch/arm/mm/cache-v7”.

Therefore by including the “asm/cacheflush.h” file in my kernel module, I could use the “flush_cache_all()” function.

However, my objective was to trigger the flush operation from the outside world, namely from my native project.

The following tutorial helped me for creating a “character driver” so I could do the following steps in order to trigger any operation through my kernel module:

my native program write a number (a character) within the device “/dev/mydriver”
the kernel module is able to detect different events that is happening on “mydriver”
the kernel module just read the written character and an operation is executed accordingly

The whole source code for the kernel module is as follows:

#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/smp.h>
#include <asm/smp.h>
#include <asm/cacheflush.h>
#include <linux/proc_fs.h>
#include <linux/miscdevice.h>
#include <linux/fs.h>
#include <linux/init.h>
#include <linux/slab.h> /* kmalloc() */
#include <linux/fcntl.h> /* O_ACCMODE */
#include <asm/system.h> /* cli(), *_flags */
#include <asm/uaccess.h> /* copy_from/to_user */
#include <linux/version.h>

#define PERF_DEF_OPTS (1 | 16)
#define PERF_OPT_RESET_CYCLES (2 | 4)
#define procfs_name "mydriver" // /dev/mydriver
#define FLUSHALL 3

#define ARMV7_CCNT      31  /* Cycle counter */
#define ENABLE_CNTR (1 << ARMV7_CCNT)
#define ARMV7_PMNC_E        (1 << 0) /* Enable all counters */
#define ARMV7_CNTENS_C (1 << ARMV7_CCNT)


#if !defined(__arm__)
#error Module can only be compiled on ARM machines.
#endif

extern long simple_strtol(const char *,char **,unsigned int);
void memory_exit(void);
int memory_init(void);
int memory_open(struct inode *inode, struct file *filp); 
int memory_release(struct inode *inode, struct file *filp); 
ssize_t memory_read(struct file *filp, char *buf, 
                    size_t count, loff_t *f_pos);
ssize_t memory_write( struct file *filp, char *buf,
                      size_t count, loff_t *f_pos);


/* Structure that declares the usual file */
/* access functions */
struct file_operations memory_fops = {
  read: memory_read,
  write: memory_write,
  open: memory_open,
  release: memory_release
};

/* Global variables of the driver */
/* Major number */
int memory_major = 69;
/* Buffer to store data */
char *memory_buffer;

int memory_open(struct inode *inode, struct file *filp) {
  /* Success */
    printk("[+]" KERN_INFO "[" DRVR_NAME "] Opening device !\n");
  return 0;
}

int memory_release(struct inode *inode, struct file *filp) {
  /* Success */
  return 0;
}

ssize_t memory_read(struct file *filp, char *buf, 
                    size_t count, loff_t *f_pos) { 
  /* Transfering data to user space */ 
  copy_to_user(buf,memory_buffer,1);

  /* Changing reading position as best suits */ 
  if (*f_pos == 0) { 
    *f_pos+=1; 
    return 1; 
  } else { 
    return 0; 
  }
}

ssize_t memory_write( struct file *filp, char *buf,
                      size_t count, loff_t *f_pos) {

  char *tmp;
  char *endptr;
  char ret;

  tmp=buf+count-1;
  copy_from_user(memory_buffer,tmp,1);
  ret = simple_strtol(memory_buffer, &endptr, 10);
  if(endptr == NULL)
  {
      printk("[+]" KERN_INFO "[" DRVR_NAME "] Failed to read an integer!\n");
  }
  else
  {
      if(ret == FLUSHALL)
      {
            printk("[+]" KERN_INFO "[" DRVR_NAME "] FLUSH CACHE ALL");
            flush_cache_all();
      }
      
  }
  return 1;
}

/* Called when a process tries to open the device file, like
 * "cat /dev/mycharfile"
 */
static int mydriver_device_open(struct inode *inode, struct file *file)
{
    printk("[+]" KERN_INFO "[" DRVR_NAME "] opening mydriver device.\n");
    return 0;
}

static void enable_cpu_counters(void* data)
{
    printk("[+]" KERN_INFO "[" DRVR_NAME "] enabling user-mode PMU access on CPU #%d", smp_processor_id());
    /* Enable user-mode access to counters. */
    asm volatile("mcr p15, 0, %0, c9, c14, 0" :: "r"(1));
    /* Enable all counters */
    asm volatile("mcr p15, 0, %0, c9, c12, 0" : : "r"(ARMV7_PMNC_E));
    asm volatile("mcr p15, 0, %0, c9, c12, 1" : : "r" (ARMV7_CNTENS_C));
}

static void disable_cpu_counters(void* data)
{
printk("[+]" KERN_INFO "[" DRVR_NAME "] disabling user-mode PMU access on CPU #%d",
smp_processor_id());
    asm volatile("mcr p15, 0, %0, c9, c12, 0" : : "r" (0));
    asm volatile("mcr p15, 0, %0, c9, c12, 2" : : "r" (ARMV7_CNTENS_C));
    /* Disable user-mode access to counters. */
    asm volatile("mcr p15, 0, %0, c9, c14, 0" :: "r"(0));
}


int init_module(void)
{
    printk("[+]" KERN_INFO "[" DRVR_NAME "] Module initialisation\n");
    on_each_cpu(enable_cpu_counters, NULL, 1);

    int result = register_chrdev(memory_major, "mydriver", &memory_fops);
    if (result < 0) {
        printk(
          "[+]" KERN_INFO "[" DRVR_NAME "]: cannot obtain major number %d\n", memory_major);
        return result;
    }

      /* Allocating memory for the buffer */
      memory_buffer = kmalloc(1, GFP_KERNEL); 
      if (!memory_buffer) { 
        result = -ENOMEM;
        goto fail; 
      } 
      memset(memory_buffer, 0, 1);

      printk("[+]" KERN_INFO "[" DRVR_NAME "] Inserting memory module\n"); 
      return 0;

      fail: 
        memory_exit(); 
        return result;
        return 0;
}

void memory_exit(void) {
  /* Freeing the major number */
  unregister_chrdev(memory_major, "mydriver");

  /* Freeing buffer memory */
  if (memory_buffer) {
    kfree(memory_buffer);
  }

  printk("[+]" KERN_INFO "[" DRVR_NAME "] Removing module\n");

}


void cleanup_module(void)
{
    printk(KERN_INFO "[" DRVR_NAME "] Module unloading\n");
    on_each_cpu(disable_cpu_counters, NULL, 1);
    memory_exit();
}

MODULE_AUTHOR("razaina.");
MODULE_LICENSE("GPL");
MODULE_VERSION("1");
MODULE_DESCRIPTION("Enables user-mode access to ARMv7 PMU counters + Trigger the
flush_cache_all() function.");

Using OpenSSL as a static library within an Android native project

2015-02-12T00:00:00+01:00

Install OpenSSL

First off, I strictly followed the steps described here » OpenSSL’s Wiki « with the latest version of OpenSSL at this time: 1.0.2.

Before to configure, compile, install, you need to properly set the cross-compilation environment. The “setenv-android.sh” script is provided within the link above.

Below are the different variables/piece of code I changed/added to fit my needs:

_ANDROID_NDK="android-ndk-r10d"
_ANDROID_EABI="arm-linux-androideabi-4.9"
_ANDROID_ARCH=arch-arm
_ANDROID_API="android-19"
...
export ANDROID_SDK_ROOT="/home/razaina/android-sdks"
export ANDROID_NDK_ROOT="/home/razaina/android-ndk-r10d"
...
# In case the following loop fails while running the script
for tool in $ANDROID_TOOLS
do
    ...
done

#Just replace it like so
for tool in arm-linux-androideabi-gcc arm-linux-androideabi-ranlib arm-linux-androideabi-ld
do
    ...
done

Once OpenSSL compiled, it will be installed in the following directory: /usr/local/ssl/android-19.

Integrate OpenSSL within your Android native project

You just need to properly configure your Android make file as follows:

LOCAL_PATH := $(call my-dir)

# Prepare the SSL static library
include $(CLEAR_VARS)
LOCAL_MODULE := libssl-prebuilt
LOCAL_SRC_FILES := /usr/local/ssl/android-19/lib/libssl.a 
LOCAL_EXPORT_C_INCLUDES := /usr/local/ssl/android-19/include
include $(PREBUILT_STATIC_LIBRARY)

# Prepare the CRYPTO static library
include $(CLEAR_VARS)
LOCAL_MODULE := libcrypto-prebuilt
LOCAL_SRC_FILES := /usr/local/ssl/android-19/lib/libcrypto.a
LOCAL_EXPORT_C_INCLUDES := /usr/local/ssl/android-19/include
include $(PREBUILT_STATIC_LIBRARY)

# Compile your executable
include $(CLEAR_VARS)
LOCAL_MODULE    := program 
LOCAL_SRC_FILES := program.c 
LOCAL_ARM_MODE := arm
LOCAL_CFLAGS := -g
LOCAL_C_INCLUDES := /usr/local/ssl/android-19/include
LOCAL_LDLIBS := -L/usr/local/ssl/android-19/lib -llog
LOCAL_STATIC_LIBRARIES := libssl-prebuilt libcrypto-prebuilt
include $(BUILD_EXECUTABLE)

Build Kernel Module for: Samsung Galaxy SIII (GT I9305)

2015-01-27T00:00:00+01:00

Here are the infos regarding the phone:

Model Number: GT-I9305
CyanogenMod version: 11-20150125-NIGHTLY-i9305
Android Version: 4.4.4
Baseband Version: I9305XXUENG1
Kernel Version: 3.0.64-CM-gc426a2f build02@cyanogenmod #1 Sun Jan 25 13:36:34 PST 2015
CPU: ARMv7 Processor rev0 (v7)
Build Number: cm_i9305-userdebug 4.4.4 KTU84Q 00e9a874q7 test-keys

Step 1: Get the sources of the kernel

Sources code can be found on Cyanogendmod’s website

Step 2: Prepare the kernel

Configure the kernel

If the running kernel has been configured to have /proc/config.gz then pull it in order to extract the configuration file. Otherwise check in the following directory the one that will fit your kernel: ../android_kernel_samsung_smdk4412/arch/arm/configs.

I picked the following default configuration file “cyanogenmod_i9305_defconfig”.

Then run the make command as follows:

export
PATH=$PATH:$NDK_HOME/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86/bin
make ARCH=arm CROSS_COMPILE=arm-linux-androideabi- cyanogenmod_i9300_defconfig

Compile the kernel

First of all, I had to modify the Makefile:

At the beginning of the file set the correct version so all modules will be compiled with the right version number.

VERSION = 3
PATCHLEVEL = 0
SUBLEVEL = 64
EXTRAVERSION = -CM-gc426a2f
NAME = Sneaky Weasel

Based on this blog post I also added the proper parameter while the “setlocalversion” script is called so the EXTRAVERSION is not ovewritten by a default version:

# Store (new) KERNELRELASE string in include/config/kernel.release
include/config/kernel.release: include/config/auto.conf FORCE
	$(Q)rm -f $@
	$(Q)echo "$(KERNELVERSION)$$($(CONFIG_SHELL) $(srctree)/scripts/setlocalversion --save-scmversion $(srctree))" > $@

Based on this diff I modified accordingly my Makefile. Otherwise I had the following error:

arm-linux-androideabi-ld: error: arch/arm/boot/compressed/piggy.gzip.o: unknown CPU architecture
arm-linux-androideabi-ld: error: arch/arm/boot/compressed/lib1funcs.o: unknown CPU architecture
make[4]: *** [arch/arm/boot/compressed/vmlinux] Error 1
make[3]: *** [arch/arm/boot/compressed/vmlinux] Error 2

Finally we can compile the kernel;

make ARCH=arm CROSS_COMPILE=arm-linux-androideabi-

Step 3: Compile the module

The source code of my module is as follows:

#include <linux/module.h>
#include <linux/kernel.h>

int init_module(void)
{
    printk(KERN_INFO "+++++++++++++++++++++++++  Hello Kernel !!! +++++++++++++++++++++\n");
    return 0;
}


void cleanup_module(void)
{
    printk(KERN_INFO "++++++++++++++++++++++++++++ Goodbye Kernel !!!+++++++++++++++++++++\n");

}

MODULE_AUTHOR("razaina");
MODULE_LICENSE("GPL");
MODULE_VERSION("1");
MODULE_DESCRIPTION("Prints hello world!");

My Makefile looks like:

obj-m := hello.o
KDIR := /home/razaina/Android-kernel-sources/android_kernel_samsung_smdk4412/
PWD := $(shell pwd)
CCPATH := /home/razaina/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86/bin/
default:
	$(MAKE) CFLAGS_MODULE=-fno-pic ARCH=arm CROSS_COMPILE=$(CCPATH)/arm-linux-androideabi- -C $(KDIR) M=$(PWD) modules

Do not forget the “-fno-pic” flag otherwise your module won’t work and you’ll get the following error message in dmesg:

c0 your_module_name: unknown relocation: 3

You can also get the following:

Unknown symbol _GLOBAL_OFFSET_TABLE_ (err 0)

This is because if you do not explicitely tell the compiler to build the module with “NO Position-Independent Code”, then your module will try to look for the “GLOBAL_OFFSET_TABLE” while trying to call some shared functions.

Then compile external modules in order to create the file “Module.symvers” in the kernel’s root directory.

make ARCH=arm CROSS_COMPILE=arm-linux-androideabi- modules_prepare
make ARCH=arm CROSS_COMPILE=arm-linux-androideabi- modules

Finally…compile your module using the make command.

Use NDK within Android Studio

2015-01-22T00:00:00+01:00

Two files have to be updated to make Android-studio automatically ndk-build:

build.gradle
local.properties

The default content of my build.gradle file is as follows:

apply plugin: 'com.android.application'

android {
    compileSdkVersion 21
    buildToolsVersion "21.1.2"

    defaultConfig {
        applicationId "fr.razaina.myapp"
        minSdkVersion 8
        targetSdkVersion 21
        versionCode 1
        versionName "1.0"
    }
    buildTypes {
        release {
            minifyEnabled false
            proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
        }
    }
}

dependencies {
    compile fileTree(dir: 'libs', include: ['*.jar'])
    compile 'com.android.support:appcompat-v7:21.0.3'
}

You just have to specify the name of the module you want to compile. In addition, in my example I did not forget to add the “log” as I am using it most of the time in my projects.

apply plugin: 'com.android.application'

android {
    compileSdkVersion 21
    buildToolsVersion "21.1.2"

    defaultConfig {
        applicationId "fr.razaina.myapp"
        minSdkVersion 8
        targetSdkVersion 21
        versionCode 1
        versionName "1.0"

        ######NDK=====
        ndk{
            moduleName "myLibName"
            ldLibs "log"
        }
    }
    buildTypes {
        release {
            minifyEnabled false
            proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
        }
    }
}

dependencies {
    compile fileTree(dir: 'libs', include: ['*.jar'])
    compile 'com.android.support:appcompat-v7:21.0.3'
}

Finally you have to specify in your local.properties file the path to NDK’s directory such as follows:

## This file is automatically generated by Android Studio.
# Do not modify this file -- YOUR CHANGES WILL BE ERASED!
#
# This file should *NOT* be checked into Version Control Systems,
# as it contains information specific to your local configuration.
#
# Location of the SDK. This is only used by Gradle.
# For customization when using a Version Control System, please read the
# header note.
sdk.dir=/home/razaina/android-sdks
ndk.dir=/home/razaina/android-ndk-r10d