Embedding Python in Android

Compile C++ Cython extensions

April 28, 2014



This article is part of a tutorial series.

  1. Embedding Python in Android
  2. Java Native Interface
  3. Embedding the Python Interpreter
  4. Adding log to the Python Interpreter
  5. Include Python Standard Library
  6. Compile C++ Cython Extensions
  7. Compile C Cython Extensions

In the previous articles of this tutorial series, I've shown how to cross-compile Python from Android (using Kivy's Python-for-Android project), deploy some scripts and needed libraries to the device and start the Python interpreter. Thus, if you have pure python modules, at this time you have everything needed to deploy an embedded python Android application – just copy your python modules somewhere inside the assets/ folder and take care of system paths. However, if you want to use compiled extensions, you will have to cross-compile them to Android. In this article, I will show how you can cross-compile a C++ Cython extension to use on your embedded python interpreter in Android.

Rectangle library

For this example, I will use the Rectangle example which is in the Cython documentation (http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html). Our Rectangle library is nothing more than a C++ Class which defines a rectangle and some methods to access some properties.

First, let's define Rectangle.h:

namespace shapes {
    class Rectangle {
    public:
        int x0, y0, x1, y1;
        Rectangle(int x0, int y0, int x1, int y1);
        ~Rectangle();
        int getLength();
        int getHeight();
        int getArea();
        void move(int dx, int dy);
    };
}

and Rectangle.cpp:

#include "Rectangle.h"

using namespace shapes;

Rectangle::Rectangle(int X0, int Y0, int X1, int Y1)
{
    x0 = X0;
    y0 = Y0;
    x1 = X1;
    y1 = Y1;
}

Rectangle::~Rectangle()
{
}

int Rectangle::getLength()
{
    return (x1 - x0);
}

int Rectangle::getHeight()
{
    return (y1 - y0);
}

int Rectangle::getArea()
{
    return (x1 - x0) * (y1 - y0);
}

void Rectangle::move(int dx, int dy)
{
    x0 += dx;
    y0 += dy;
    x1 += dx;
    y1 += dy;
}

Cython wrapper

For wrapping the Rectangle library, we will use the following code in file rect.pyx. I could have used rectangle.pyx, but Cython would compile it to rectangle.cpp which would overwrite the already existent file.

(Please note that this comes directly from Cython C++ documentation, and make sure you have Cython installed in your system)

# distutils: language = c++
# distutils: sources = Rectangle.cpp

cdef extern from "Rectangle.h" namespace "shapes":
    cdef cppclass Rectangle:
        Rectangle(int, int, int, int) except +
        int x0, y0, x1, y1
        int getLength()
        int getHeight()
        int getArea()
        void move(int, int)

cdef class PyRectangle:
    cdef Rectangle *thisptr      # hold a C++ instance which we're wrapping
    def __cinit__(self, int x0, int y0, int x1, int y1):
        self.thisptr = new Rectangle(x0, y0, x1, y1)
    def __dealloc__(self):
        del self.thisptr
    def getLength(self):
        return self.thisptr.getLength()
    def getHeight(self):
        return self.thisptr.getHeight()
    def getArea(self):
        return self.thisptr.getArea()
    def move(self, dx, dy):
        self.thisptr.move(dx, dy)

The first cdef extern imports symbols from the Rectangle.h header file to Cython. The PyRectangle class is the Python wrapping class around our C++ class. Note that we have a pointer to a object of the Rectangle class which is set on PyRectangle's initialization (__cinit__), and destroyed on __dealloc__. The rest are forward calls to Rectangle methods.

If you are already familiar with Cython, you know that one of the best ways to build a Cython extension module is using the distutils process. Basically, you create a setup.py file with your distutils recipe and invoke it to build your extension. Here is my setup.py file:

from distutils.core import setup
from distutils.extension import Extension

# Test if Cython is available
try:
    from Cython.Distutils import build_ext
    USE_CYTHON = True
except ImportError:
    USE_CYTHON = False

print "USE_CYTHON =", USE_CYTHON

# If no Cython, we assume a 'rect.cpp' compiled with: 'cython -t --cplus rect.pyx'
ext = '.pyx' if USE_CYTHON else '.cpp'
extensions = [Extension('rect', ['rect' + ext], language = "c++")]

# Select Extension
if USE_CYTHON:
    from Cython.Build import cythonize
    extensions = cythonize(extensions)
else:
    # If not using Cython, we have to add 'Rectangle.cpp'
    extensions[0].sources.append('Rectangle.cpp');

setup(name = "rect", ext_modules = extensions)

Basically, this script tests for the existence of Cython on the python distribution. If it exists, it uses cythonize to build the extensions from a file named rect.pyx. If there is no Cython, the script assumes that there is a filed named rect.cpp, which is already Cython-compiled (and adds Rectangle.cpp as source file to build the library). This means that, if we do not have Cython available in our Python distribution, we can still compile a Cython extension if we have rect.cpp. This is useful for distributing Cython extensions for people that do not have Cython in their systems. It is also useful for us to compile to Android, since the Python distribution which we built to deploy to Android does not have Cython installed.

So far, we should have four files (Rectangle.h, Rectangle.cpp, rect.pyx and setup.py) which are sufficient if we want to test it locally. To build the extension in the same place as the other files are, just execute:

python setup.py build_ext --inplace

This should result with a filed named rect.so in the folder. Start the Python interpreter, import PyRectangle, and test.

Python-for-Android recipe

To cross-compile the Rectangle library for Android, we will make use again of Python-for-Android. First, in the python-for-android/ folder (which should already exist as explained in a previous article), open the recipes folder. The recipes folder is a folder where the scripts for building several python libraries are located. For instance, you can find the recipes of Python, Kivy, Numpy, etc.

In recipes, create a new folder named rectangle. Inside recipes/rectangle/ create another folder named src/. Copy the source files already created to that folder (Rectangle.h, Rectangle.cpp, rect.pyx and setup.py). On recipes/rectangle/ create a file named recipe.sh. That will be the script to build our Rectangle extension for Android. Set the contents of recipe.sh as follows:

#!/bin/bash

VERSION_rectangle=
URL_rectangle=
DEPS_rectangle=(python)
MD5_rectangle=
BUILD_rectangle=$BUILD_PATH/rectangle/rectangle
RECIPE_rectangle=$RECIPES_PATH/rectangle

function prebuild_rectangle() {
    cd $BUILD_PATH/rectangle
    try cp -a $RECIPE_rectangle/src $BUILD_rectangle
}

function shouldbuild_rectangle() {
    true
}

function build_rectangle() {
    cd $BUILD_rectangle

    push_arm

    export CFLAGS="$CFLAGS -I$ANDROIDNDK/sources/cxx-stl/stlport/stlport"
    export LDFLAGS="$LDFLAGS -L$ANDROIDNDK/sources/cxx-stl/stlport/libs/armeabi -lstlport_shared"

    # Cythonize .pyx files
    try find . -iname '*.pyx' -exec $CYTHON --cplus {} \;
    try $HOSTPYTHON setup.py build_ext -v
    try $HOSTPYTHON setup.py install -O2

    pop_arm
}

function postbuild_rectangle() {
    true
}

The interesting bits are:

  • Line 5: We are declaring that our library needs Python. This will force the compilation of Python for Android, since the compilation of native extensions need the Python headers.
  • Lines 7-8: We are setting up the location of the recipe's source and of the build path.
  • Lines 10-13: This function is called before the build happens, and we guarantee that the source files to compile are in the build location. We do that by using cp (copy).
  • Line 19: This is the function which gets called for building the extension.
  • Line 22: This is a function declared in python-for-android/distribute.sh which sets up some environment variables to cross compile. Basically, it sets some flags and the android-aeabi-gcc compiler as default C compiler.
  • Line 24: We include some header files from stlport. They are necessary to successfully compile C++ for Android.
  • Line 25: We link our extension to the stlport shared library. I will explain this in much detail further down, but we must use stlport (we could also use libgnustl_shared) if we do not want Android runtime to stall because of missing symbols. The Android runtime support for C++ is quite limited (check http://www.kandroid.org/ndk/docs/CPLUSPLUS-SUPPORT.html).
  • Line 28: We cythonize manually the .pyx files to C++ (using –cplus flag). We must use Cython manually because the Python distribution of python-for-android does not have Cython installed.
  • Lines 29-30: This is were the extension finally gets built!

To build the extension, cd? into the python-for-android* root folder, set up the environment variables as explained previously, and run:

./distribute -m "python rectangle"

If everything worked as it should, you will find rect.so in python-for-android/dist/default/private/lib/python2.7/site-packages/.

Deploy on Android

If you followed the previous articles, you should have a folder in assets/ (assets/lib/python2.7/site-packages/) where you can deploy you extensions since the python interpreter will look for extensions there (well, not really there, but on /data/data/package_name/files/lib... on the device). You must copy the generated rect.so library to that location on the assets/ folder.

To test the new library, you can add the following to the end of boot.py:

# Import a site-packages compiled module
from rect import PyRectangle
r = PyRectangle(0, 0, 10, 20)
print "PyRectangle area:", r.getArea()

Recompile you Android application, and deploy it on the device. If everything worked as expected, you should see something like the following in your logcat:

I/pyjni   (  713): import rect
I/pyjni   (  713): ImportError: Cannot load library: reloc_library[1318]:   142 cannot locate '__cxa_rethrow'...

Ok, remember that part, in the recipe, when I said that should link our Rectangle library to stlport_shared? This is what happens when we do not do that link. However, we have done that link, why do we still have this error? Again, Android has terrible C++ support! In this case, it is missing some symbols which may exist in a better C++ runtime library. Fortunately for us, the Android NDK has prebuilt libraries inside NDK_PATH/sources/cxx-stl.

In this case, since I've linked my Cython extension to stlport_shared, we have to make the libstlport_shared.so file appear in our libs/armeabi/ folder where all shared libraries should be located. There are many ways (check 1.3 in this reference, or search for "use APP_STL"), but if you include the following in your jni/Android.mk file, it works.

# Include STLPort C++ Runtime shared lib (libstlport_shared.so)
# This is for use of some C++ Python extensions
NDK_PATH = PATH/TO/NDK

include $(CLEAR_VARS)

LOCAL_MODULE := stlport_shared
LOCAL_SRC_FILES := $(NDK_PATH)/sources/cxx-stl/stlport/libs/armeabi/libstlport_shared.so

include $(PREBUILT_SHARED_LIBRARY)

Re-run ndk-build and you should have libstlport_shared.so in your libs/armeabi/ folder along with the rest of the libraries. Finally, in our Java code, we must start the library. Just add System.loadLibrary("stlport_shared") somewhere, as for instance, in PythonWrapper together with the initialization of libpython2.7.so and pyjni.so. If everything is now working fine, you should see the following in your logcat.

I/pyjni   ( 2154): PyRectangle area: 200

Congratulations, you have successfully built and deployed a C++ Cython extension for Python on Android!