Creating an embeddable Python distribution on OS X

From homebrew binaries

June 05, 2016



I've made a new blog post that shows how you can build a Python 3 distribution from the Python source code. I suggest to read the other one first and then read this one because it deals a lot with some things you may need after you compile from the source code.

I am currently working on a cross-platform Electron application that needs to make network calls to an embedded Python web server. Since this application must run on OS X and Windows, and I don't want my users to have to install Python 3 themselves, I have to include the Python interpreter with the application. Since Python 3.5 we can use the embeddable Python distribution on Windows, but there's not such thing for OS X and we have to roll our own.

This blog post documents one of several possible ways of building a standalone Python distribution on OS X that you can use to embed in other applications. I know that there are solutions such as py2app or cx_freeze, but I've never had much success with them previously. Besides, I want to embed Python on another application, not create a standalone executable (although we can do it).

Install a working Python 3 distribution

We are going to build our standalone distribution from Homebrew's Python distribution. You can probably use the official Python installer from Python.org, although I prefer to do brew install python3 on the terminal.

Homebrew will install Python 3 in /usr/local/Frameworks/Python.framework/Versions/3.5/.

These are the contents of the directories:

  • bin: includes several programs such as the python3 and python3.5 binary programs and the pyvenv-3.5 to build virtual environments.
  • Headers: symbolic link to include.
  • include: includes the python development headers, necessary if we need to compile anything against Python.
  • lib: includes symlinks for the Python shared library (symlinks are named libpython3.5.dylib and libpython3.5m.dylib) and the python standard library inside lib/python3.5.
  • Python: the python shared library.
  • Resources: includes the OS X Info.plist file and the Python launcher (Python.app).
  • share: documentation.

As a side note, if you couldn't remember where Homebrew installed Python 3, here's how you could find it:

MacBook-Pro$ which python3
/usr/local/bin/python3

MacBook-Pro$$ readlink /usr/local/bin/python3
../Cellar/python3/3.5.1/bin/python3

MacBook-Pro$ readlink /usr/local/Cellar/python3/3.5.1/bin/python3
../Frameworks/Python.framework/Versions/3.5/bin/python3

The first command shows us the location of the homebrew's python3 command. Since it is just a symlink to another symlink, we use the readlink command to find the destination. We could have used the Finder as well..

Cherry pick some files

Create an empty folder named python3.5 somewhere (I like do it on my Desktop). We are going to copy files from the Homebrew's source (I'll use the $homebrew prefix to refer to it) to build our Python distribution.

First we are going to copy the python shared lib to our destination. Copy the $homebrew/Python file and rename it to libpython3.5.dylib.

Next, copy the include/ and lib/ folders to the destination. You can delete lib/libpython3.5.dylib, lib/libpython3.5m.dylib and lib/pkgconfig as we won't be using them. If you are not going to compile anything, you can also delete the include folder.

Congratulations, you now have a portable Python distribution! Unfortunately, you cannot do much with it as you don't have any binary programs that use it.

Python Interactive Interpreter

So far, we have our distribution almost ready, but we cannot make use of it. In this section we are going to make use of the binary program which allows us to do things such as "python3.5 a_file.py" or just "python3.5" to launch the interactive interpreter.

For those with more interest in this kind of things, the source code for the python launcher can be found at https://github.com/python/cpython/blob/3.5/Programs/python.c. The most relevant line is where the application calls the main function from the dynamic library (Py_Main(argc, argv_copy)). In simple words, the python dynamic library does all the heavy work.

Although we could compile the launcher from the source code, we will use the binary from the homebrew distribution. Right click on $homebrew/Resources/Python.app and select "Show package contents". OS X apps are just plain folders. Copy the Contents/MacOSX/Python file and rename it to python3.5.

Now, execute the file. You will be greeted with the python interactive interpreter:

MacBook-Pro$ ./python3.5
Python 3.5.1 (default, Apr 18 2016, 11:55:04)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
Stop, you are being fooled!!!!!

Wanna see?

>>> import sys
>>> sys.path
['', '/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python35.zip',
 '/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5',
 '/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/plat-darwin',
 '/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/lib-dynload',
 '/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages']
>>>

This program is using homebrew's python distribution (/usr/local/Cellar/...) and not the one we have been building. If you were to distribute these files to a computer without homebrew or if you deleted your python 3, this would not work anymore. So how do we fix it?

First, let's see which dynamic library the python3.5 binary is using:

MacBook-Pro$ otool -L python3.5
python3.5:
   /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 855.17.0)
   /usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/Python (compatibility version 3.5.0, current version 3.5.0)
   /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)

As you can see, this program is using the python dynamic library from the homebrew directory! Let's change it to use libpython3.5.dylib from our distribution:

MacBook-Pro$ install_name_tool -change /usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/Python @executable_path/libpython3.5.dylib python3.5

Let's run the otool command again:

MacBook-Pro$ otool -L python3.5
python3.5:
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 855.17.0)
    @executable_path/libpython3.5.dylib (compatibility version 3.5.0, current version 3.5.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)

The program now seems to use "our" dynamic library. Let's run the interactive interpreter to make sure:

MacBook-Pro$ ./python3.5
Python 3.5.1 (default, Apr 18 2016, 11:55:04)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/Users/user/Desktop/python3.5/lib/python35.zip',
'/Users/user/Desktop/python3.5/lib/python3.5',
'/Users/user/Desktop/python3.5/lib/python3.5/plat-darwin',
'/Users/user/Desktop/python3.5/lib/python3.5/lib-dynload']
>>>

Congratulations, now we have a workable portable python distribution!

Compressing the standard library

As you can see above, python also includes a python35.zip file in sys.path. This means that we can compress the standard library to save some space.

Open lib/python3.5/, select all files and folders except lib-dynload (compiled modules) and plat-darwin (platform specific things) and add them to a zip file named python35.zip. Make sure that you create the zip file directly on lib/python3.5/ as you don't want any sub-folder between the root of the zip file and the standard lib files. Move the file to lib/python35.zip.

This is how it should be like:

Run your local python3.5 file and everything should work as expected:

MacBook-Pro$ ./python3.5
Python 3.5.1 (default, Apr 18 2016, 11:55:04)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!
>>>

Compiling a binary program

To end this post I will show how you could compile your own program to make use of this embedded python distribution that you made yourself. Let's create a file named hello.c on the same directory with the following content:

#include "Python.h"

int
main(int argc, char **argv)
{
    // Add the standard lib as zip to system path
    // Lib-dynload must be included because of zlib (we could delete the rest if we wanted!)
    wchar_t *stdlib = L"lib/python35.zip:lib/python3.5/lib-dynload";
    Py_SetPath(stdlib);

    // Initialize Python interpreter
    printf("Initializing the Python interpreter\n");
    Py_Initialize();

    // Run something
    PyRun_SimpleString("print('Hello World!')");

    // Finalize
    printf("Finalizing the Python interpreter\n");
    Py_Finalize();

    return 0;
}
Note that we are including python35.zip and lib-dynload because we are following a kind-of tutorial and I left the standard library as zip from the previous section, but we could add lib/python3.5 if we still had the uncompressed files.

Let's compile the file using the headers in include and our local shared library, and run it:

MacBook-Pro$ cc hello.c -o hello -I include/python3.5m/ -L ./ -l python3.5
MacBook-Pro$ ./hello
Initializing the Python interpreter
Hello World!
Finalizing the Python interpreter
Stop, you are being fooled again!!!!!

The thing works but let's use our otool friend again:

MacBook-Pro$ otool -L hello
hello:
    /usr/local/opt/python3/Frameworks/Python.framework/Versions/3.5/Python (compatibility version 3.5.0, current version 3.5.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)

As you can see, our "hello" binary it is using a dynamic library on /usr/local/opt/. If we were to use this in a computer without homebrew's python3, it wouldn't work anymore. There are two solutions for this:

One is to force the "hello" binary to use our libpython3.5.dylib file:

MacBook-Pro$ install_name_tool -change /usr/local/opt/python3/Frameworks/Python.framework/Versions/3.5/Python @executable_path/libpython3.5.dylib hello

The problem with this approach is that only works for this binary, or in other words, if you needed to recompile "hello.c" you would have to execute the same command again.

But what is the other alternative? Let's use the otool command on libpython3.5.dylib:

MacBook-Pro$ otool -L libpython3.5.dylib
libpython3.5.dylib:
    /usr/local/opt/python3/Frameworks/Python.framework/Versions/3.5/Python (compatibility version 3.5.0, current version 3.5.0)
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 855.17.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)

There's a /usr/local/opt which is quite suspect. Let's check in more detail:

MacBook-Pro$ otool -l libpython3.5.dylib
...
cmd LC_ID_DYLIB
  cmdsize 96
     name /usr/local/opt/python3/Frameworks/Python.framework/Versions/3.5/Python (offset 24)
...
cmd LC_LOAD_DYLIB
  cmdsize 104
     name /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (offset 24)
...
cmd LC_LOAD_DYLIB
  cmdsize 56
     name /usr/lib/libSystem.B.dylib (offset 24)

As you can see, while CoreFoundation and libSystem are being loaded (LC_LOAD_DYLIB), (usr/local/opt/.../Python) seems to be some kind of identifier (LC_ID_DYLIB). There's not much information about LC_ID_DYLIB out there, but install_name_tool has an -id parameter that we can try:

MacBook-Pro$ chmod 777 libpython3.5.dylib
MacBook-Pro$ install_name_tool -id @executable_path/libpython3.5.dylib libpython3.5.dylib

MacBook-Pro$ otool -l libpython3.5.dylib
...
cmd LC_ID_DYLIB
  cmdsize 64
     name @executable_path/libpython3.5.dylib (offset 24)

It seems to have worked, let's recompile "hello.c" and check the libraries used:

MacBook-Pro$ cc hello.c -o hello -I include/python3.5m/ -L ./ -l python3.5
MacBook-Pro$ otool -L hello
hello:
    @executable_path/libpython3.5.dylib (compatibility version 3.5.0, current version 3.5.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)

Great, now our python dynamic library is automatically identified when we compile. The only thing I notice is that since we are using @executable_path, we must have the "hello" library in the same directory as libpython3.5.dylib. If we had to move it, we would have to use the install_name_tool to point to the location of our dynamic library. This guy explains better than me alternatives to @executable_path.

I've explored this in two days, so there's probably some simpler ways of doing everything I talk about here. Leave a comment if you know about any simpler alternative..

Some remarks

It was brought to my attention that homebrew compiles the Python distribution and sets a minimum OS X version. This means that you may have problems using your distribution in 10.9 if you build it on 10.11.

MacBook-Pro$ otool -l libpython3.5.dylib
...
cmd LC_VERSION_MIN_MACOSX
  cmdsize 16
  version 10.9
  sdk 10.9

I'm still looking if it is possible to compile Python with a previous OS X sdk with Homebrew, but I've confirmed that the official Python binaries from python.org target OS X 10.6 and newer and do not have LC_VERSION_MIN_MACOSX set.

The only catch is that the dynamic library is 6MB instead of the 2MB of the homebrew version because it includes 32 and 64 bits instructions inside (fat binary). OS X provides a tool called lipo which may be used to extract only the x86_64 instructions (e.g: lipo -extract x86_64 libpython3.5.dylib -output libpython3.5.tmp.dylib). The file size of the standard library of the official distribution is also 2-3 times larger (23MB for the python35.zip file instead of 10MB for the homebrew version).

If you need to use pyvenv-3.5 and pip-3.5, you can copy the files from $homebrew/bin into your target distribution. Since these are only python files without extension, don't forget to edit and change the first line to point to the python launcher on your distribution:

Replace this -> #!/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/bin/python3.5
#!./python3.5     <- with this

if __name__ == '__main__':
    import sys
    rc = 1
    try:
        import venv
        venv.main()
        rc = 0
    except Exception as e:
        print('Error: %s' % e, file=sys.stderr)
    sys.exit(rc)

Good luck!