Creating an embeddable Python distribution on OS X

From python sources

June 06, 2016

Yesterday I published a blog post on how to create a standalone python 3.5 distribution from the homebrew binaries. However, as I mentioned in the final remarks, the compiled dynamic library has a minimum OS X version which matches the OS X version where you compile it. This means that if you compile your homebrew python distribution in OS X 10.11, the minimum version is set to 10.11 and you may have problems running your dynamic library on previous OS X versions. Also, the method that I described is a little bit hackish and you have little control on the final result as you may want a static library instead of a dynamic library. In this post I will show how you can create an embeddable Python distribution from the Python source code.

Compiling Python 3.5 from sources

First download the source release from Python.org and extract it to a location on your hard disk. I usually like to extract such things to ~/Desktop/ as it is faster to open with Finder.

Open a terminal window in the extracted folder and run the following command:

./configure MACOSX_DEPLOYMENT_TARGET=10.8 --prefix=/path/to/folder/out

Basically we are defining the OS X target sdk and the folder which we want Python to be installed at. If you do not set the prefix, it will assume that you want install Python in /usr/local and it will probably mess with your local Python distributions.

The configure command generates a Makefile that you can use to make further changes:

# Generated automatically from Makefile.pre by makesetup.
# Top-level Makefile for Python
#
# As distributed, this file is called Makefile.pre.in; it is processed
# into the real Makefile by running the script ./configure, which
# replaces things like @spam@ with values appropriate for your system.
# This means that if you edit Makefile, your changes get lost the next
# time you run the configure script.  Ideally, you can do:
#
#   ./configure
#   make
#   make test
#   make install
#
# If you have a previous version of Python installed that you don't
# want to overwrite, you can use "make altinstall" instead of "make
# install".  Refer to the "Installing" section in the README file for
# additional details.
#
# See also the section "Build instructions" in the README file.

(...)

# Install prefix for architecture-independent files
prefix=/path/to/folder/out

(...)

# Deployment target selected during configure, to be checked
# by distutils. The export statement is needed to ensure that the
# deployment target is active during build.
MACOSX_DEPLOYMENT_TARGET=10.8
export MACOSX_DEPLOYMENT_TARGET

(...)

To compile, run:

make altinstall

Wait a while and in out/ you will find your compiled Python distribution.

These are the contents of the directories:

bin: includes several programs such as the python3.5 launcher and pip-3.5 and pyenv-3.5 scripts to play with virtual environments.
include: includes the python development headers, necessary if we need to compile anything against Python.
lib: includes a static library (libpython3.5.a) and the python standard library inside lib/python3.5.
share: includes documentation.

By default, the bin/python3.5 launcher is built as a static library. There's two ways to confirm that: the file size is around 3MB and if you run the otool command, you can check that it is self-contained as it only depends on system libraries:

MacBook-Pro$ cd out
MacBook-Pro$ otool -L bin/python3.5
bin/python3.5:
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 855.17.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)

Later I will show how to build python as dynamic library but for now let's be sure that everything is working and is self-contained:

MacBook-Pro$ ./bin/python3.5
Python 3.5.1 (default, Jun  6 2016, 16:44:14)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/Users/jventura/Desktop/Python-3.5.1/out/lib/python35.zip',
'/Users/jventura/Desktop/Python-3.5.1/out/lib/python3.5',
'/Users/jventura/Desktop/Python-3.5.1/out/lib/python3.5/plat-darwin',
'/Users/jventura/Desktop/Python-3.5.1/out/lib/python3.5/lib-dynload',
'/Users/jventura/.local/lib/python3.5/site-packages',
'/Users/jventura/Desktop/Python-3.5.1/out/lib/python3.5/site-packages']

As you can see, by default all paths in sys.path are located on our out/ folder. You can move your Python distribution to any other location in your hard disk and things will mostly work fine because the /bin/python3.5 file is self-contained.

Compressed standard library

Similar to what I've done on my previous post, we can compress the standard library to save some disk space. If you want, you can even delete unnecessary modules from the standard library.

Open lib/python3.5/ on Finder and select all files and folders except config-3.5m (configuration files), lib-dynload (compiled modules), plat-darwin (platform specific things) and site-packages (external modules such as pip and setuptools). Then, add them to a zip file named python35.zip. Make sure that you create the zip file with lib/python3.5/ as root as you don't want any sub-folder between the root of the zip file and the standard lib files. Move the file to lib/python35.zip.

I suggest to delete all pycache files using find . -name "__pycache__" -exec rm -rf {} \; before zipping them.

This is how it should look like:

If you want to save more space, you can safely delete config-3.5m and plat-darwin before deleting other standard library modules. For what I could see so far, for Python 3 you need at least the encodings and some codecs in lib-dynload which are imported from encodings. Without those your Python interpreter will refuse to start. In Python 2 you didn't even need the standard library.

Another optimization that you can do is to include only .pyc files in the zip (or on lib/python3.5/) which are faster to load and prevents the interpreter from having to build the pycache folders. Here's a sequence of commands to generate the .pyc files and copying them into out/lib/tmp/:

cd out/

# Create legacy .pyc files
./bin/python3.5 -m compileall -b -f lib/python3.5/
# Delete pycaches
find ./lib/python3.5 -name "__pycache__" -exec rm -rf {} \;
# Copy legacy .pyc files and then delete them
find ./lib/python3.5 -name "*.pyc" -exec ditto {} ./lib/tmp/{} \; -exec rm {} \;

Now you just need to delete or zip the .pyc files you need..

Build as shared library

For some cases you may need to compile Python as a shared library, as for instance, to compile Python extensions against libpython. To build it as shared library, open a terminal window in the folder where you've extracted the Python source code and do:

./configure MACOSX_DEPLOYMENT_TARGET=10.8 --prefix=/path/to/folder/out --enable-shared
make altinstall

Basically you are configuring a Makefile similar to the one above with --enable-shared in CONFIG_ARGS:

(...)

# Install prefix for architecture-independent files
prefix=/path/to/folder/out

(...)

# Deployment target selected during configure, to be checked
# by distutils. The export statement is needed to ensure that the
# deployment target is active during build.
MACOSX_DEPLOYMENT_TARGET=10.8
export MACOSX_DEPLOYMENT_TARGET

(...)

# configure script arguments
CONFIG_ARGS='MACOSX_DEPLOYMENT_TARGET=10.8' '--prefix=/path/to/folder/out' '--enable-shared'

The out/ folder has the same structure as before, but now you can find the shared library in lib/libpython3.5m.dylib. Also, the binary python launcher at bin/python3.5 is now only 9KB since it links to the shared lib.

The only problem that you now have with the shared library is that every binary linked to it expects that the dynamic library to be at /path/to/folder/out/lib/libpython3.5m.dylib. If you move it from there, you will have to change the path in the calling binaries. For instance, this is what you get if you use otool on bin/python3.5

MacBook-Pro-de-Joao:out jventura$ otool -L ./bin/python3.5
./bin/python3.5:
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 855.17.0)
    /path/to/folder/out/lib/libpython3.5m.dylib (compatibility version 3.5.0, current version 3.5.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)

As you can see, the location of libpython3.5m.dylib is hardcoded on the calling binary. You will have to use tools such as install_name_tool to change that reference. Check my other post for examples on how to use it.

Virtual environments

The virtual environments scripts (pyvenv-3.5 and pip3.5) are located in out/bin. They are completely usable but you just need to take care that if you do any changes on the folders, such as moving the distribution to another folder, you will need change the first line of those scripts as they are using the hardcoded location of /bin/python3.5:

#!/bin/sh
"exec" "`dirname $0`/python3.5" "$0" "$@"

# -*- coding: utf-8 -*-
import re
import sys

from pip import main

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

This guarantees that pyvenv-3.5 and pip3.5 will always use the python3.5 interpreter that is in the same directory as they are.

Final remarks

This is the starting point for building your own customized python distribution. You may want to join all files in a directory, change the location of compiled modules to another path or compile a new python launcher which links against libpython. You will probably need to mess somewhat with otool, install_name_tool and sys.path. I have written about many of these things in my other post, so check it if you need some tips.

Have fun!