Building Python 3.6 from Source on OpenSUSE like an idiot

Recently I was humbled to find out that I am less good at building things from source than I used to be.  Let me state first that what I am doing below is not something most Python users should ever want to do.  On OpenSUSE and other Linux distributions, we install our software using packages. That's your normal first way of getting python.

Your normal secondary way of getting python, and something a lot of serious python developers do, is use pyenv, which is similar in nature to the rbenv tool ruby developers use.   Both installing system distributions of Python, and installing your own local python with pyenv are generally recommended ways of getting python on Linux and unix systems.

This way I'm showing is generally NOT recommended.

Enormous things like Python are not normally manually built from sources by most users, on most common Linux distros. If you're one of those Arch-style linux or other source-driven-linux distributions, or if you're a FreeBSD-style ports BSD unix user, you're probably more used to configuring and installing things from source.  The instructions below took me a long time to figure out.

First, I have to admit that I did something really stupid. I installed directly into /usr. Never never do that. That area of the filesystem belongs to "openSUSE package management" land, it's a "system folder".   There is no "make uninstall" in python or in any other common C and unix classic "configure+make+make install" source "tarballs".

The sensible thing to do is use a prefix like /opt/mypythonjunk, which I have chosen in the example below. Then you can wipe (rm -rf /opt/mypythonjunk).

After the obvious thing (download the tarball and unpack it, cd into that directory containing the source code, here are the configuration and build steps I used.

You need some stuff installed on your computer. I'm not able to make a complete list but it includes all the developer basics, gcc,make, binutils, etc.  You also need the "dev" (devel) packages which include the libraries and headers for things.  On opensuse that means "zypper install ncurses-dev", and similar things. You must install all that stuff before you can build from sources.

I was working from ~/python/Python-3.6.0  but you can work from anywhere you like.

make clean

mkdir  -p /opt/mypythonjunk

# --enable-optimizations might be useful for some real world uses.

# --with-pydebug might be useful for some development and test uses.

./configure --with-pydebug --enable-shared --prefix=/opt/mypythonjunk

for m in _struct binascii unicodedata _posixsubprocess math pyexpat _md5 _sha1 _sha256 _sha512 select _random _socket zlib fcntl; do 

    sed -i "s/^#\(${m}\)/\1/" Modules/Setup; 

done

make

sudo make install

sudo ln -s /opt/mypythonjunk/lib64/python3.6/lib-dynload/ /opt/mypythonjunk/lib/python3.6/lib-dynload

When you read the above, translate /opt/mypythonjunk to /usr mentally in your head, and imagine the mess I made when I did that. Be smart. Don't do that.

There are instructions on Python's web site. Read those.  But they're not enough for some distros, and OpenSUSE is one of them.

The initial make clean thing is pretty standard advice in case you ran a previous build and your working copy is dirty. It's harmless if your working copy isn't dirty. 

Next, I'm making a single output folder to contain all installed stuff.  I'm configuring with a few flags that I discovered are useful in various cases, and I'm specifying where the installation target folder should be.

The for m... part is something that someone on IRC pointed me at, which is really a nice way of telling you that you need to edit that file Modules/Setup;  the edits above appear to be enabling some of the optional Python modules you might want or not want.

If you run make it will work, but if it fails in some bizarre way, try again with more verbose output ( make -p or otherwise), and save your logs for later analysis.   If you want to ask a question on #python on irc, you will want to paste your make output logs into the python channel's own pastebin service.

 Something that surprises me during the make install phase is to see a lot of gcc (compilation) and tests being run (re-run? recompiled?). I'm told on the IRC #python channel that this is due to something called PGO (profile guided optimization). It should occur if you used --enable-optimizations during the configure phase.

What tripped me up was that the python binary installed and ran and could not find readline or any other python core modules.  The ln -s line above is the fix for that, and it took two days of scratching my head to realize I should do this:

/opt/something/bin/python3.6

import sys

sys.path

Hopefully obviously the path I put above is not the correct path for you. That's a hint to put your own actual direct fully qualified path name in python.

Then look at the output above, and find what directories it's searching for, then go see if those directories exist. If not, the thing to do is either edit sys.path (no thank you!) or hack it (yes please!) with the symlink command (ln -s).

Another trick I find helps to get a recalcitrant python to start up is that you can force a PYTHONHOME value for one command:

PYTHONHOME=/opt/something /opt/something/bin/python3.6

And yet another fun trick is to start python up without installing it, like this:

LD_LIBRARY_PATH=/home/wpostma/python/Python-3.6.0rc1 ./python

I hope these tips help.  I'm posting it on my blog so that the Mighty Google will index it, and other people who are as frustrated as I got trying to get this work, might get additional tips.  Corrections, and suggestions are welcome.