slides

z, ?	toggle help (this)
space, →	next slide
shift-space, ←	previous slide
d	toggle debug mode
## <ret>	go to slide #
c, t	table of contents (vi)
f	toggle footer
r	reload slides
n	toggle notes
p	run preshow

loading presentation...

Reverse-engineering

Ian Bicking's brain

inside pip and virtualenv

Carl J Meyer

@carljm

carl@oddbird.net

virtualenv

From scratch!

What should it do?

Fully isolated environments.
Use its own site-packages directory
...for everything (imports and installation).
Ignore the system site-packages.

`site-packages`

Where third-party installed modules go.
e.g. /usr/lib/python2.6/site-packages/
(Standard library in /usr/lib/python2.6)

`site-packages`

How does it work?
Python imports site.py at startup.
e.g. usr/lib/python2.6/site.py
Or Lib/site.py in the Python source.

`site.py`

# Prefixes for site-packages;
PREFIXES = [sys.prefix, sys.exec_prefix]

def getsitepackages():
    """Returns a list containing all global site-packages
    directories (and possibly site-python).

    For each directory present in the global ``PREFIXES``,
    this function will find its `site-packages` subdirectory
    depending on the system environment, and will return a
    list of full paths.
    """
    sitepackages = []
    seen = set()

    for prefix in PREFIXES:
        if not prefix or prefix in seen:
            continue
        seen.add(prefix)

        if sys.platform in ('os2emx', 'riscos'):
            sitepackages.append(os.path.join(prefix, "Lib", "site-packages"))
        elif os.sep == '/':
            sitepackages.append(os.path.join(prefix, "lib",
                                        "python" + sys.version[:3],
                                        "site-packages"))
            sitepackages.append(os.path.join(prefix, "lib", "site-python"))
        else:
            sitepackages.append(prefix)
            sitepackages.append(os.path.join(prefix, "lib", "site-packages"))
        if sys.platform == "darwin":
            # for framework builds *only* we add the standard Apple
            # locations.
            from sysconfig import get_config_var
            framework = get_config_var("PYTHONFRAMEWORK")
            if framework and "/%s.framework/"%(framework,) in prefix:
                sitepackages.append(
                        os.path.join("/Library", framework,
                            sys.version[:3], "site-packages"))
    return sitepackages

Where does `sys.prefix` come from?

Python/sysmodule.c
Down the rabbit hole...
Modules/getpath.c

`Modules/getpath.c`

#ifndef LANDMARK
#define LANDMARK "os.py"
#endif

static int
search_for_prefix(char *argv0_path, char *home)
{
/* ... */

    /* Search from argv0_path, until root is found */
    copy_absolute(prefix, argv0_path);
    do {
        n = strlen(prefix);
        joinpath(prefix, lib_python);
        joinpath(prefix, LANDMARK);
        if (ismodule(prefix))
            return 1;
        prefix[n] = '\0';
        reduce(prefix);
    } while (prefix[0]);

The hunt for `sys.prefix`

If PYTHONHOME is set, that's sys.prefix.
Otherwise...

Starting from the location of the Python binary, search upwards.
At each step, look for lib/pythonX.X/os.py.
If it exists, we've found sys.prefix.
If we never find it, fallback on hardcoded configure --prefix from build.

Yes, `os.py` is the hardcoded landmark.

`Modules/getpath.c`

#ifndef LANDMARK
#define LANDMARK "os.py"
#endif

Let's do this.

$ mkdir scratch; cd scratch

Copy the Python binary.

$ mkdir bin

$ cp /usr/bin/python bin/

$ tree
.
`-- bin
    `-- python

$ ./bin/python -c "import sys; print(sys.prefix)"
/usr

`sys.prefix` is still `/usr`.

No PYTHONHOME set.
Binary is in /home/carljm/scratch/bin/.
No /home/carljm/scratch/bin/lib/python2.6/os.py
No /home/carljm/scratch/lib/python2.6/os.py
No /home/carljm/lib/python2.6/os.py
No /home/lib/python2.6/os.py
/usr is the fallback build prefix.

Create the landmark.

$ mkdir -p lib/python2.6

$ touch lib/python2.6/os.py

$ tree
.
|-- bin
|   `-- python
`-- lib
    `-- python2.6
        `-- os.py

$ ./bin/python -c "import sys; print(sys.prefix)"
'import site' failed; use -v for traceback
/home/carljm/scratch

$ ./bin/python -c "import sys; print(sys.path)"
'import site' failed; use -v for traceback
['',
 '/home/carljm/scratch/lib/python2.6/',
 '/home/carljm/scratch/lib/python2.6/plat-linux2',
 '/home/carljm/scratch/lib/python2.6/lib-tk',
 '/home/carljm/scratch/lib/python2.6/lib-old',
 '/usr/lib/python2.6/lib-dynload']

Fixing `sys.exec_prefix`

$ mkdir lib/python2.6/lib-dynload

$ ./bin/python -c "import sys; print(sys.exec_prefix)"
'import site' failed; use -v for traceback
/home/carljm/scratch

$ ./bin/python -c "import sys; print(sys.path)"
'import site' failed; use -v for traceback
['',
 '/home/carljm/scratch/lib/python2.6/',
 '/home/carljm/scratch/lib/python2.6/plat-linux2',
 '/home/carljm/scratch/lib/python2.6/lib-tk',
 '/home/carljm/scratch/lib/python2.6/lib-old',
 '/home/carljm/scratch/lib/python2.6/lib-dynload']

We have an isolated virtualenv!

It's broken.
No standard library.
No site.py, thus no site-packages at all.

How could we fix it?

Copy the entire stdlib (including site.py) into our lib/python2.6.
Works, but heavy.
We're not going to modify the standard library; could we just use the system one?

Bootstrap like a `virtualenv`.

Write orig-prefix.txt to lib/python2.6.
Contains the system Python's sys.prefix.
Place our own, modified site.py in lib/python2.6.
Our site.py reads orig-prefix.txt and adds system stdlib paths to sys.path.

$ tree
.
|-- bin
|   `-- python
`-- lib
    `-- python2.6
        |-- lib-dynload
        |-- orig-prefix.txt
        |-- os.py
        `-- site.py

From virtualenv's modified `site.py`

def virtual_install_main_packages():
    f = open(os.path.join(os.path.dirname(__file__),
                          'orig-prefix.txt'))
    sys.real_prefix = f.read().strip()
    # ...
    if sys.platform == 'win32':
        paths = [os.path.join(sys.real_prefix, 'Lib'),
                 os.path.join(sys.real_prefix, 'DLLs')]
    # ...
    else:
        paths = [os.path.join(sys.real_prefix,
                              'lib',
                              'python'+sys.version[:3])]
    # ...
    sys.path.extend(paths)

But `site.py` depends on the standard library!

We have a bootstrapping problem.
Solution: symlink (or copy) just the stdlib bits that we need to bootstrap site.py.
After site.py runs, we'll have access to the full stdlib thanks to its sys.path additions.

From `virtualenv.py`

REQUIRED_MODULES = [
    'os', 'posix', 'posixpath', 'nt', 'ntpath', 'genericpath',
    'fnmatch', 'locale', 'encodings', 'codecs',
    'stat', 'UserDict', 'readline', 'copy_reg', 'types',
    're', 'sre', 'sre_parse', 'sre_constants', 'sre_compile',
    'zlib']

def copy_required_modules(dst_prefix):
    import imp
    for modname in REQUIRED_MODULES:
        # ...
        try:
            f, filename, _ = imp.find_module(modname)
        except ImportError:
            logger.info("Cannot import bootstrap module: %s" % modname)
        else:
            if f is not None:
                f.close()
            dst_filename = change_prefix(filename, dst_prefix)
            copyfile(filename, dst_filename)
            # ...

We can also add the global `site-packages`, if we want it.

`site.py`

def virtual_addsitepackages(known_paths):
    # ...
    return addsitepackages(known_paths,
                           sys_prefix=sys.real_prefix)

# ...

def main():
    # ...
    if GLOBAL_SITE_PACKAGES:
        paths_in_sys = virtual_addsitepackages(paths_in_sys)
    # ...

What makes a virtualenv work is the location of its binary, and its hacked `site.py`.

Corollary: if you aren't using the virtualenv's binary, you aren't really using the virtualenv.

What about `bin/activate`?

A shell convenience only.
Adds the virtualenv's bin/ to the front of your shell PATH.
Bad mental model.

Problems

Sometimes you can't use the virtualenv's binary.

Faking it:

import sys
sys.path.insert(0,
    "/path/to/venv/lib/python2.6/site-packages")

import site
site.addsitedir(
    "/path/to/venv/lib/python2.6/site-packages")

execfile("/path/to/venv/bin/activate_this.py")

Forked `site.py`.

Distributors modify site.py (e.g. Debian dist-packages).
New Python versions modify site.py.
Virtualenv has to play catchup.

Other approaches

virtualenv-small-sitepy

`https://bitbucket.org/brandon/virtualenv-small-sitepy`

Brandon Craig Rhodes' experiment.
Make virtualenv's site.py a light wrapper around the system site.py.
Extend, rather than copy-paste-modify.
Very close, but not 100%; currently dormant.

rvirtualenv

`https://github.com/kvbik/rvirtualenv`

Jakub Vysoky.
Completely different approach.
Uses PYTHONPATH.
No copied Python binary.
Relocatable, inheritable.
Doesn't support --no-site-packages.

Just use `PYTHONHOME`!

Sets sys.prefix with no copied binary.
Still requires copied stdlib or site.py hacks.
Harder to pin a "virtualenv" to a Python version.
Ian: "I don't like environment variables."

pythonv

Larry Hastings.
C wrapper that calls system Python binary.
Most promising direction for integration into Python.
Needs a PEP and a patch?

Pip!

Pip installs (Python) Packages

How?

Pip's dirty little secret

Pip doesn't install packages at all...
Pip <3 setuptools!

`pip/req.py`

def install(self, install_options, global_options=()):
    # ...
    install_args = [
        sys.executable, '-c',
        "import setuptools;__file__=%r;"\
        "execfile(__file__)" % self.setup_py] +\
        list(global_options) + [
        'install',
        '--single-version-externally-managed',
        '--record', record_filename]

    # ...

    call_subprocess(install_args + install_options,
        cwd=self.source_dir,
        filter_stdout=self._filter_install,
        show_stdout=False)

Translation:

Ugliest.
Hack.
Evar.
(Sorry, Ian.)
(Like a disturbing number of Ian's ugly hacks, it works quite well.)

Install like pip:

python -c
  "import setuptools;__file__=/p/to/setup.py;execfile(__file__)"
  install
  --single-version-externally-managed
  --record /tmp/pip-ZgGVWG-record/install-record.txt

`"import setuptools;file=/p/to/setup.py;execfile(file)"`

Setuptools works by monkeypatching the builtin distutils.
Thus, importing setuptools is enough to activate its features.
This hack allows pip to do setuptools-only things (generate .egg-info metadata, develop/editable installs) on all projects, even those that stick to pure distutils in their setup.py.

`install`

Just like a normal python setup.py install.

`--single-version-externally-managed`

Tells setuptools to do a "flat" (distutils-style) install and put the metadata next to it in a .egg-info directory, rather than a zipped egg or egg-directory.
A setuptools feature.

Setuptools normally installs like this:

$ easy_install yolk

# cd to site-packages
$ ls | egrep "(yolk|easy)"
easy-install.pth
yolk-0.4.1-py2.6.egg

$ cat easy-install.pth
./setuptools-0.6c11-py2.6.egg
./pip-0.8.1-py2.6.egg
./yolk-0.4.1-py2.6.egg
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)

With `--single-version-externally-managed`:

$ pip install yolk

$ ls | grep yolk
yolk/
yolk-0.4.1-py2.6.egg-info/

`--record /tmp/pip-ZgGVWG-record/install-record.txt`

Generates a record of all files installed.
distutils and setuptools: Y U NO DO THIS BY DEFAULT?!?!??!
(Fortunately PEP 376 mandates it, and distutils2 does it.)
Pip generates it in a temporary location, then moves it to installed-files.txt in the .egg-info metadata directory.

Using `pip` as an API

`pip/commands/install.py`

class InstallCommand(Command):
    name = 'install'
    usage = '%prog [OPTIONS] PACKAGE_NAMES...'
    summary = 'Install packages'
    bundle = False

    def __init__(self):
        super(InstallCommand, self).__init__()
        self.parser.add_option(
            '-e', '--editable',
            dest='editables',
            action='append',
            default=[],
            metavar='VCS+REPOS_URL[@REV]#egg=PACKAGE',
    # ...

    def run(self, options, args):
        # ...

from pip.req import InstallRequirement, RequirementSet
from pip.index import PackageFinder
from pip.locations import build_prefix, src_prefix
reqset = RequirementSet(build_dir=build_prefix,
                        src_dir=src_prefix,
                        download_dir=None)
req = InstallRequirement.from_line("yolk==0.4.1",
                                   comes_from=None)
reqset.add_requirement(req)
finder = PackageFinder(
    find_links=[],
    index_urls=["http://pypi.python.org/simple"])
reqset.prepare_files(finder)
reqset.install(install_options=[], global_options=[])
reqset.cleanup_files()

Lots more to pip

PackageFinder: easy_install-compatible HTML link scraping to find source distributions/versions.
VCS backends.
Requirements files.
Bundles.
Freeze, search, uninstall.

So.

What have we learned about Ian Bicking's brain?

...
Left as an exercise!
Thanks Ian!

Future

virtualenv

Integrate it into Python 3?
Can be done more cleanly if integrated.

pip

Convert to wrapper around distutils2.
Provide end-user conveniences (e.g. requirements files).
Can iterate this more rapidly outside stdlib.

Questions?

Resources:

#pip on Freenode
http://groups.google.com/group/python-virtualenv
https://github.com/pypa/pip
https://github.com/pypa/virtualenv
@carljm
carl@oddbird.net
slides: https://github.com/carljm/pipvirtualenv-preso

Reverse-engineering

Ian Bicking's brain

inside pip and virtualenv

Carl J Meyer

@carljm

carl@oddbird.net

virtualenv

What should it do?

site-packages

site-packages

site.py

Where does sys.prefix come from?

Modules/getpath.c

The hunt for sys.prefix

Yes, os.py is the hardcoded landmark.

Modules/getpath.c

Let's do this.

Copy the Python binary.

sys.prefix is still /usr.

Create the landmark.

Fixing sys.exec_prefix

We have an isolated virtualenv!

How could we fix it?

Bootstrap like a virtualenv.

From virtualenv's modified site.py

But site.py depends on the standard library!

From virtualenv.py

We can also add the global site-packages, if we want it.

site.py

What makes a virtualenv work is the location of its binary, and its hacked site.py.

Corollary: if you aren't using the virtualenv's binary, you aren't really using the virtualenv.

What about bin/activate?

Problems

Sometimes you can't use the virtualenv's binary.

Faking it:

Forked site.py.

Other approaches

virtualenv-small-sitepy

https://bitbucket.org/brandon/virtualenv-small-sitepy

rvirtualenv

https://github.com/kvbik/rvirtualenv

Just use PYTHONHOME!

pythonv

Pip!

How?

Pip's dirty little secret

pip/req.py

Translation:

Install like pip:

"import setuptools;__file__=/p/to/setup.py;execfile(__file__)"

install

--single-version-externally-managed

Setuptools normally installs like this:

With --single-version-externally-managed:

--record /tmp/pip-ZgGVWG-record/install-record.txt

Using pip as an API

pip/commands/install.py

Lots more to pip

So.

What have we learned about Ian Bicking's brain?

Future

virtualenv

pip

Questions?

Resources:

`site-packages`

`site-packages`

`site.py`

Where does `sys.prefix` come from?

`Modules/getpath.c`

The hunt for `sys.prefix`

Yes, `os.py` is the hardcoded landmark.

`Modules/getpath.c`

`sys.prefix` is still `/usr`.

Fixing `sys.exec_prefix`

Bootstrap like a `virtualenv`.

From virtualenv's modified `site.py`

But `site.py` depends on the standard library!

From `virtualenv.py`

We can also add the global `site-packages`, if we want it.

`site.py`

What makes a virtualenv work is the location of its binary, and its hacked `site.py`.

What about `bin/activate`?

Forked `site.py`.

`https://bitbucket.org/brandon/virtualenv-small-sitepy`

`https://github.com/kvbik/rvirtualenv`

Just use `PYTHONHOME`!

`pip/req.py`

`"import setuptools;file=/p/to/setup.py;execfile(file)"`

`install`

`--single-version-externally-managed`

With `--single-version-externally-managed`:

`--record /tmp/pip-ZgGVWG-record/install-record.txt`

Using `pip` as an API

`pip/commands/install.py`