Taming your dependencies with pip
Carl J Meyer
@carljm
carl@oddbird.net
even Django hates them
from django.utils import unittestbut you don't have to!
scope
- a web project
- not a library
- not an end-user program
deployment desiderata
- repeatable
- reliable
- fast
virtualenv
- use it
- use
--no-site-packages - recreate freely and often
pip!
by Ian Bicking
"Pip Installs (Python) Packages"
the pip requirements file
Django==1.3
South==0.7.3
django-compressor==0.9.2
BeautifulSoup==3.2.0
slimmer==0.1.30
django-floppyforms==0.4.2
django-classy-tags==0.3.0
...habits for highly effective requirements files
- pin everything
- (even deps of deps)
- pin exactly
Django==1.2.5, notDjango>=1.2,<1.3- test regularly in a clean venv
how it works
Django==1.3- default "index url":
http://pypi.python.org/simple/ http://pypi.python.org/simple/Django/1.3http://pypi.python.org/simple/Django/- link scraping, wheee!
rel=[homepage,download]or links that look like PyPI home/download links- archive files (zip/tar) named like
Django-1.3.tar.gz- "sdists" - picks highest found version that fits criterion
- use
pip install -vv ...to debug
re-downloading is too slow
in ~/.pip/pip.conf:
[global]
download-cache=/path/to/some/dirPIP_DOWNLOAD_CACHE=/path/to/some/dir
oh noes! PyPI is down!
in ~/.pip/pip.conf:
[global]
use-mirrors=truePIP_USE_MIRRORS=true
sure, PyPI mirrors are nice
- but
Some Projecthost their own downloads- and now their site's down and I can't deploy
if you want it done right, do it yourself
--index-url--find-linksvendor/
option 1
host your own index
- pip lets us replace the default PyPI index URL (or add more index URLs)
- directory per project, with links to sdists
- automate with
chishop(orEggBasket,ClueReleaseManager,basketweaver...) - can also accept
setup.py uploadlike PyPI - just use
--index-urloption with base URL - or
--extra-index-urlif you want PyPI too - can go in reqs file (applies to whole file)
--index-url http://myindex.example.com/
$ tree
.
|-- Django
| `-- Django-1.3.tar.gz
`-- django-floppyforms
`-- django-floppyforms-0.4.2.tar.gz
your own index
- easy to upload internal packages with
setup.py upload - higher maintenance
- separate from project but needs to be kept in sync
option 2
dump some tarballs in a directory
--find-linkscan inject arbitrary URLs into the link-scraping- point it at a web-served directory of sdists with indexes on
- this option can also go directly in a requirements file
- file:// URLs work too, pip will get the directory listing
- if you don't want PyPI used at all, add
--no-index - easily populate with
pip install -d my-sdists-dir/ SomePackage
--find-links http://internal.example.com/sdists/
or
--find-links file:///path/to/my/sdists/
$ tree
.
|-- Django-1.3.tar.gz
`-- django-floppyforms-0.4.2.tar.gz
find-links
- trivial to maintain manually
- very fast with
file://URL and--no-index - can be self-contained with project
option 3
commit deps (unpacked) in a vendor/ directory
pip install --install-option="--home=`pwd`/vendor/" ...export PYTHONPATH=`pwd`/vendor/lib/python- commit
vendor/with the rest of your code
$ tree -L 4
.
`-- vendor
|-- bin
| `-- django-admin.py
`-- lib
`-- python
|-- django
|-- Django-1.3-py2.6.egg-info
|-- django_floppyforms-0.4.7-py2.6.egg-info
`-- floppyforms
vendor/
- fastest, simplest deployment story (no moving parts)
- platform/path issues with compiled extensions
- requires path hackery (PYTHONPATH or runtime sys.path mods)
- temptation to patch stuff directly in
vendor/
but what about VCS support?!?
-e git://github.com/carljm/django-model-utils@8c6c1cf8#egg=django-model-utils
it's excellent
use it in development
not for production deploys
- more moving parts in every deploy
- have to install VCSes on every server
- VCS clones / checkouts are slow
- every VCS host is another point of failure
- internal releases are easy to make (
setup.py sdist)
So.
pipis flexible- Installing your dependencies can be fast and reliable.
- Go make it so.
Questions?
Resources:
- pip-installer.org
- virtualenv.org
#pipon Freenode- groups.google.com/group/python-virtualenv
- @carljm
- carl@oddbird.net
- slides: github.com/carljm/tamingdeps