Setting up Python Projects with Virtual Environments

Setting Up Python Projects in Virtual Environments

Table of Contents

Concept

The idea is to set up everything in a sandboxed Virtual Environment — pip, Python and any Python packages you need to install in a project. This can be achieved by using the cross-platform packages virtualenv with virtualenvwrapper, or the more advancedpyenv and its plugins.

Check Python location and version

bash$ which python
bash$ python --version
(Same for python3)

Check pip location and version

bash$ which pip
bash$ pip --version

Install pip if not found

bash$ sudo easy_install pip
(**Note: pip, setuptools, and pyenv are installed with Python >= 3.5)

Show installed Python packages

bash$ pip list

You can also use bash$ pip freeze, the difference is that pip list shows all installed packages, including editables, while pip freeze outputs installed packages in requirements format, which is used in the requirements.txt file generated for a project with:

bash$ pip install -r requirements.txt

Create Working and Project Directories

Do this if you want different Virtual Environments and Projects directories.

bash$ mkdir ~/.virtualenvs
bash$ mkdir ~/.Projects

Set up your directory environment variables

Add these lines to ~/.bashrc:

export WORKON_HOME=~/.virtualenvs (or whatever)
export PROJECT_HOME=$HOME/Projects (or whatever)

Register the changes to ~/.bashrc

bash$ source .bashrc

Install and Use virtualenv

Check that virtualenv is installed

bash$ virtualenv

Install virtualenv if necessary

bash$ pip install virtualenv

Test virtualenv

bash$ virtualenv
You must provide a DEST_DIR
Usage: virtualenv [OPTIONS] DEST_DIR

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -v, --verbose         Increase verbosity.
  -q, --quiet           Decrease verbosity.
  -p PYTHON_EXE, --python=PYTHON_EXE
                        The Python interpreter to use, e.g.,
                        --python=python2.5 will use the python2.5 interpreter
                        to create the new environment.  The default is the
                        interpreter that virtualenv was installed with
                        (/Users/Stuart/.virtualenvs/django-cms/bin/python3.6)
  --clear               Clear out the non-root install and start from scratch.
  --no-site-packages    DEPRECATED. Retained only for backward compatibility.
                        Not having access to global site-packages is now the
                        default behaviour.
  --system-site-packages
                        Give the virtual environment access to the global
                        site-packages.
  --always-copy         Always copy files rather than symlinking.
  --unzip-setuptools    Unzip Setuptools when installing it.
  --relocatable         Make an EXISTING virtualenv environment relocatable.
                        This fixes up scripts and makes all .pth files
                        relative.
  --no-setuptools       Do not install setuptools in the new virtualenv.
  --no-pip              Do not install pip in the new virtualenv.
  --no-wheel            Do not install wheel in the new virtualenv.
  --extra-search-dir=DIR
                        Directory to look for setuptools/pip distributions in.
                        This option can be used multiple times.
  --download            Download preinstalled packages from PyPI.
  --no-download, --never-download
                        Do not download preinstalled packages from PyPI.
  --prompt=PROMPT       Provides an alternative prompt prefix for this
                        environment.
  --setuptools          DEPRECATED. Retained only for backward compatibility.
                        This option has no effect.
  --distribute          DEPRECATED. Retained only for backward compatibility.
                        This option has no effect.

Create a Virtual Environment in $WORKON_HOME

virtualenv newenv -p python3

forces the installation of the latest Python 3. You can specify any version of Python currently installed on your computer with:

virtualenv -p python3.2 newenv

You must first activate the Virtual Environment before you start installing software packages into it. This keeps your system Python entirely separate from your project. It makes your project self-contained within its directory and thus transportable.

Activate a Virtual Environment

You must do this from within the Virtual Environment directory:

bash$ cd $WORKON_HOME/newenv
bash$ source bin/activate

Deactivate an active Virtual Environment

From within the Virtual Environment directory:

source bin/deactivate

Install and use virtualenvwrapper

bash$ pip install virtualenvwrapper (sudo if necessary)

Add this line to ~/.bashrc:

source /usr/local/bin/virtualenvwrapper.sh

Test virtualenvwrapper

bash$ virtualenvwrapper
  
virtualenvwrapper is a set of extensions to Ian Bicking's virtualenv
tool.  The extensions include wrappers for creating and deleting
virtual environments and otherwise managing your development workflow,
making it easier to work on more than one project at a time without
introducing conflicts in their dependencies.

For more information please refer to the documentation:

    http://virtualenvwrapper.readthedocs.org/en/latest/command_ref.html

Commands available:

  add2virtualenv: add directory to the import path

  allvirtualenv: run a command in all virtualenvs

  cdproject: change directory to the active project

  cdsitepackages: change to the site-packages directory

  cdvirtualenv: change to the $VIRTUAL_ENV directory

  cpvirtualenv: duplicate the named virtualenv to make a new one

  lssitepackages: list contents of the site-packages directory

  lsvirtualenv: list virtualenvs

  mkproject: create a new project directory and its associated virtualenv

  mktmpenv: create a temporary virtualenv

  mkvirtualenv: Create a new virtualenv in $WORKON_HOME

  rmvirtualenv: Remove a virtualenv

  setvirtualenvproject: associate a project directory with a virtualenv

  showvirtualenv: show details of a single virtualenv

  toggleglobalsitepackages: turn access to global site-packages on/off

  virtualenvwrapper: show this help message

  wipeenv: remove all packages installed in the current virtualenv

  workon: list or change working virtualenvs

What does virtualenvwrapper do?

  1. Organizes all of your virtual environments in one place.
  2. Wrappers for managing your virtual environments (create, delete, copy).
  3. Use a single command to switch between environments.
  4. Tab completion for commands that take a virtual environment as argument.
  5. User-configurable hooks for all operations (see Per-User Customization).
  6. Plugin system for more creating sharable extensions (see Extending Virtualenvwrapper).

List installed Virtual Environments

bash$ lsvirtualenv

Create a new Virtual Environment

mkvirtualenv newenv (or whatever you want to call it)

The new environment will be created in the current directory. The full command options are:

mkvirtualenv [-a project_path] [-i package] [-r requirements_file] [virtualenv options] ENVNAME

See http://virtualenvwrapper.readthedocs.org/en/latest/command_ref.html

bash$ mkvirtualenv newenv

Using base prefix '/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6'
New python executable in /Users/Stuart/.virtualenvs/newenv/bin/python3.6
Also creating executable in /Users/Stuart/.virtualenvs/newenv/bin/python
Installing setuptools, pip, wheel...done.
virtualenvwrapper.user_scripts creating /Users/Stuart/.virtualenvs/newenv/bin/predeactivate
virtualenvwrapper.user_scripts creating /Users/Stuart/.virtualenvs/newenv/bin/postdeactivate
virtualenvwrapper.user_scripts creating /Users/Stuart/.virtualenvs/newenv/bin/preactivate
virtualenvwrapper.user_scripts creating /Users/Stuart/.virtualenvs/newenv/bin/postactivate
virtualenvwrapper.user_scripts creating /Users/Stuart/.virtualenvs/newenv/bin/get_env_details
pyenv-virtualenv: deactivate must be sourced. Run 'source deactivate' instead of 'deactivate'
(newenv) bash$

Activate a Virtual Environment

You can switch to any Virtual Environment at any time, even from another VE.

(newenv) bash$ workon otherenv
(otherenv) bash$

Note that the new Virtual Environment is now shown in parentheses before the bash prompt.

Associate a Virtual Environment with a Project directory

This allows you to arrange your directory structure however you please.

bash$ setvirtualenvproject [virtualenv_path project_path]

These must be absolute paths. If no arguments are given, the current Virtual Environment and current directory are assumed.

Your system is now set up to create new Python projects inside a Virtual Environment.

Install Python, pip and virtualenv on Windows

Installing a Python development environment on Windows is slightly more complicated, although the principle software is the same. Here are some guides:

https://zignar.net/2012/06/17/install-python-on-windows/
https://www.davidbaumgold.com/tutorials/set-up-python-windows/

Install and use pyenv

As described in its Wiki on Github,:

“pyenv is a tool for simple Python version management that offers many advantages.”

pyenv:

  • Lets you change the global Python version on a per-user basis;
  • Provides support for per-project Python versions;
  • Allows you to override the Python version with an environment variable;
  • Searches commands from multiple versions of Python at a time; and
  • Consisting only of shell scripts, there is no Python bootstrap problem; and
  • Its shim approach works by adding a directory to your $PATH so it does not need to be loaded into your shell.

pyenv does not manage virtualenv, so you can still use virtualenv to create virtual environments, perhaps with pyenv-virtualenv to automate the process.

Python3.4+ come with pyenv already installed, so as development moves forward pyenv will become the de facto way of installing and managing Virtual Environments.

Install on Linux

Linux users can checkout pyenv from Github

$ git clone https://github.com/pyenv/pyenv.git ~/.pyenv

**You must then define the environment variable PYENV_ROOT to point to the path where the pyenv Git repository is cloned, and add $PYENV_ROOT/bin to your $PATH for access to the pyenv command-line utility:

$ echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bash_profile
$ echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bash_profile

pyenv init must also be written to your .bash_profile_ to enable shims and autocompletion. eval "$(pyenv init -)"must be placed toward the end of the shell configuration file since it manipulates PATH during the initialization.

$ echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n  eval "$(pyenv init -)"\nfi' >> ~/.bash_profile`

Restart your shell so the path changes take effect.

$ exec "$SHELL"

You can now begin using pyenv.

Install on a Mac

Mac users can install with Homebrew:

$ brew update
$ brew install pyenv

Add PYENV_ROOT, $PYENV_ROOT/bin, and pyenv init to your PATH as above, then restart your shell.

Install a different Python version

To install different Python versions into $(pyenv root)/versions file, for example to download and install Python 2.7.8, run:

$ pyenv install 2.7.8

Install Several Python Versions locally

pyenv allows you to access several Python version libraries should your project require this. For example:

$ pyenv local 2.7.6 3.3.3
$ pyenv versions
  system
*2.7.6 (set by /Users/yyuu/path/to/project/.python-version)
*3.3.3 (set by /Users/yyuu/path/to/project/.python-version)
$ python --version
Python 2.7.6
$ python2.7 --version
Python 2.7.6
$ python3.3 --version
Python 3.3.3

A full description of pyenv commands can be found in its COMMAND.md file.

Install pyenv plugins

pyenv-virtualenv

A pyenv plugin to manage virtualenv, pyenv-virtualenv provides features to manage virtualenvs and conda environments for Python on UNIX-like systems. It installs into the (pyenv root)/plugins/pyenv-virtualenv directory. From inside that directory you can:

  • Check out a specific release tag.
  • Get the latest dev release by running git pull to download the latest changes.

Check out directly into the (pyenv root)/plugins directory:

$ git clone https://github.com/pyenv/pyenv-virtualenv.git $(pyenv root)/plugins/pyenv-virtualenv

You can also add pyenv virtualenv-init to your shell to enable auto-activation of virtualenvs. This is optional but very useful.

$ echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.bash_profile`
$ exec "$SHELL"

Mac users can install pyenv-virtualenv with Homebrew:

$ brew install pyenv-virtualenv

Using pyenv-virtualenv

For example:

$ pyenv virtualenv 2.7.10 my-virtual-env-2.7.10

….will create a Virtual Environment based on Python 2.7.10 under $(pyenv root)/versions in a folder called my-virtual-env-2.7.10

pyenv-virtualenvwrapper

An alternative approach to manage Virtual Environments , pyenv-virtualenvwrapper is a pyenv plugin that provides a pyenv virtualenvwrapper command that lets you manage your environments with virtualenvwrapper.

pyenv-virtualenvwrapper helps in interacting with virtualenvwrapper, but pyenv-virtualenv (above) provides more convenient commands where virtualenvs are first-class pyenv versions that can be activated and deactivated.

Many other pyenv plugins are available on the pyenv Github repository page.

 

ENDS