What do you need to consider when setting up your Python development environment? What are the best tools for the job? What are the tradeoffs you need to be aware of?

GitHub - tomoum/python at pipenv

Python templates for publishing to pypi or binary distribution - GitHub - tomoum/python at pipenv

GitHubtomoum

What is a Development Environment?

In a broad sense, your project's development environment is everything you need to develop, test, and debug your application. This should not be confused with your IDE (Integrated Development Environment) such as VScode, PyCharm, or Vim. These are just software tools that streamline and simplify your development environment. Some companies and teams will actually allow you to use the IDE of your choice as the development environment is independent.

The development environment includes but is not limited to:

• source code (the files that are supposed to run🤞)
• source control software (e.g git )
• software installations (e.g the Python interpreter)
• dependency installations (e.g python packages like numpy )
• environment variables (e.g PYTHONPATH)
• shell scripts (e.g bash or PowerShell)
• documentation (e.g markdown docs)

What do you need from your Development Environment?

Your development environment can have a major influence on the project's success, speed, and your team's sanity😄. If it's not set up correctly you will spend countless hours trying to just get things to run. If you've ever seen the online meme "My code doesn't work I don't know why? my code works now, I don't know why?". Nine times out of ten when you have seemingly made no changes to the code but things are mysteriously not running anymore the problem is most likely in your development environment.

On the other hand, if you set up your environment correctly you can enjoy fast development times and optimize your developer's workflows. For instance, one thing I love to add to all my projects is the very strict auto-formatting tool black. This increases productivity by eliminating a source of endless debates during pull requests about trivial, subjective formatting choices.  

Repeatability & Ease-of-use

If your computer dies you need to know you can be up and running on a new machine easily and quickly. Your setup should be fast and easy to deploy. This also ensures that onboarding a new team member is a breeze. If you need to write a long document for onboarding a new member to run the code on their machine. Something has gone horribly wrong. I find that the best way to document any one-time procedures for your setup is by putting it in a bash or PowerShell script. This way at least you have a chance of it just running as is instead of additional manual steps. Watch out for that script exploding in size though it should not be more than 100 lines.

Control All Package Dependencies

You need to be able to specify the exact Python interpreter version and every non-standard Python package your project depends on. Preferably the exact version required for each package. You typically install these packages using the Python package manager pip which ships with all Python 3 interpreters. This installs the package to your global Python interpreter. This means that all projects on your machine using this python version now have access to this package. This is bad because you could potentially break one of your projects if that new package you installed is a newer version than what is expected by that project's code.  

Furthermore, this makes it hard to track down package installations that might be interfering with one other project dependencies. You will also lose track of which package belongs to which project. You do not want that to happen because you will be on a wild goose chase for hours maybe days.💩

Doing this will go a long way to curing what I call the developer flu "It works on my machine". You do not want a project behaving differently on two machines for the exact same code commit and the same inputs😤🤦‍♂️.

Resembles Deployment Environment Closely

One thing you need to watch out for is making sure your

Python Virtual Environments

Virtual environments are Python packages/tools that will help you manage your development tools. There are a variety of tools that can do this job that works in slightly different ways. However, the basic idea across all these tools is to essentially create a separate Python interpreter and its set of packages for each project folder. This means you can clone the same repository multiple times on your machine and create an isolated, separate environment for each project folder. This is great if you are experimenting or reviewing a different branch in your repository without affecting your current work.      

Pipenv

This tool not only controls the environment but also (as the name suggests)
replaces pip and adopts the pipfile project. Pipfiles are the new way of specifying the recipe for your project's environment. Replacing the antiquated requirements.txt file. Think of it as pip, pipfile and virtualenv together in one tool...but the sum is more than the parts. One of the great features of pipenv is that it works with pyenv-win or pyenv to automatically install the required Python version for the project. Furthermore, if you have multiple versions of Python installed it will load the correct version to create the virtual environment with. This is the only way I install and manage Python on my machines.

Top Pipenv Features:

• Load the correct Python interpreter version or install it if it doesn't exist using pyenv-win or pyenv.
• Automatically load a .env file with your project's environment variables.
• Specify some packages to only be installed for the development virtual environment which is not required for deployment. (e.g code linter)  
• Create a Pipfile.lock file to hash all the project dependencies for added security and repeatability.

Top Pipfile features:

• Uses TOML syntax to allow for easier advanced configurations.
• Specifies multiple package sources inside or outside your organization.
• Specify package versions per host OS or many other conditionals. e.g if the host os is Windows install version a if it's a Linux install version b  
• Specify the exact version of Python you want. e.g python 3.7.4  
• Define two dependency groups packages and dev-packages which makes it easier to track packages you need in development but not when the code is deployed. e.g. code liters.

and so much more...

Alternative Virtual Environment Tools

I do not recommend you use any of these for new projects as they are quite outdated by now unless you have a very good reason.

virutalenv

Virtualenv is the next best tool you could use. It does the job and does it well. Except the CLI commands are cumbersome and not user-friendly. So if you choose to use virtualenvwrapper instead. As the name suggests it's a thin wrapper that makes your day-to-day with this tool smoother.

venv

The only advantage I see for this tool is that it comes with Python. So if you work in a very security stringent organization where installing 3rd party packages is a pain. Use venv if you have no other option. In other words, the only reason for you to use this tool should be "someone else has already set it up and it's too much work to switch now"😁. Then this might be the tool for you. Otherwise don't do that to yourself 😄.

Reference Material

1. Pipenv Docs.
2. The corey shafer tutorial for virtualenv.
3. Installing pipenv, virutalenv, and venv.