The case of the missing development environment

Today is C01t’s first day on project JAM. JAM is developed in python. Since this is his first python project, C01t does not have a python development environment configured. So C01t sets out to setup a python environment on his Microsoft Surface Book.

building ‘twisted.test.raiser’ extension
error: Microsoft Visual C++ 14.0 is required. Get it with “Microsoft Visual C++ Build Tools”: http://landinghub.visualstudio.com/visual-cpp-build-tools

Nick runs Linux. So he is not going to be of any help. The rest of the team run on OS X. So they too have no helpful suggestions. Google it is then. A quick search reveals that there are official python installers for Windows. So C01t downloads and installs the latest version, python 3.6.2. Eager to test his setup he runs make inside the root folder of project JAM. Seems like things are working. pip is installing packages. And then progress comes to a crashing halt. The twisted package failed to install because of missing Microsoft Visual C++.

raiser.c
c:\users\xxx\appdata\local\programs\python\python36\include\pyconfig.h(222): fatal error C1083: Cannot open include file: ‘basetsd.h’: No such file or directory

Maybe, installing the suggested Microsoft Visual C++ Build Tools will help. C01t downloads the install package from >Microsoft’s website and installs the tools. A few minutes later C01t runs the make command again. This time the error is about a missing header file.

ModuleNotFoundError: No module named ‘win32api’

Google to the rescue! http://www.lfd.uci.edu/~gohlke/pythonlibs/ has a twisted 17.5.0 wheel package for python 3.6 built for windows. After downloading and manually installing the twisted package, pip install completes successfully. However, running the scraper fails with a long call stack. The error is a missing win32api module.

overlapped.c
c:\users\xxx\appdata\local\programs\python\python36\include\pyconfig.h(222): fatal error C1083: Cannot open include file: ‘basetsd.h’: No such file or directory

The internet once again has the solution! After running the command pip install pypiwin32, C01t can successfully run make. As a result having JAM running on his machine, the very next day C01t is ready starting to work on his first item. He pulls the latest code from the repository and runs make again. The command fails. JAM now has a dependency on the trollius package. And attempting to install the package fails with a compilation error due to a missing header file error.

Exasperated C10t wonders: “Will I ever get to write any code!”.

Building a development environment

All of C01t’s troubles could have been avoided if project JAM had a standardized development environment. We use Docker and Docker Compose to create multi-platform containerized development environment. First of all, it is critical to create a “devbox” environment for compiling/running, unit-testing and debugging the code.

We use the following folder structure to store the configuration files for the development environment in the projects repository:

dev
  docker
     docker-compose.yml
     devbox
       Dockerfile
       ...

Building the container

We are building a “devbox” environment for project JAM. To build the “devbox” container, we create a Dockerfile in the dev/docker/devbox folder. The starting point for the new containers is one of the official python containers.

FROM python:3.6

Then we customize this container further by installing some additional useful tools such as sudo, vim, wget, iputils-ping etc. We can also update some package already installed, such as pip for python.

RUN apt-get update && apt-get -y install curl wget sudo vim iputils-ping && apt-get clean && rm -rf /var/lib/apt/lists/*
RUN pip install --upgrade pip

Finally we make the startup command ping. This allows us to start the container with compose and attach an interactive shell to it later.

CMD ["/bin/ping", "-i 360", "localhost"]

The complete Dockerfile looks like:

FROM python:3.6

RUN apt-get update && apt-get -y install curl wget sudo vim iputils-ping && apt-get clean && rm -rf /var/lib/apt/lists/*
RUN pip install --upgrade pip

CMD ["/bin/ping", "-i 360", "localhost"]

Running the container

Now we are ready to add a devbox service to the docker-compose.yml file in the dev/docker folder. The new service in the docker-compose.yml file is our “devbox” environment.

We want to be able to run “devbox” environments from multiple projects on the same machine. Therefore, we assign unique names to all artifacts associated with the service. We pick a unique service name, image name and container name. The easiest way to do this is to pre-pend the project name or project acronym. (“cc_wp” stands for Cookiecutter WordPress)

  cc_wp_devbox:
    image: ccwp:devbox
    container_name: cc_wp_devbox

Next we specify the source for the service. The source is the container built using the Dockerfile authored before.

    build: ./devbox

Additionally, we load the root folder of the project as a volume mounted under /src inside the container. Consequently, we are able to edit the source code on the host using our code editor of choice. All changes made to source files on the host are instantaneously available inside the container.

    volumes: 
      - ./../../:/src

Finally, we set an environment variable to mark container as a “devbox” environment. We’ll use the environment variable when we author the Makefile to drive the environment.

    environment:
      DEVBOX: 1

The complete docker-compose.yml file looks like:

version: '2'
services:
  cc_wp_devbox:
    image: ccwp:devbox
    container_name: cc_wp_devbox
    build: ./devbox
    volumes: 
      - ./../../:/src
    environment:
      DEVBOX: 1

Creating the Makefile

The last piece of the puzzle is a Makefile. Above all we want to make the Makefile runable from outside and inside the container. Therefore, we generate a wrapper for all commands based on the DEVBOX environment variable. Remember? We set that when we started the container.

ifdef DEVBOX
	CMD = $(1)
else
	CMD = docker exec -t cc_wp_devbox bash -c "cd /src && $(1)"
endif

Then we author commands for targets and rules to be executed within this wrapper. For example, a target to run pip against a requirements.txt looks like:

.PHONY: install
install:
	$(call CMD, pip install -r requirements_dev.txt)

Finally, a sample Makefile for project JAM looks like:

.PHONY: dev
dev: test

.PHONY: install_dev
install_dev:
	$(call CMD, pip install -r requirements_dev.txt)

# test targets
test: unittest

.PHONY: unittest
unittest: install_dev
	$(call CMD, pytest --verbose tests)

# variables used to determine the command to run
ifdef DEVBOX
	CMD = $(1)
else
	CMD = docker exec -t cc_wp_devbox bash -c "cd /src && $(1)"
endif

One Reply to “The case of the missing development environment”

Leave a Reply

Your email address will not be published. Required fields are marked *