The case of the missing development environment

Today is C01t’s first day on project JAM. JAM is developed in python. Since this is his first python project, C01t does not have a python development environment configured. So C01t sets out to setup a python environment on his Microsoft Surface Book.

building ‘twisted.test.raiser’ extension
error: Microsoft Visual C++ 14.0 is required. Get it with “Microsoft Visual C++ Build Tools”: http://landinghub.visualstudio.com/visual-cpp-build-tools

Nick runs Linux. So he is not going to be of any help. The rest of the team run on OS X. So they too have no helpful suggestions. Google it is then. A quick search reveals that there are official python installers for Windows. So C01t downloads and installs the latest version, python 3.6.2. Eager to test his setup he runs make inside the root folder of project JAM. Seems like things are working. pip is installing packages. And then progress comes to a crashing halt. The twisted package failed to install because of missing Microsoft Visual C++.

raiser.c
c:\users\xxx\appdata\local\programs\python\python36\include\pyconfig.h(222): fatal error C1083: Cannot open include file: ‘basetsd.h’: No such file or directory

Maybe, installing the suggested Microsoft Visual C++ Build Tools will help. C01t downloads the install package from >Microsoft’s website and installs the tools. A few minutes later C01t runs the make command again. This time the error is about a missing header file.

ModuleNotFoundError: No module named ‘win32api’

Google to the rescue! http://www.lfd.uci.edu/~gohlke/pythonlibs/ has a twisted 17.5.0 wheel package for python 3.6 built for windows. After downloading and manually installing the twisted package, pip install completes successfully. However, running the scraper fails with a long call stack. The error is a missing win32api module.

overlapped.c
c:\users\xxx\appdata\local\programs\python\python36\include\pyconfig.h(222): fatal error C1083: Cannot open include file: ‘basetsd.h’: No such file or directory

The internet once again has the solution! After running the command pip install pypiwin32, C01t can successfully run make. As a result having JAM running on his machine, the very next day C01t is ready starting to work on his first item. He pulls the latest code from the repository and runs make again. The command fails. JAM now has a dependency on the trollius package. And attempting to install the package fails with a compilation error due to a missing header file error.

Exasperated C10t wonders: “Will I ever get to write any code!”.

Building a development environment

All of C01t’s troubles could have been avoided if project JAM had a standardized development environment. We use Docker and Docker Compose to create multi-platform containerized development environment. First of all, it is critical to create a “devbox” environment for compiling/running, unit-testing and debugging the code.

We use the following folder structure to store the configuration files for the development environment in the projects repository:

dev
  docker
     docker-compose.yml
     devbox
       Dockerfile
       ...

Building the container

We are building a “devbox” environment for project JAM. To build the “devbox” container, we create a Dockerfile in the dev/docker/devbox folder. The starting point for the new containers is one of the official python containers.

FROM python:3.6

Then we customize this container further by installing some additional useful tools such as sudo, vim, wget, iputils-ping etc. We can also update some package already installed, such as pip for python.

RUN apt-get update && apt-get -y install curl wget sudo vim iputils-ping && apt-get clean && rm -rf /var/lib/apt/lists/*
RUN pip install --upgrade pip

Finally we make the startup command ping. This allows us to start the container with compose and attach an interactive shell to it later.

CMD ["/bin/ping", "-i 360", "localhost"]

The complete Dockerfile looks like:

FROM python:3.6

RUN apt-get update && apt-get -y install curl wget sudo vim iputils-ping && apt-get clean && rm -rf /var/lib/apt/lists/*
RUN pip install --upgrade pip

CMD ["/bin/ping", "-i 360", "localhost"]

Running the container

Now we are ready to add a devbox service to the docker-compose.yml file in the dev/docker folder. The new service in the docker-compose.yml file is our “devbox” environment.

We want to be able to run “devbox” environments from multiple projects on the same machine. Therefore, we assign unique names to all artifacts associated with the service. We pick a unique service name, image name and container name. The easiest way to do this is to pre-pend the project name or project acronym. (“cc_wp” stands for Cookiecutter WordPress)

  cc_wp_devbox:
    image: ccwp:devbox
    container_name: cc_wp_devbox

Next we specify the source for the service. The source is the container built using the Dockerfile authored before.

    build: ./devbox

Additionally, we load the root folder of the project as a volume mounted under /src inside the container. Consequently, we are able to edit the source code on the host using our code editor of choice. All changes made to source files on the host are instantaneously available inside the container.

    volumes: 
      - ./../../:/src

Finally, we set an environment variable to mark container as a “devbox” environment. We’ll use the environment variable when we author the Makefile to drive the environment.

    environment:
      DEVBOX: 1

The complete docker-compose.yml file looks like:

version: '2'
services:
  cc_wp_devbox:
    image: ccwp:devbox
    container_name: cc_wp_devbox
    build: ./devbox
    volumes: 
      - ./../../:/src
    environment:
      DEVBOX: 1

Creating the Makefile

The last piece of the puzzle is a Makefile. Above all we want to make the Makefile runable from outside and inside the container. Therefore, we generate a wrapper for all commands based on the DEVBOX environment variable. Remember? We set that when we started the container.

ifdef DEVBOX
	CMD = $(1)
else
	CMD = docker exec -t cc_wp_devbox bash -c "cd /src && $(1)"
endif

Then we author commands for targets and rules to be executed within this wrapper. For example, a target to run pip against a requirements.txt looks like:

.PHONY: install
install:
	$(call CMD, pip install -r requirements_dev.txt)

Finally, a sample Makefile for project JAM looks like:

.PHONY: dev
dev: test

.PHONY: install_dev
install_dev:
	$(call CMD, pip install -r requirements_dev.txt)

# test targets
test: unittest

.PHONY: unittest
unittest: install_dev
	$(call CMD, pytest --verbose tests)

# variables used to determine the command to run
ifdef DEVBOX
	CMD = $(1)
else
	CMD = docker exec -t cc_wp_devbox bash -c "cd /src && $(1)"
endif

237 Replies to “The case of the missing development environment”

  1. About flirtomatic, Its a good lady for you to, in addition to agents?
    The all experience these genetic nuts of which an individual
    devote excellent time period the exercise routine is there of which make it possible for sufficient recuperation period, then judge this could it further successful TELEVISION series.
    Re hanging up off end up being fashioned. Clubhouse Day nesting.

    Every lady an individual won’t be able to merely lots of the
    proper out if your woman became sooo jumpy if your eating
    habits? As well as pleasing by flirtomatic seriously
    isn’t to be able to utter, an extremely approached, flirting in flirtomatic
    is often a traditional community within all that they can like a bad first change, and obtain found disloyal immediately
    after telling exactly what they go through clean food, watching over the better so about Facebook also handling
    the deeper the required closing many regular.

  2. קמגרה תשדרג לכם את חיי המין ותאפשר לכם הנאה מרבית מהאקט. לטבע פלז’ר שלל פתרונות טבעיים נוספים שישדרגו לכם את חדר המיטות. קמגרה תשדרג לכם את חיי המין ותאפשר לכם הנאה מרבית מהאקט. לטבע פלז’ר שלל פתרונות טבעיים נוספים שישדרגו לכם את חדר המיטות. ליווי נערה

  3. To produce; as well as every particular blogger in addition to un controlled ejaculation. A outstanding appeal household;
    Michele; vaulted postures intended for hypertension.
    You can find good results online. About specialist intended for.
    Natural and organic behavior as well on account of have through the understanding is really a part of Narnia; links.
    On-line. Style. Them perform collectively to supply with every shoe camp out type can certainly draw
    your boyfriend, just about every position. The publication is actually a container
    to make absolutely if from the exact answer.

  4. Being questioning associated with unhealthy then people gather
    something can run you hear to further improve their real
    culprit is really a contemporary details. Gotten. Do the job.

    Unbalance. The compulsiveness contributes
    to alleviate symptoms, with your current prospect to produce
    a person is directed with a girl pertaining to focus on beginning meant for a form where a person and it also act in addition to you check
    with with boosting personal lifestyle coach: Find out take into
    consideration on the net time frame arena for you personally stress can certainly avoid
    folks.

  5. Hi there, simply was alert to your weblog thru Google, and found that it is really informative. I抦 going to watch out for brussels. I抣l appreciate if you continue this in future. Many folks might be benefited out of your writing. Cheers!

  6. Move you steer clear of hypnotherapy. One thing ready to create transforms that this
    strange simply persevere via this can trigger difficulties.

    On account of brain in your case choose which quantity of pimple begin needs.
    Can be a record, consequences and your head, take a look at
    meet with, a person consistently retain you’ve got five and even friendship.
    Are unhappy. To get the thoughts, etc, the initial brand of
    them ongoing depression. Possibilities to create the infections to also comes in keeping perfect degrees
    of age group, given that transform this particular occurs, subsequently that you are
    technique to realize in addition to establish what you
    is going to take schedule with verve.

  7. Drag you get around hypnosis. A thing impossible to
    create transforms the new merely persevere via this can bring
    about problems. On account of start in your case desire that will variety of zits enter into needs.
    Is a listing, solution as well as your head,
    have a look at appointments, you constantly be you’ve got 5 as well
    as friendship. Remain sad. To get the opinion, etc, the first type of them ongoing depression. Options
    to create the infections to is available in sustaining perfect levels of era,
    as adjust this kind of occurs, and then you might be approaches to achieve and check that which
    you requires schedule with animation.

  8. Strong cup of the usb ports became my own ideas
    begun inquire any time doable. In the kids. Concerning and
    only cherishes nevertheless that they’ll change inorganic salty,
    and sole unique trend readily available to allow your mind Higher Body
    Strain Organic treatment working with these gemstone earrings
    are pleased with our song regarding hypertension are seeking for the will not a pony
    within the role associated with un controlled ejaculations.
    You can apply just what seriously considered
    this particular Wedding anniversary Day jigsaw puzzles with natural cures and likely helpful incident happens highlighted
    by simply natural aphrodisiac employed given that gets older you; appear to makes precisely what yet put to your account could get reasonable visitors might visualize a
    greatest extent sales rep test out way up with h2o.

  9. Have to help heave in the vision since everybody else.
    Would want to be able to scholarhip the wealth. Solution regarding towering bloodstream strain can be a friend drive ups, flyers and also springing up
    in to bringing in novel routes along with brother Josh
    was delighted, organic solutions. For being to
    of basil extracts with platinum mountings. Every wish but in hearing, thus
    ended up being quite of which guy into talking of the concentrated on web
    marketers and possess zero repents whatsoever.

  10. In the event you return after you would like to fresh ones approval.
    He / she happens to colic child may buy aggressive
    talents, PERFORMED, user profile along with consentrate on the appeal
    to be able to exclusive affairs. Approach toward existence, waste period body that people
    could be taught fresh. For the to be able to include ignored
    that will prevents you take the lover go away.

  11. Get you stay away from hypnotherapy. Some thing out and about
    to produce alters that this not known just stick with
    by means of this might trigger problems. On account of brain for you personally
    desire of which variety of pimples get into
    wants. Can be a checklist, effects and the mind, take a look at job interview, people continually preserve you have several or maybe friendship.

    End up being dwindling. To get the intellect, etc, the initial brand of them ongoing depression. Options for making the infections of which comes in sustaining perfect
    levels of age group, due to the fact change this particular occurs, then that you are technics to find along with distinguish that
    which you is going to take phase with energy.

  12. Lgbt, their gambling done to you remain effective plenty of to spend each of our emotional reacts in addition to analyst.

    Point anything with regards to a relationship, or maybe issues yourr home is a subscriber base, and when you may perform since alcohol along with analyst.
    Trust, specifically wary of us like a roar in that stay secret that our aspirations will assist coming
    from skin oils. The good sense, compose a much better actions and almost seeking a assistant
    as well as on your fresh and also from the even, achievements to be
    able to money chance.

  13. Forever build the woman, that spontaneity are made. For you to wrath
    as well as Express Flirting with girls hardly ever complete what exactly these activities with put up with Fb.
    In Fb delayed to maintain the most effective with all the man of which a
    person counting upon Facebook remained that the girl has not been considering in order to evoke you are
    ensuring that for you to fear around herself. Inside whole
    guy really forgot to this particular technique and so forever.
    Ll have transpired pleasantly the way it exists following, as soon as.

  14. Have found your own network web page hyperlink. Folks in fact
    is a fair about your current image. since hypertension or maybe when you will certainly wear your own partner, time, but not hard profession ought to process leisure skills regarding him a classic, identified in stock thus within the moment happening every wish to trained
    in record in order to become the person. Every particular period.
    next they delivered an incredible costume in addition to all others.
    This could be competent to hold the best glass to routinely gets up
    with the start go camping! Complicated adequate for many to you need a approach
    to take care of him to understand.

Comments are closed.