Table of Contents

Background

At my last job, buzzwords were used constantly. One of the many that was used a lot was “closing the sim-to-real gap”. This referred (I think) to how to make simulators as close to real-life as possible. While in the context of flight simulators, I tend to think it’s silly, one thing I feel passionately about is making a software development environment as close to the real thing as possible.

Differences between a local development environment and where code is actually deployed or tested always causes problems. CI/CD jobs fail while it worked fine on your machine, missing dependencies in one environment, or differing versions of libraries wrecking havoc. Since this a purely software problem with (hopefully!) no physical forces acting on the developer, this is far more achievable problem to solve. Today I want to go over 3 tools I have been using lately to close the gap between my development environment and production and CI/CD.

Pre-Commit

I’ve talked about this before so I won’t repeat too much here. Basically, pre-commit allows you to easily set up tasks that must run before a commit can be made. Everything is configured in a single YAML file, and can easily be run locally and in CI/CD with the command pre-commit run --all-files.

This is really awesome for things like enforcing formatting, preventing checked-in merge conflicts, checking for trailing whitespace, static analysis, etc. There’s an entire library of pre-made hooks available here and it’s pretty straight forward to create your own in a variety of languages.

For me, this has helped considerably standardize formatting. While I’m a big believer in enforcing consistent formatting in a codebase, remembering to always run formatting before committing and pushing code was a constant challenge. And even if I did remember, sometimes I would forget command line options (some tools I loved completely lacked any configuration file support). So then I’d inevitably write a script to do that, but if it was in Bash, then it wouldn’t really work on Windows, so I’d convert it to Python, and whoops! I re-invented the same thing but worse.

Example

.pre-commit-config.yaml

 1default_language_version:
 2  python: python3.11
 3repos:
 4  - hooks:
 5      - id: check-json
 6      - id: check-toml
 7      - id: check-yaml
 8      - id: check-case-conflict
 9      - id: trailing-whitespace
10      - id: check-merge-conflict
11    repo: https://github.com/pre-commit/pre-commit-hooks
12    rev: v4.4.0
13  - hooks:
14      - id: poetry-check
15      - args:
16          - --no-update
17        id: poetry-lock
18    repo: https://github.com/python-poetry/poetry
19    rev: 1.5.1
20  - hooks:
21      - id: black
22    repo: https://github.com/psf/black
23    rev: 23.3.0
24  - hooks:
25      - args:
26          - --fix
27        id: ruff
28    repo: https://github.com/astral-sh/ruff-pre-commit
29    rev: v0.0.275
30  - hooks:
31      - id: pyleft
32    repo: https://github.com/nathanvaughn/pyleft
33    rev: v1.2.2
34  - hooks:
35      - id: pyright
36    repo: https://github.com/RobertCraigie/pyright-python
37    rev: v1.1.316
38  - hooks:
39      - id: markdownlint
40    repo: https://github.com/igorshubovych/markdownlint-cli
41    rev: v0.35.0

 1$ pre-commit run --all-files
 2check json...............................................................Passed
 3check toml...............................................................Passed
 4check yaml...............................................................Passed
 5check for case conflicts.................................................Passed
 6trim trailing whitespace.................................................Passed
 7check for merge conflicts................................................Passed
 8poetry-check.............................................................Passed
 9poetry-lock..............................................................Passed
10black....................................................................Passed
11ruff.....................................................................Passed
12pyleft...................................................................Passed
13pyright..................................................................Passed
14markdownlint.............................................................Passed

Dev Containers

Dev Containers are the most game-changing development in the last few years for software development, in my opinion. To explain why, I’d like to tell a story.

Right before Dev Containers were being integrated in VS Code and were really a thing, a coworker and myself were being contracted as DevOps engineers to a large defense company. One of the problems this defense company faced was that they had a group of third-party contractors that worked on the codebase for this project. These third parties were contributing drivers and interfaces for their hardware devices. This defense company did not specify a standardized development environment to their contractors. They only gave them some vague outlines like “we use Boost, it runs on Linux, … and make sure it compiles with GCC!”.

One week before a major test event, all the third parties were brought together to merge their code together.

All hell broke loose. Every third party had a different interpretation of what the development environment should be. Some used CentOS, others used various versions Ubuntu, and I’m pretty sure one did all of their work on Windows in Visual Studio with Comic Sans on a purple background. Every single one of these “environments” had a different version of gcc and much of the code wouldn’t compile on the other versions of gcc that the other third parties were using.

My coworker and I got called in to unravel the mess and rapidly.

What we did was give all the third parties a version of the Docker container we were working on for deployment and packaging of the software. This was a Linux distribution with every package and library needed to compile and run the software with a pinned version number. We then made a script to easily launch the container with the repository bind-mounted in. We told all the third parties:

Use this to do your development work. If your code compiles here, it will compile when we package and deploy the software.

Once that got sorted out, we made another container using the development image as the base, copied in the code, and basically just ran make. From then on out, we had no more problems with compiler versions, code not compiling, and we looked like heros.

All of this is to say Dev Containers is an actual proper specification to do the same thing, and is not something two people hacked together in a very panicked week. VS Code natively supports it, and there is a GitHub Action and Azure Pipelines Task available as well: https://github.com/devcontainers/ci.

As described in my story, a smart way to structure containers is to have your production image be the same or based on the development image. This is a great way to have the same environment no matter where code is developed/tested/run. I understand this does not always make sense. As always, discretion is required.

Lastly, another good way to use Dev Containers is for development environments that are not cross-platform, or have lots of dependencies. For example, I worked on a web app that was a Python monolith. Python was obviously required, Node was required for some of the frontend assets, and Java was also needed for some of the AWS emulation tools. Getting 3 runtimes installed was always a pain, so a Dev Container was an amazing way to easily get everything set up and repeatedly. I use Dev Containers like this to compile PX4 which requires lot of Linux-only ARM compilers: https://github.com/bellflight/AVR-PX4-Firmware/blob/main/.devcontainer/devcontainer.json

Example

Here is a quick example.

.devcontainer/devcontainer.json

 1{
 2  "build": {
 3    "dockerfile": "Dockerfile"
 4  },
 5  "customizations": {
 6    "vscode": {
 7      "extensions": [
 8        "actboy168.tasks",
 9        "charliermarsh.ruff",
10        "christian-kohler.path-intellisense",
11        "cschlosser.doxdocgen",
12        "davidanson.vscode-markdownlint",
13        "eamodio.gitlens",
14        "github.vscode-github-actions",
15        "gruntfuggly.todo-tree",
16        "jeff-hykin.better-cpp-syntax",
17        "matepek.vscode-catch2-test-adapter",
18        "ms-azuretools.vscode-docker",
19        "ms-python.python",
20        "ms-vscode.cpptools-extension-pack",
21        "ms-vscode.cpptools",
22        "njpwerner.autodocstring",
23        "ue.alphabetical-sorter"
24      ]
25    }
26  }
27}

GitHub Actions Workflow file

 1name: Build & Test
 2
 3on:
 4  push:
 5    branches: ["main"]
 6  pull_request:
 7    branches: ["main"]
 8
 9jobs:
10  build-and-test:
11    name: Build & Test
12    runs-on: ubuntu-latest
13    if: "${{ !contains(github.event.head_commit.message, 'ci skip') }}"
14
15    steps:
16      - uses: actions/checkout@v3
17
18      - name: Build
19        uses: devcontainers/ci@v0.3
20        with:
21          runCmd: |
22            cmake -B build/ -DCMAKE_BUILD_TYPE=${{ inputs.build_type }}
23            cmake --build build/ -j

VS Code Task Runner

One day, my good friend was telling me about how he likes using VS Code Tasks to run common build commands. After looking at his CI/CD configuration, I realized he had the exact same commands laid out there as well. This made me think, why do we have to type this out twice? I thought about options:

  • Makefiles: As a Python person, I find the syntax confusing, and they really only work on Linux.
  • Task: This looks pretty cool, but it’s still something I have to figure out how to install repeatedly and get into PATH. My experiences with protoc have shown me how frustrating this can be. And still, this would require another configuration file.
  • Other language-specific tools like npm: Michael in particular was working on a C++ project. I write a lot of Python, and Python doesn’t have a de facto option either.

The .vscode/tasks.json file seemed like a good enough option. I already liked using it, it has a lot of features and dependency system, and with extensions like actboy168.tasks you can get handy buttons in the editor to run tasks.

Microsoft has said they won’t add functionality to run VS Code tasks programmatically in microsoft/vscode#112594. I was only able to find one project that tried to do the same thing, but it is pretty limited and only supports running tasks with Bash. So like any developer procrastinating on other projects, I decided to write my own.

I created vscode-task-runner which can be easily installed with pip or pipx. I recommend reading the GitHub page for a full explanation of usage, but the jist is that you can run tasks with the vtr command followed by the task label(s).

1vtr pre-commit tests

I tried to implement as much of the same functionality from VS Code as I felt reasonable. This required a lot of digging through the VS Code source code. It turns out escaping characters in a variety of different shells is a nightmarish endeavour, and I did not want to reinvent the wheel here. As someone who writes as little Javascript as I can, this was a challenge. Thankfully ChatGPT became very useful for converting some of the Typescript code into Python. Additionally, some of the documentation Microsoft has on the VS Code website is different from what VS Code really does. VS Code tended to be more lenient than described, so I tried to as closely replicate the leniency.

Overall, I’m super pleased with how it turned out. It’s now super easy to run the same commands reliably on my computer and in CI/CD.

Example

.vscode/tasks.json

 1{
 2  "version": "2.0.0",
 3  "tasks": [
 4    {
 5      "label": "install",
 6      "type": "shell",
 7      "command": "poetry install --sync"
 8    },
 9    {
10      "label": "build",
11      "type": "shell",
12      "command": "poetry build",
13      "dependsOn": ["install"]
14    }
15  ]
16}

My computer:

 1$ vtr build
 2[1/2] Executing task "install": C:\Program Files\WindowsApps\Microsoft.PowerShell_7.3.5.0_x64__8wekyb3d8bbwe\pwsh.exe -Command poetry install --sync
 3Installing dependencies from lock file
 4
 5No dependencies to install or update
 6
 7Installing the current project: pyleft (1.2.2)
 8[2/2] Executing task "build": C:\Program Files\WindowsApps\Microsoft.PowerShell_7.3.5.0_x64__8wekyb3d8bbwe\pwsh.exe -Command poetry build
 9Building pyleft (1.2.2)
10  - Building sdist
11  - Built pyleft-1.2.2.tar.gz
12  - Building wheel
13  - Built pyleft-1.2.2-py3-none-any.whl

GitHub Actions Workflow File

 1name: Publish
 2
 3on:
 4  workflow_dispatch:
 5  push:
 6    branches:
 7      - main
 8
 9jobs:
10  publish:
11    runs-on: ubuntu-latest
12    permissions:
13      id-token: write
14      contents: write
15
16    if: "${{ !contains(github.event.head_commit.message, 'ci skip') }}"
17
18    steps:
19      - name: Checkout Code
20        uses: actions/checkout@v3
21
22      - name: Install poetry/vscode-task-runner
23        run: |
24          pipx install poetry
25          pipx install vscode-task-runner          
26
27      - name: Build
28        run: vtr build
29
30      - name: Publish package distributions to PyPI
31        uses: pypa/gh-action-pypi-publish@release/v1

GitHub Actions Log

 1Run vtr build
 2  vtr build
 3  shell: /usr/bin/bash -e {0}
 4  env:
 5    PROJECT_VERSION: 1.2.2
 6
 7[1/2] Executing task "install": /usr/bin/bash -c poetry install --sync
 8Installing dependencies from lock file
 9
10Package operations: 18 installs, 1 update, 0 removals
11
12  • Installing distlib (0.3.6)
13  • Installing exceptiongroup (1.1.1)
14  • Installing filelock (3.12.0)
15  • Installing iniconfig (2.0.0)
16  • Installing packaging (23.1)
17  • Installing platformdirs (3.5.0)
18  • Installing pluggy (1.0.0)
19  • Updating setuptools (67.8.0 -> 67.7.2)
20  • Installing tomli (2.0.1)
21  • Installing cfgv (3.3.1)
22  • Installing coverage (7.2.5)
23  • Installing identify (2.5.23)
24  • Installing nodeenv (1.7.0)
25  • Installing pytest (7.4.0)
26  • Installing pyyaml (6.0)
27  • Installing virtualenv (20.23.0)
28  • Installing pathspec (0.11.1)
29  • Installing pre-commit (2.21.0)
30  • Installing pytest-cov (4.1.0)
31
32Installing the current project: pyleft (1.2.2)
33[2/2] Executing task "build": /usr/bin/bash -c poetry build
34Building pyleft (1.2.2)

Conclusion

Hopefully this gives you some ideas on how to get your local dev environment to more closely resemble your production environment. An exact duplicate is not always possible, but I find the closer you can make it, the less headache you will have later. As always, exact implementation details depend on the languages, application type, industry, etc.

Particularly with Python, certain things like testing multiple versions of Python or multiple operating systems are not things I’ve personally found benefit setting up locally. These I tend to find much easier to configure in CI/CD, which can take advantage of parallel runners. If it was something that caused a lot of heartache, then setting something up locally could be done.