Integration Tests

Info

WIP, the underlying software is still being developed/ designed. Please treat these docs as an outline and not as an authorative source.

ISM has rudimentary integration tests based on NixOS VM tests for the incus executor.

The end-goal here is to provide full e2e tests however we will try to start small with only ISM. The problem with integrating PB backend is:

  1. no stable tests yet, are supposed to come soon-ish
  2. heavy on dependencies (OIDC server, (eventually) PostgreSQL)
  3. no stable modules yet

Code structure

The base for all integration tests is in nix/tests/lib.nix. It is effectively a prelude for tests, which configures an Incus instance with an appropiate firewall for testing with ISM.

The function within the file is supposed to wrapp all tests and based on Jörg Thalheim's blog entry: 'How to use NixOS testing framework with flakes'.

This function has been adjusted to also pass through the container and isalink attribute sets. These sets contain the required context for bootstrapping incus with container images and using isa-replay.

Testing approach

The main goal of the integration tests is to ensure that less broken code ships (duh). This is done with NixOS VM tests to ensure we:

  1. Have portable, isolated environment approximating songbird
  2. Have reproducible and automated interfaces for running tests on each code iteration

As such we would like to test for general functionality (creating sessions, using mounts, ...) and eventually also cover more edge cases.

Debugging

Debugging a NixOS test can be somewhat tricky. It requires you to be familiar with NixOS, the underlying test, the software under test and some Python.

  • For ISM, usually the software under test, you should be familiar with the architecture and networking.
  • For the test, consult the nix code in nix/tests/<test> and get familiar with the system under test (SUT), see nix/tests/lib.nix

Our current structure is an iteration of the design for NixOS tests is based on Jörg Thalheim's blog entry: 'How to use NixOS testing framework with flakes'. The twin article, How to execute NixOS tests interactively for debugging, outlines how you can get an interactive repl for the test.

To aid with the bootstrapping in the test repl, build the test with verbose logging:

  • nix build .#checks.x86_64-linux.<test>.driver --verbose

All tests have been written such that the required intstructions to, e.g., bootstrap the incus store are printed as verbose logs during the build process. This enables you to get the specific instructions required for setting up the test by hand.

Example: replays test

Let's assume the replays test fails after an ISM update due to a networking issue (firewall, etc.).

To build the driver use,nix build .#checks.x86_64-linux.replays.driver --verbose. The verbose log will contain (specific store paths may differ):

server.succeed("incus image import /nix/store/4a220p405kbyvia5ba7rvafvg0sz275b-tarball/tarball/*.xz /nix/store/hvcx74jvdk71zzmjccdlyghf4v91wha9-tarball/tarball/*.xz --alias ism")
# import isabelle lxc image from proveit.nix as 'isabelle'
server.succeed("incus image import /nix/store/4a220p405kbyvia5ba7rvafvg0sz275b-tarball/tarball/*.xz /nix/store/rx6v79s608j0qg1k2bl9gqb7mq2p3mc5-tarball/tarball/*.xz --alias isabelle")
# launch container named ism from 'ism' image
server.succeed("incus launch ism ism --no-profiles < /nix/store/2algkfbncip8cmf2rh9la8gf109pl2fx-incus.yml")

Copy these instructions to your clipboard or a scratch file.

Next, start the repl use: ./result/bin/nixos-test-driver --interactive

You should paste the previously saved instructions, press enter and wait a bit for the steps to complete. Afterward, your good to go for the next steps of the test.

You are then able to:

  • execute a command on the SUT,
    • like listing routes: server.succeed("ip a")
    • listing incus containers: server.succeed("incus ls")
  • execute a command an incus container,
    • like listing routes: server.succeed("incus exec ism ip a")
    • or viewing the journal of ISM: server.succeed("incus exec ism -- journalctl -xeu ism-main")
  • enter an interactive shell in the SUT: server.shell_interact()

Tests

  • replays: nix/tests/replays.nix
    • basic test, for incus executor, which executes replays from isalink