Add an automated VM test against a real Invoice Ninja instance
CI / test (push) Canceled after 0s
CI / Build on the declared MSRV (push) Canceled after 0s
CI / Latest stable (advisory) (push) Canceled after 0s

Adds an invoiceninja-nixos flake input and two things built from it:

- nixosConfigurations.invoiceninja-test-nginx/-caddy: manual test
  VMs, re-exported as-is from that repo rather than redefined here —
  same principle as this crate being a client of the SDK, not a
  second implementation of it.
- checks.invoiceninja-cli: an automated nixosTest (tests/, adapted
  from invoiceninja-nixos's own test scaffolding) that boots a real
  Invoice Ninja instance and runs the actual invoiceninja-cli binary
  against it - create/list/get/update/invoice/payment/delete across
  clients, invoices, and payments - rather than raw curl.

tests/common.nix's module import had to change from a relative
../nixos-module/invoiceninja.nix (which doesn't exist here) to a
parameter threaded from the flake input's nixosModules.invoiceninja.

Building the VM pulls from the invoiceninja-nixos cachix cache via a
nixConfig block, mirroring that repo's own.

Also fixes a wrong assumption the first version of this test's final
assertion made, caught by actually running it against a real server:
`clients list` with no `--status` does not exclude soft-deleted
records - confirmed empirically (Invoice Ninja returns everything
regardless of is_deleted unless status=active is passed explicitly).
Corrected the test to check is_deleted via `clients get` instead, and
fixed the misleadingly-incomplete --status doc comment in cli.rs that
had the same wrong assumption baked in.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-10-26 16:57:46 -04:00
co-authored by Claude Sonnet 5
parent 96fba9a303
commit c8a67b1208
10 changed files with 466 additions and 4 deletions
+3
View File
@@ -5,3 +5,6 @@
/.cargo/config.toml
/result
/result-*
# The test VM's disk image — created in the invocation directory by
# run-nixos-vm (see CLAUDE.md's "Testing against a real server").
/nixos.qcow2
+49
View File
@@ -112,6 +112,55 @@ The `test` and `msrv` jobs skip scheduled runs (`if: github.event_name !=
'schedule'`) — a pinned toolchain against a locked dependency graph can't
change on a timer, so the weekly run is for `stable` only.
### Testing against a real server
`flake.nix` has an `invoiceninja-nixos` input (Gitea, same host as this
repo's own remote) and re-exports two of its `nixosConfigurations` as
`invoiceninja-test-nginx`/`invoiceninja-test-caddy` — real NixOS VMs
running actual Invoice Ninja (PHP/Laravel + MariaDB + Redis + chromium for
PDF generation), not a mock. Reused as-is rather than redefined here, same
principle as this crate being a client of the SDK rather than a second
implementation of it — see that repo's own `CLAUDE.md` for what's actually
inside.
```sh
nix build --accept-flake-config '.#nixosConfigurations.invoiceninja-test-nginx.config.system.build.vm'
QEMU_NET_OPTS="hostfwd=tcp::8080-:80" ./result/bin/run-nixos-vm
```
`--accept-flake-config` is what lets the `nixConfig` cachix substituter
declared at the top of this file actually get used — without it (or if the
daemon's `trusted-users` rejects it anyway, which it did when scripted
through an agent sandbox in practice — built fine regardless, just slower)
Nix silently falls back to building from source. `QEMU_NET_OPTS` forwards
the guest's port 80 to host `8080`; it's a runtime flag to the generated
`run-nixos-vm` script, not baked into the VM config, so it has to be set on
every boot.
The VM has one fixed admin account baked in by the upstream repo's own test
fixtures (`admin@email.com` / `password`, from its `tests/invoice_ninja_test_password`) —
not meant to be secret, since it only ever exists inside a throwaway local
VM. There's no standing API token to copy anywhere; get one fresh each boot
via `/api/v1/login` (the SDK and CLI only ever use API-token auth, never
this login endpoint themselves — it's purely a test-fixture bootstrap
step):
```sh
TOKEN=$(curl -s -X POST http://localhost:8080/api/v1/login \
-H 'Content-Type: application/json' \
-d '{"email":"admin@email.com","password":"password"}' \
| jq -r '.data[0].token.token')
IN_API_TOKEN="$TOKEN" IN_BASE_URL=http://localhost:8080/api/v1 \
cargo run -- clients list
```
`jq` is in this flake's `devShells.default` for exactly this. There's also
an automated check (`checks.invoiceninja-cli` in `flake.nix`, backed by
`tests/`) that boots the same kind of VM and runs this CLI's own binary
against it non-interactively — see `tests/nixos-test.nix` for exactly what
it exercises. Verified working end to end (boot → login → token →
`clients create`/`list`) as of writing.
## Common commands
```sh
Generated
+168 -1
View File
@@ -1,5 +1,43 @@
{
"nodes": {
"flake-compat": {
"flake": false,
"locked": {
"lastModified": 1767039857,
"narHash": "sha256-vNpUSpF5Nuw8xvDLj2KCwwksIbjua2LZCqhV1LNRDns=",
"owner": "NixOS",
"repo": "flake-compat",
"rev": "5edf11c44bc78a0d334f6334cdaf7d60d732daab",
"type": "github"
},
"original": {
"owner": "NixOS",
"repo": "flake-compat",
"type": "github"
}
},
"flake-parts": {
"inputs": {
"nixpkgs-lib": [
"invoiceninja-nixos",
"nix",
"nixpkgs"
]
},
"locked": {
"lastModified": 1782949081,
"narHash": "sha256-vp6Y/Grm98ESt6ceOkWiHWyZRDV3J1RID4w+6NWK9yA=",
"owner": "hercules-ci",
"repo": "flake-parts",
"rev": "17c9d6cdfc60c64f4ee8d306f9bc0b4ccb51481e",
"type": "github"
},
"original": {
"owner": "hercules-ci",
"repo": "flake-parts",
"type": "github"
}
},
"flake-utils": {
"inputs": {
"systems": "systems"
@@ -18,7 +56,135 @@
"type": "github"
}
},
"git-hooks-nix": {
"inputs": {
"flake-compat": [
"invoiceninja-nixos",
"nix"
],
"nixpkgs": [
"invoiceninja-nixos",
"nix",
"nixpkgs"
]
},
"locked": {
"lastModified": 1783008725,
"narHash": "sha256-jGiy6+sxjNWXSjp25uoJuNfyH9zBK1PEDY0lVoL4ibQ=",
"owner": "cachix",
"repo": "git-hooks.nix",
"rev": "bca82caa46d5ec0f5d422c61fb1e30bc51313cbe",
"type": "github"
},
"original": {
"owner": "cachix",
"repo": "git-hooks.nix",
"type": "github"
}
},
"invoiceninja-nixos": {
"inputs": {
"nix": "nix",
"nixpkgs": "nixpkgs_2"
},
"locked": {
"lastModified": 1792704906,
"narHash": "sha256-RnfRh7mVgwqiTA+lbBCq35SllrINUu1I+klov26Aiq0=",
"ref": "refs/heads/master",
"rev": "badcdd25666d5a59d221fc4331f8691ce3167755",
"revCount": 71,
"type": "git",
"url": "ssh://gitea@gitea.awkawb.cloud/awkawb/invoiceninja-nixos.git"
},
"original": {
"type": "git",
"url": "ssh://gitea@gitea.awkawb.cloud/awkawb/invoiceninja-nixos.git"
}
},
"nix": {
"inputs": {
"flake-compat": "flake-compat",
"flake-parts": "flake-parts",
"git-hooks-nix": "git-hooks-nix",
"nixpkgs": "nixpkgs",
"nixpkgs-23-11": "nixpkgs-23-11",
"nixpkgs-regression": "nixpkgs-regression"
},
"locked": {
"lastModified": 1786478010,
"narHash": "sha256-/9wXGp+aLObxqQuOGhDOnbdV0QCRDHH8uSh/4URKVAg=",
"owner": "NixOS",
"repo": "nix",
"rev": "24b65e35d9d4c9d64c1bd5844a35f2a779356d63",
"type": "github"
},
"original": {
"id": "nix",
"type": "indirect"
}
},
"nixpkgs": {
"locked": {
"lastModified": 1783148766,
"narHash": "sha256-H9+N+GFtsbVC8ZniHliChM7ndizxtqVZs6bnGOLM3WQ=",
"rev": "a50de1b7d8a586adc18d2395c19de7d6058e6030",
"type": "tarball",
"url": "https://releases.nixos.org/nixos/26.05/nixos-26.05.4193.a50de1b7d8a5/nixexprs.tar.xz"
},
"original": {
"type": "tarball",
"url": "https://channels.nixos.org/nixos-26.05/nixexprs.tar.xz"
}
},
"nixpkgs-23-11": {
"locked": {
"lastModified": 1717159533,
"narHash": "sha256-oamiKNfr2MS6yH64rUn99mIZjc45nGJlj9eGth/3Xuw=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "a62e6edd6d5e1fa0329b8653c801147986f8d446",
"type": "github"
},
"original": {
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "a62e6edd6d5e1fa0329b8653c801147986f8d446",
"type": "github"
}
},
"nixpkgs-regression": {
"locked": {
"lastModified": 1643052045,
"narHash": "sha256-uGJ0VXIhWKGXxkeNnq4TvV3CIOkUJ3PAoLZ3HMzNVMw=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "215d4d0fd80ca5163643b03a33fde804a29cc1e2",
"type": "github"
},
"original": {
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "215d4d0fd80ca5163643b03a33fde804a29cc1e2",
"type": "github"
}
},
"nixpkgs_2": {
"locked": {
"lastModified": 1786430034,
"narHash": "sha256-Vux08kA5PICwS2sViCMfwVLAHNoH8TkKAeBo25LjpMI=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "70cc4559b10a6062b05ff1af17e0add065ccaed9",
"type": "github"
},
"original": {
"owner": "NixOS",
"ref": "nixos-26.05",
"repo": "nixpkgs",
"type": "github"
}
},
"nixpkgs_3": {
"locked": {
"lastModified": 1792773339,
"narHash": "sha256-rZLrzrC6i+jnTAlvPZHogKDulY7WWDEv9hJeHieO7Zw=",
@@ -37,7 +203,8 @@
"root": {
"inputs": {
"flake-utils": "flake-utils",
"nixpkgs": "nixpkgs",
"invoiceninja-nixos": "invoiceninja-nixos",
"nixpkgs": "nixpkgs_3",
"rust-overlay": "rust-overlay"
}
},
+59 -2
View File
@@ -1,14 +1,32 @@
{
description = "invoiceninja-cli: a command-line client for the Invoice Ninja v5 API";
# Mirrors invoiceninja-nixos's own nixConfig: without this, building
# `nixosConfigurations.invoiceninja-test-*` here rebuilds the whole PHP/UI
# app (and its chromium-headless-shell closure) from source instead of
# pulling the cached result.
nixConfig = {
extra-substituters = [ "https://invoiceninja-nixos.cachix.org" ];
extra-trusted-public-keys = [
"invoiceninja-nixos.cachix.org-1:TmRZaji8FFVjjIw9KZmvLrxMxfP+JDThOyOX04z526A="
];
};
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
rust-overlay.url = "github:oxalica/rust-overlay";
rust-overlay.inputs.nixpkgs.follows = "nixpkgs";
flake-utils.url = "github:numtide/flake-utils";
# For a real Invoice Ninja instance to test the CLI against — see
# CLAUDE.md's "Testing against a real server". Not `follows`-ed to this
# flake's own `nixpkgs`: it pins its own (nixos-26.05, as of writing),
# and `nixosConfigurations.test-nginx`/`test-caddy` below are consumed
# as already-built flake outputs, not re-evaluated against our
# nixos-unstable — the two never need to agree.
invoiceninja-nixos.url = "git+ssh://gitea@gitea.awkawb.cloud/awkawb/invoiceninja-nixos.git";
};
outputs = { self, nixpkgs, rust-overlay, flake-utils, ... }:
outputs = { self, nixpkgs, rust-overlay, flake-utils, invoiceninja-nixos, ... }:
let
cargoToml = builtins.fromTOML (builtins.readFile ./Cargo.toml);
@@ -60,7 +78,10 @@
in
{
devShells.default = pkgs.mkShell {
packages = [ rustToolchain pkgs.pkg-config pkgs.openssl ];
# jq is for parsing the /api/v1/login response when testing
# against a real server — see CLAUDE.md's "Testing against a
# real server".
packages = [ rustToolchain pkgs.pkg-config pkgs.openssl pkgs.jq ];
RUST_SRC_PATH = "${rustToolchain}/lib/rustlib/src/rust/library";
};
@@ -73,6 +94,30 @@
inherit (pkgs) lib pkg-config openssl;
inherit rustPlatform;
};
# Automated: boots a real Invoice Ninja instance (invoiceninja-nixos's
# module, not a mock) and runs actual `invoiceninja-cli` commands
# against it — create/list/get/update/delete across clients,
# invoices, and payments, including that `update` preserves fields
# it wasn't told about. `tests/` is a vendored, adapted copy of
# invoiceninja-nixos's own test scaffolding; see its
# nixos-test.nix for what it does and doesn't check (it
# deliberately skips things like PDF generation and the language-
# list regression, which are that repo's own concern, not this
# CLI's). Uses invoiceninja-nixos's own pinned nixpkgs (26.05), not
# this flake's nixos-unstable, to build+evaluate the VM/module —
# matches what that module was actually tested against. The CLI
# binary itself (`self.packages.${system}.default`, built against
# our own nixos-unstable) runs fine inside it regardless: Nix
# binaries carry their own closure, so differing nixpkgs revisions
# in the same system closure is an explicitly supported case, not a
# version mismatch to work around.
checks.invoiceninja-cli = import ./tests/nixos-test.nix {
nixpkgsPath = invoiceninja-nixos.inputs.nixpkgs;
inherit system;
invoiceninjaModule = invoiceninja-nixos.nixosModules.invoiceninja;
invoiceninjaCliPkg = self.packages.${system}.default;
};
}) // {
# For a flake that wants `invoiceninja-cli` in its own `pkgs`, not for
# this flake's own `packages.default`/`devShells.default` above (those
@@ -85,5 +130,17 @@
overlays.default = final: _prev: {
invoiceninja-cli = final.callPackage mkInvoiceninjaCli { };
};
# Real Invoice Ninja instances to test this CLI against, reusing
# invoiceninja-nixos's own test VMs as-is rather than redefining them
# here — same principle as this crate being a client of the SDK, not
# a second implementation of it. Known test credentials baked into
# that repo's tests/common.nix: admin@email.com / password (from
# tests/invoice_ninja_test_password). See CLAUDE.md for how to boot
# one and get an API token from it.
nixosConfigurations = {
invoiceninja-test-nginx = invoiceninja-nixos.nixosConfigurations.test-nginx;
invoiceninja-test-caddy = invoiceninja-nixos.nixosConfigurations.test-caddy;
};
};
}
+6 -1
View File
@@ -161,7 +161,12 @@ pub struct ListArgs {
pub sort: Option<String>,
#[arg(long)]
pub filter: Option<String>,
/// e.g. "active,archived,deleted" to include archived/deleted records
/// Restricts which records come back, e.g. "active" to exclude
/// archived/deleted ones. **Omitting this flag does not do that
/// filtering for you** — confirmed against a real server, `list` with
/// no `--status` returns every record regardless of `is_deleted`/
/// `archived_at`. Pass "active,archived,deleted" (or any subset) to
/// narrow it.
#[arg(long)]
pub status: Option<String>,
/// Comma-separated relationships to eager-load, e.g. "contacts"
+23
View File
@@ -0,0 +1,23 @@
{ proxyServer ? "nginx", invoiceninjaModule }:
{ ... }:
{
# Unlike the invoiceninja-nixos repo this was copied from, there's no
# sibling `nixos-module/` directory here — the module comes in through
# this parameter instead, threaded from flake.nix as
# `invoiceninja-nixos.nixosModules.invoiceninja` (the flake input). The
# module's own `../package.nix` relative import inside that file still
# resolves correctly regardless of who imports it, since it's relative to
# the module file's own location in the Nix store, not to this directory.
imports = [ invoiceninjaModule ];
services.invoiceninja = {
enable = true;
proxy.server = proxyServer;
adminAccount.passwordFile = ./invoice_ninja_test_password;
secretFile = ./test.env;
settings.APP_DEBUG = true;
};
networking.firewall.enable = false;
}
+52
View File
@@ -0,0 +1,52 @@
{ proxyServer ? "nginx", invoiceninjaModule }:
{ pkgs, modulesPath, ... }:
{
imports = [
(modulesPath + "/profiles/qemu-guest.nix")
# Not imported by default for a plain nixosSystem in this nixpkgs (only nixosTest nodes get it
# automatically) - needed explicitly so virtualisation.cores/memorySize below are real options.
(modulesPath + "/virtualisation/qemu-vm.nix")
(import ./common.nix { inherit proxyServer invoiceninjaModule; })
];
# `nix flake check` evaluates config.system.build.toplevel directly, which asserts a real root
# filesystem and bootloader are configured. `just build-vm` builds config.system.build.vm
# instead, where the qemu-vm module overrides both of these with a tmpfs root and direct
# kernel boot (virtualisation.useBootLoader defaults to false) - so these values just need to
# satisfy the assertions and are otherwise unused.
fileSystems."/" = {
device = "/dev/disk/by-label/nixos";
fsType = "ext4";
};
boot.loader.grub.enable = false;
users.users.root = {
initialPassword = "toor";
};
services.resolved.enable = true;
# `just boot-vm` forwards the guest's port 80 to host port 8080 (QEMU_NET_OPTS in the justfile),
# but the backend builds absolute URLs (e.g. the stored company logo path) from APP_URL, which
# defaults to plain "http://localhost" with no port. The SPA's own relative URLs are fine (they
# inherit whatever port the browser is already on), but any backend-generated absolute URL then
# points at host port 80 - where nothing is listening - instead of the forwarded 8080, giving a
# silent net::ERR_CONNECTION_REFUSED for things like the logo image. Override to match the actual
# forwarded port.
services.invoiceninja.settings.APP_URL = "http://localhost:8080";
# The qemu-vm module's defaults (1 core, 1024MB) are too tight for real interactive use: MySQL +
# Redis + PHP-FPM + chromium (PDF generation, spawned per request - each is a full multi-process
# browser, not a lightweight renderer) running concurrently can exhaust 1GB fast, and this VM
# boots with panic_on_oom - so it's not a graceful slowdown, the whole VM goes down. Confirmed by
# reproducing a "logo upload hangs" report this way: at the 1024MB default, even a couple of
# concurrent PDF-generation requests triggered `redis-server invoked oom-killer` and a kernel
# panic. 2 cores / 2560MB comfortably handles 2 concurrent PDF generations alongside normal
# traffic (verified: ~1.4GB still free afterward); a deliberate flood of many concurrent PDF
# requests can still exhaust even this, since each is a full chromium instance - if that's a
# concern for your deployment, consider capping `phpfpm.settings."pm.max_children"` below its
# default of 8, or provisioning more RAM.
virtualisation.cores = 2;
virtualisation.memorySize = 2560;
}
+1
View File
@@ -0,0 +1 @@
password
+103
View File
@@ -0,0 +1,103 @@
{ nixpkgsPath, system, invoiceninjaModule, invoiceninjaCliPkg }:
let
pkgs = import nixpkgsPath { inherit system; };
adminEmail = "admin@email.com";
adminPassword = pkgs.lib.strings.trim (builtins.readFile ./invoice_ninja_test_password);
in
pkgs.testers.nixosTest {
name = "invoiceninja-cli";
nodes = {
# Only one node/proxy backend: which proxy Invoice Ninja sits behind is
# invoiceninja-nixos's own concern (its checks exercise both nginx and
# caddy already). This test is about whether invoiceninja-cli works
# against a real server, which doesn't depend on which proxy fronts it.
machine = {
imports = [ (import ./common.nix { proxyServer = "nginx"; inherit invoiceninjaModule; }) ];
environment.systemPackages = [ invoiceninjaCliPkg ];
};
};
testScript = ''
import json
def api_post(machine, path, payload, token=None):
auth_header = f"-H 'X-API-TOKEN: {token}' " if token else ""
return json.loads(machine.succeed(
f"curl -sf -X POST http://localhost{path} "
+ auth_header
+ "-H 'Content-Type: application/json' "
+ f"-d '{json.dumps(payload)}'"
))
def cli(machine, token, *args):
quoted_args = " ".join(f"'{a}'" for a in args)
return machine.succeed(
f"IN_API_TOKEN={token} IN_BASE_URL=http://localhost/api/v1 "
f"invoiceninja-cli --output json {quoted_args}"
)
machine.start()
# phpfpm-invoiceninja/invoiceninja-queue-worker both Require= invoiceninja-data-setup,
# so systemd already waits for it to finish before either starts.
machine.wait_for_unit("mysql.service")
machine.wait_for_unit("redis-invoiceninja.service")
machine.wait_for_unit("phpfpm-invoiceninja.service")
machine.wait_for_unit("invoiceninja-queue-worker.service")
machine.wait_for_unit("nginx.service")
machine.wait_for_open_port(80)
with subtest("admin account can log in via the API"):
login_response = api_post(machine, "/api/v1/login", {"email": "${adminEmail}", "password": "${adminPassword}"})
assert "${adminEmail}" in json.dumps(login_response), f"login did not succeed: {login_response}"
token = login_response["data"][0]["token"]["token"]
with subtest("invoiceninja-cli can create and list a client"):
cli(machine, token, "clients", "create", "--data", '{"name": "CLI Test Client"}')
listing = cli(machine, token, "clients", "list")
clients = json.loads(listing)
assert any(c["name"] == "CLI Test Client" for c in clients), f"client not in list output: {listing}"
client_id = next(c["id"] for c in clients if c["name"] == "CLI Test Client")
with subtest("invoiceninja-cli update preserves fields it wasn't told about"):
cli(machine, token, "clients", "update", client_id, "--data", '{"city": "Springfield"}')
updated = json.loads(cli(machine, token, "clients", "get", client_id))[0]
assert updated["city"] == "Springfield", f"city not updated: {updated}"
assert updated["name"] == "CLI Test Client", f"update blanked a field it wasn't told about: {updated}"
with subtest("invoiceninja-cli can create an invoice against that client and see it in a payment"):
invoice_out = cli(
machine, token, "invoices", "create", "--data",
json.dumps({
"client_id": client_id,
"line_items": [{"quantity": 1, "cost": 100, "product_key": "test-product"}],
}),
)
invoice = json.loads(invoice_out)[0]
assert invoice["amount"] == 100, f"unexpected invoice amount: {invoice}"
payment_out = cli(
machine, token, "payments", "create", "--data",
json.dumps({
"client_id": client_id,
"amount": invoice["amount"],
"invoices": [{"invoice_id": invoice["id"], "amount": invoice["amount"]}],
}),
)
payment = json.loads(payment_out)[0]
assert payment["applied"] == invoice["amount"], f"payment not fully applied: {payment}"
with subtest("invoiceninja-cli delete soft-deletes the client"):
cli(machine, token, "clients", "delete", client_id)
# Not `clients list`: Invoice Ninja's default /clients (no `status`
# param) returns every record regardless of is_deleted — confirmed
# against a real server, contrary to what CLAUDE.md's `--status`
# doc comment implies. Check the flag directly instead of relying
# on list-exclusion.
deleted = json.loads(cli(machine, token, "clients", "get", client_id))[0]
assert deleted["is_deleted"], f"delete did not set is_deleted: {deleted}"
'';
}
+2
View File
@@ -0,0 +1,2 @@
APP_KEY=base64:gJnNI8pOx4c0/Z6kl9mIjn0X9XkbJUWjtqnDFl9prvo=
API_KEY=test-api-token