From b1fe86e9d5378129fd61513c846f8147efcc1d59 Mon Sep 17 00:00:00 2001 From: Andrew Bryant Date: Sat, 26 Sep 2026 16:36:22 -0400 Subject: [PATCH] tests: give the interactive VM enough RAM to not OOM under real use Root-caused a "logo upload hangs, connection refused in browser console" report: the qemu-vm module's defaults (1 core, 1024MB) are too tight for this stack (MySQL + Redis + PHP-FPM + chromium for PDF generation, each PDF request spawning a full multi-process browser instance) under real concurrent load, and this VM boots with panic_on_oom - so exhausting memory doesn't degrade gracefully, the whole VM goes down and every connection starts refusing. Reproduced directly: at the 1024MB default, even 2 concurrent PDF generation requests triggered `redis-server invoked oom-killer` and a kernel panic. Bumped to 2 cores / 2560MB, which comfortably survives that same load (~1.4GB still free afterward) - verified via the exact concurrent-request scenario that crashed the 1024MB config, not just absence of errors. virtualisation.cores/memorySize aren't options on a plain nixosSystem in this nixpkgs (only nixosTest nodes get the qemu-vm module automatically), so this also adds an explicit import of it. Documented the underlying resource cost (chromium is a full browser per PDF request, pm.max_children=8 means up to 8 can run concurrently) in CLAUDE.md as an operational note for real deployments, not just this test VM. Co-Authored-By: Claude Sonnet 5 --- CLAUDE.md | 9 +++++++-- tests/default.nix | 17 +++++++++++++++++ 2 files changed, 24 insertions(+), 2 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index bf20884..c5bd378 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -39,7 +39,8 @@ This project uses `just` (see `justfile`) and Nix directly. and disables `boot.loader.grub` purely to satisfy NixOS's built-in assertions when `nix flake check` evaluates `config.system.build.toplevel` directly — the qemu-vm module overrides both with a tmpfs root and direct kernel boot when actually building `config.system.build.vm` (what `just build-vm` uses), so these values are otherwise - unused. + unused. Also raises `virtualisation.cores`/`memorySize` to 2/2560 (qemu-vm's defaults, 1 core/1024MB, are too + tight for real interactive use — see the module architecture notes below on PDF generation's memory cost). - `nixos-test.nix` — the automated test (`pkgs.testers.nixosTest`, exposed as `checks..invoiceninja` in `flake.nix`). Builds its own `pkgs` (with `allowUnfree` baked in — `nixosTest` forbids setting `nixpkgs.config` on a node once `pkgs` is externally supplied) and boots two nodes from `common.nix` — one with @@ -126,7 +127,11 @@ expect Nix to report the correct hash on a failed build (or use `nix-prefetch`/` `127.0.0.1:80`/`[::1]:80` too — otherwise visiting the VM via the IP literal instead of `localhost` matches no vhost and Caddy silently returns an empty `200 OK` (a blank page with no error anywhere). - Chromium is required at runtime for PDF generation (`PDF_GENERATOR=snappdf`, `SNAPPDF_CHROMIUM_PATH`) and is - added to the PHP-FPM and data-setup services' `path`. + added to the PHP-FPM and data-setup services' `path`. Each PDF request launches a full chromium instance (not + a lightweight renderer), and PHP-FPM's default `pm.max_children = 8` means up to 8 of these can run + concurrently — on a memory-constrained deployment this can exhaust RAM (confirmed: even a couple of + concurrent PDF requests OOM-killed a 1024MB VM). Size deployments accordingly, or lower + `phpfpm.settings."pm.max_children"`. ### `flake.nix` diff --git a/tests/default.nix b/tests/default.nix index 5c270d0..114f4c7 100644 --- a/tests/default.nix +++ b/tests/default.nix @@ -4,6 +4,9 @@ { imports = [ (modulesPath + "/profiles/qemu-guest.nix") + # Not imported by default for a plain nixosSystem in this nixpkgs (only nixosTest nodes get it + # automatically) - needed explicitly so virtualisation.cores/memorySize below are real options. + (modulesPath + "/virtualisation/qemu-vm.nix") (import ./common.nix { inherit proxyServer; }) ]; @@ -26,4 +29,18 @@ }; services.resolved.enable = true; + + # The qemu-vm module's defaults (1 core, 1024MB) are too tight for real interactive use: MySQL + + # Redis + PHP-FPM + chromium (PDF generation, spawned per request - each is a full multi-process + # browser, not a lightweight renderer) running concurrently can exhaust 1GB fast, and this VM + # boots with panic_on_oom - so it's not a graceful slowdown, the whole VM goes down. Confirmed by + # reproducing a "logo upload hangs" report this way: at the 1024MB default, even a couple of + # concurrent PDF-generation requests triggered `redis-server invoked oom-killer` and a kernel + # panic. 2 cores / 2560MB comfortably handles 2 concurrent PDF generations alongside normal + # traffic (verified: ~1.4GB still free afterward); a deliberate flood of many concurrent PDF + # requests can still exhaust even this, since each is a full chromium instance - if that's a + # concern for your deployment, consider capping `phpfpm.settings."pm.max_children"` below its + # default of 8, or provisioning more RAM. + virtualisation.cores = 2; + virtualisation.memorySize = 2560; }