tests: give the interactive VM enough RAM to not OOM under real use
Root-caused a "logo upload hangs, connection refused in browser console" report: the qemu-vm module's defaults (1 core, 1024MB) are too tight for this stack (MySQL + Redis + PHP-FPM + chromium for PDF generation, each PDF request spawning a full multi-process browser instance) under real concurrent load, and this VM boots with panic_on_oom - so exhausting memory doesn't degrade gracefully, the whole VM goes down and every connection starts refusing. Reproduced directly: at the 1024MB default, even 2 concurrent PDF generation requests triggered `redis-server invoked oom-killer` and a kernel panic. Bumped to 2 cores / 2560MB, which comfortably survives that same load (~1.4GB still free afterward) - verified via the exact concurrent-request scenario that crashed the 1024MB config, not just absence of errors. virtualisation.cores/memorySize aren't options on a plain nixosSystem in this nixpkgs (only nixosTest nodes get the qemu-vm module automatically), so this also adds an explicit import of it. Documented the underlying resource cost (chromium is a full browser per PDF request, pm.max_children=8 means up to 8 can run concurrently) in CLAUDE.md as an operational note for real deployments, not just this test VM. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -39,7 +39,8 @@ This project uses `just` (see `justfile`) and Nix directly.
|
|||||||
and disables `boot.loader.grub` purely to satisfy NixOS's built-in assertions when `nix flake check` evaluates
|
and disables `boot.loader.grub` purely to satisfy NixOS's built-in assertions when `nix flake check` evaluates
|
||||||
`config.system.build.toplevel` directly — the qemu-vm module overrides both with a tmpfs root and direct kernel
|
`config.system.build.toplevel` directly — the qemu-vm module overrides both with a tmpfs root and direct kernel
|
||||||
boot when actually building `config.system.build.vm` (what `just build-vm` uses), so these values are otherwise
|
boot when actually building `config.system.build.vm` (what `just build-vm` uses), so these values are otherwise
|
||||||
unused.
|
unused. Also raises `virtualisation.cores`/`memorySize` to 2/2560 (qemu-vm's defaults, 1 core/1024MB, are too
|
||||||
|
tight for real interactive use — see the module architecture notes below on PDF generation's memory cost).
|
||||||
- `nixos-test.nix` — the automated test (`pkgs.testers.nixosTest`, exposed as `checks.<system>.invoiceninja` in
|
- `nixos-test.nix` — the automated test (`pkgs.testers.nixosTest`, exposed as `checks.<system>.invoiceninja` in
|
||||||
`flake.nix`). Builds its own `pkgs` (with `allowUnfree` baked in — `nixosTest` forbids setting `nixpkgs.config`
|
`flake.nix`). Builds its own `pkgs` (with `allowUnfree` baked in — `nixosTest` forbids setting `nixpkgs.config`
|
||||||
on a node once `pkgs` is externally supplied) and boots two nodes from `common.nix` — one with
|
on a node once `pkgs` is externally supplied) and boots two nodes from `common.nix` — one with
|
||||||
@@ -126,7 +127,11 @@ expect Nix to report the correct hash on a failed build (or use `nix-prefetch`/`
|
|||||||
`127.0.0.1:80`/`[::1]:80` too — otherwise visiting the VM via the IP literal instead of `localhost` matches no
|
`127.0.0.1:80`/`[::1]:80` too — otherwise visiting the VM via the IP literal instead of `localhost` matches no
|
||||||
vhost and Caddy silently returns an empty `200 OK` (a blank page with no error anywhere).
|
vhost and Caddy silently returns an empty `200 OK` (a blank page with no error anywhere).
|
||||||
- Chromium is required at runtime for PDF generation (`PDF_GENERATOR=snappdf`, `SNAPPDF_CHROMIUM_PATH`) and is
|
- Chromium is required at runtime for PDF generation (`PDF_GENERATOR=snappdf`, `SNAPPDF_CHROMIUM_PATH`) and is
|
||||||
added to the PHP-FPM and data-setup services' `path`.
|
added to the PHP-FPM and data-setup services' `path`. Each PDF request launches a full chromium instance (not
|
||||||
|
a lightweight renderer), and PHP-FPM's default `pm.max_children = 8` means up to 8 of these can run
|
||||||
|
concurrently — on a memory-constrained deployment this can exhaust RAM (confirmed: even a couple of
|
||||||
|
concurrent PDF requests OOM-killed a 1024MB VM). Size deployments accordingly, or lower
|
||||||
|
`phpfpm.settings."pm.max_children"`.
|
||||||
|
|
||||||
### `flake.nix`
|
### `flake.nix`
|
||||||
|
|
||||||
|
|||||||
@@ -4,6 +4,9 @@
|
|||||||
{
|
{
|
||||||
imports = [
|
imports = [
|
||||||
(modulesPath + "/profiles/qemu-guest.nix")
|
(modulesPath + "/profiles/qemu-guest.nix")
|
||||||
|
# Not imported by default for a plain nixosSystem in this nixpkgs (only nixosTest nodes get it
|
||||||
|
# automatically) - needed explicitly so virtualisation.cores/memorySize below are real options.
|
||||||
|
(modulesPath + "/virtualisation/qemu-vm.nix")
|
||||||
(import ./common.nix { inherit proxyServer; })
|
(import ./common.nix { inherit proxyServer; })
|
||||||
];
|
];
|
||||||
|
|
||||||
@@ -26,4 +29,18 @@
|
|||||||
};
|
};
|
||||||
|
|
||||||
services.resolved.enable = true;
|
services.resolved.enable = true;
|
||||||
|
|
||||||
|
# The qemu-vm module's defaults (1 core, 1024MB) are too tight for real interactive use: MySQL +
|
||||||
|
# Redis + PHP-FPM + chromium (PDF generation, spawned per request - each is a full multi-process
|
||||||
|
# browser, not a lightweight renderer) running concurrently can exhaust 1GB fast, and this VM
|
||||||
|
# boots with panic_on_oom - so it's not a graceful slowdown, the whole VM goes down. Confirmed by
|
||||||
|
# reproducing a "logo upload hangs" report this way: at the 1024MB default, even a couple of
|
||||||
|
# concurrent PDF-generation requests triggered `redis-server invoked oom-killer` and a kernel
|
||||||
|
# panic. 2 cores / 2560MB comfortably handles 2 concurrent PDF generations alongside normal
|
||||||
|
# traffic (verified: ~1.4GB still free afterward); a deliberate flood of many concurrent PDF
|
||||||
|
# requests can still exhaust even this, since each is a full chromium instance - if that's a
|
||||||
|
# concern for your deployment, consider capping `phpfpm.settings."pm.max_children"` below its
|
||||||
|
# default of 8, or provisioning more RAM.
|
||||||
|
virtualisation.cores = 2;
|
||||||
|
virtualisation.memorySize = 2560;
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user