Methodology
Every number on this site comes from a stated source. This page explains how we get the numbers and how we label the pages where we did not measure anything ourselves.
How a test runs
- The app runs in Docker on our lab host. We use the pinned image version from the Compose file and record that version and the test date on the page.
- We bring the stack up from a clean state and wait for it to report healthy. We do thisthree times, each time from a fresh stack, and publish the median and the range (next section).
- We run the app's setup steps (for example creating the first user), then measureidle RAM and CPU once the stack's memory has settled. What counts as settled is defined below.
- We apply a stated load, then record peak RAM under that load. The page says what the load was.
- We record image size for every container and startup time from start to first healthy response.
- We keep the verified Compose file and the raw results so you can check them.
Runs, median and range
A single run is one sample, and our lab host used to differ from one session to the next (see below). So every published result is the median of three full runs. Each run starts from nothing: the stack is brought up, set up, left to settle, measured at idle, put under the stated load, and torn down with its volumes removed. Image downloads are cached between runs; nothing else is.
- Median. The published idle RAM is the median of the three idle figures. The published peak RAM is the median of the three run peaks (each run's peak is the highest simultaneous total in that run). Startup time, startup peak and idle CPU are medians too.
- Range. Under each figure we print the lowest and highest run, for example "268 MiB" with "range 234–270 · 3 runs" below it. The Test box has a table with every run, including the CPU model of the lab host in that run. The raw JSON holds every run in full.
- Variance flag. If the highest run is more than 1.5 times the lowest for idle RAM or for peak RAM, the result carries a varies between runs note, on the Test box (which also lists the per-run values) and in the homepage table. We still publish the median, but you should read it together with the range.
- If the lab host changes between runs of one test (a different CPU model), the Test box says so.
- A result from fewer than three runs, or with a fixed settle wait, is a development run. The site build refuses to publish it. A page that still shows "1 run" is a result from before 2026-10-06 that we have not re-run yet; it has no range.
Why we repeat runs
On Oct 4, 2026 Jellyfin 12.1 idled at 189 MiB in our lab, which at that time was a cloud sandbox host with 4 vCPUs. On Oct 6 the same version, the same compose file and the same media idled at about 270 MiB. In between, that host had changed: its CPU model string went from 2.80 GHz to 2.10 GHz. Three runs of Jellyfin on the Oct 6 host (two on 12.2, one on 12.1) gave 234, 268 and 270 MiB, and the three runs that now back the published page, later the same day, gave 260, 225 and 222 MiB. We cannot say what caused the 80 MiB gap to Oct 4, but the app version was not it, so a single run presented as exact would have been wrong by that much depending on the day.
Some apps also looked bimodal from run to run on that sandbox host. On Oct 6, Vaultwarden 1.37.4 idled at 15.1 MiB in one run and 7.7 MiB in the next, and a control run of 1.37.3 idled at 17.1 MiB. On the GitHub Actions runner (below) Vaultwarden 1.37.4 idled at 7.07 MiB in the median run and between 7.06 and 7.33 MiB in all three, with no variance flag. We do not know why the sandbox runs split. A median with a range and a flag tells you about such a split, where one number would not. The repeats cost lab time, not accuracy: the rule for what counts as idle is unchanged.
How we measure idle RAM
Some apps use more memory for a few minutes after they start than they do afterwards. A fixed wait would measure that start-up phase and call it idle, so we wait for the memory to settle and then measure. The rule, exactly:
- From the moment the stack is healthy, we sample memory continuously: one
docker statspass for every container, about every 2 to 3 seconds. Memory is the cgroup working set (usage minus inactive file cache), in MiB. - The settle clock starts when the setup steps finish (or at healthy, if the app has none). The stack is settled when, over the last 60 seconds, the total memory of all containers at the same instant varied (highest minus lowest sample) by less than the larger of 2% of the window's mean and 10 MiB.
- We never call it settled before 120 seconds, and we stop waiting after 600 seconds. Both count from the settle clock. All five numbers (window, percentage, floor, minimum, maximum) are defaults that a test can override. The Test box prints the ones used.
- Then we take the idle window: at least 30 seconds and at least 10 samples. If the idle window is not itself flat by the same threshold, or its mean differs from the settle window's by more than the threshold, the stack was not really settled and we keep waiting (still within the 600-second limit).
- If 600 seconds pass without settling, we still measure an idle window, but the result is marked not settled and the Test box says so in plain text.
We also record the startup peak (the highest memory, per container and for the stack total, between healthy and the end of the settle phase) and a memory timeline from healthy to the end of the idle window. The Test box shows the startup peak, the time it took to settle and a small chart of the timeline with the same data as a table.
One exception: for an app that declares a scheduled job whose memory changes after its first run (today only Nextcloud's cron container), the idle window starts only after one scheduled run of that job (counted from the clock, not read from the job). See the 2026-10-18 entry.
What this does not catch: a stack that is flat for a whole 60-second window and only then changes. The idle window check catches a change within the next 30 seconds, nothing later. Memory growth over days is not measured at all.
About the lab host
Since Oct 8, 2026 every published result comes from the same machine type: a GitHub Actions standard hosted runner (ubuntu-latest, Ubuntu 24.04) with 2 vCPUs and 8 GB of RAM(7.7 to 7.8 GiB visible to the operating system), running Docker 28.0.4 and Docker Compose 2.38.2 with the overlay2 storage driver. The Nov 3, 2026 re-tests (Jellyfin and its ladder, Uptime Kuma, Vaultwarden with Caddy and the Vaultwarden backup lab, Pi-hole) ran on a newer runner image (20261026.180.1): Ubuntu 26.04.1, kernel 7.0.0, Docker 29.4.2 and Compose 5.1.3 with the containerd image store, which the result files record (containerd_image_store: true). Unpacked image sizes from those jobs are therefore the containerd figures and are larger than the overlay2 ones for the same image, as the table notes on each page say. Each lab job (one app, three runs) starts on a fresh runner, and the runner has no proxy in front of its internet access. The result files record the runner image and the host label ("github-actions ubuntu-latest (2 vCPU, 8 GB)").
The same machine type is not always the same physical CPU, and the vendor can differ too. The Oct 8 runs landed on three AMD EPYC models (7763, 9V74 and 9V45); the Oct 11 concurrency ladder ran on an Intel Xeon Platinum 8573C; the Oct 15 Immich re-test landed on EPYC 9V74 and 7763 and, for one variant, an Intel Xeon Platinum 8370C; the Oct 21 patch re-test (Immich, Home Assistant, Uptime Kuma) landed on EPYC 7763, 9V74 and 9V45 and, for one Immich stack, an Intel Xeon 6973P-C. Five of those seven results (all but the Immich variant without machine learning and Uptime Kuma) were also the first on a newer runner image, with Linux kernel 7.0.0. The Oct 28 patch re-test (Immich, Home Assistant, Jellyfin and its concurrency ladder, Infinite Scale, Syncthing, FileBrowser Quantum) landed on EPYC 7763, 9V74 and 9V45 and, for two results, on an Intel Xeon: Platinum 8370C for the 2 GiB Immich stack and Platinum 8573C for FileBrowser Quantum. It used two runner images, kernel 7.0.0 (image 20261026) for six results and kernel 6.17.0 (image 20261023) for five, so the same app can land on either. Other published results ran on kernel 6.17.0. The Nov 3 patch re-test (Jellyfin, Jellyfin ladder, Uptime Kuma, Vaultwarden with Caddy, Vaultwarden backup, Pi-hole) landed on EPYC 7763 for four results, EPYC 9V74 for the Jellyfin ladder and an Intel Xeon Platinum 8573C for Pi-hole, all on runner image 20261026.180.1. Its date matches the Oct 28 image (20261026.354.1), but it is a different build: Docker 29.4.2 with the containerd image store instead of Docker 28.0.4 with overlay2. More models and images may appear. In every result so far all three runs were on one model, and the Test box says so if they are not. We print the CPU model and RAM next to every result, and the CPU model of each run in the Test box. Results from the earlier sandbox host (4 vCPUs, 15.7 GiB, Intel Xeon at 2.10 or 2.80 GHz, Docker 29.6.2) are no longer published; where an article still quotes them, it says "earlier host".
RAM and image size carry over to other machines well. CPU, throughput and tokens-per-second figures are only valid for a 2-vCPU runner, so we label them that way and never compare them with numbers from other people's hardware or with our own earlier host. The unpacked size Docker reports for an image depends on the Docker version and image store, so it differs between our two hosts for the same image; compressed sizes are the registry's numbers and do not. How we turn that into a disk recommendation is below.
How we size hardware
The "Where to run it" section of a guide turns our measurements into a RAM recommendation. The rule is: take the highest reading in any of our runs, that is, for each run the higher of the peak under load and the startup peak, and then the highest of those over the runs. Multiply it by 1.5 and round up to the next of 1, 2, 4, 8, 16 or 32 GB. The extra half covers the operating system, Docker, file cache and spikes our sampling can miss.
Sizing deliberately uses the highest run, not the median. The page shows the medians as the typical figures, but a machine sized for the median would be too small on the day the app lands on the high side of its range.
Disk for the images
Docker has two image stores and they use different amounts of disk for the same image. The classicoverlay2 store keeps only the unpacked layers. The containerd image store, which is the default on fresh installs of Docker Engine 29 and later, keeps the compressed layers as downloaded and the unpacked layers. Docker's documentation says the containerd store "uses more disk space than the legacy storage drivers for the same images" (containerd image store) and that an upgraded install keeps using overlay2 until you switch.
Our lab host uses overlay2, so the unpacked size in a result would understate the disk a new Docker 29+ install uses. Each result records both the unpacked size (from Docker on the lab host) and the compressed size (from the registry). For each stack we add up the distinct images once, then show two figures: the unpacked size, which is the disk with the classic overlay2 store, and anestimate for the containerd store: unpacked plus compressed size, plus 10%, rounded up. The 10% comes from the same image digests on our earlier lab host (Docker 29.6.2, containerd store), where Docker reported between 1.01 and 1.07 times unpacked plus compressed. "Where to run it" recommends the larger figure, and the Test box shows both. This covers the images only. Your data volumes, database growth and logs come on top, as before.
Limits: the containerd figure is an estimate from two measured sizes, not a measurement on a containerd host (the lab host runs overlay2). Docker has changed how it counts image size in several 29.x releases, so what docker image ls ordocker system df prints on your machine can differ from our figures. Sources and the comparison table are in our lab note lab/notes/disk-sizing-2026-10-10.md.
Evidence labels
Each page shows one label near the top.
| Label | What it means |
|---|---|
| Tested | Backed by a Docker run in our lab, with published results. |
| Researched | Specs, prices and programme terms read from vendor pages on a stated date. Nothing was benchmarked. |
| Mixed | Researched specs combined with our measured app numbers, for example for sizing. |
A page only changes label when new data is actually published on it, and the change is noted in its update log.
What we do not claim
We never claim hands-on testing of hardware we have not tested. If we have not run a mini PC, NAS, drive or VPS ourselves, the page says "Researched" and tells you where the specs came from.
Who does the work
AI agents (Anthropic's Claude) run the lab tests and draft the pages. A separate AI review checks every figure against the lab data. Everything is published under the editorial oversight of Simon Carter, the site editor, who is accountable for what is published. We say this openly because you should know how the site is made.
Every article carries a short line near the top that says the same thing for that page. It reads differently by evidence label, because a page we did not test must not claim a test:
- Tested. How this was made: AI agents ran the lab test and drafted this page; a separate AI review checked every figure against the lab data; published under the editorial oversight of Simon Carter, the site editor.
- Mixed. How this was made: AI agents ran the lab test for the measured app numbers and drafted this page; a separate AI review checked those figures against the lab data; the specs and prices are researched from vendor pages, not tested by us; published under the editorial oversight of Simon Carter, the site editor.
- Researched. How this was made: AI agents researched and drafted this page from vendor and project sources; nothing on it was lab-tested; a separate AI review checked it against those sources; published under the editorial oversight of Simon Carter, the site editor.
In the page markup, the author of each article is the site (SelfHostBench) and Simon Carter is listed as its editor. The byline on each article reads "Site editor: Simon Carter".
Methodology changes
2026-10-18: Nextcloud idle is measured after the first 5-minute mark of its cron job
Before. Nextcloud's cron container runs cron.php every 5 minutes, at the 5-minute marks of the clock. The idle window sometimes came before the container's first run and sometimes after it. The container holds 0.6 MiB before its first run and about 34 MiB after it (about 1 MiB in one run we measured), so the same stack read about 35 MiB apart. The container also bursts to 150 to 190 MiB in some runs, and single-run peaks of the same app ranged from 362 to 497 MiB.
Why we changed it. Two batches of the same app disagreed for a reason unrelated to the app version. The Oct 8 batch (Nextcloud 35.0.1) read 299 MiB idle and the Oct 17 batch (35.0.2) read 265 MiB, while the other three containers idled at 264.7 and 264.2 MiB together. The difference sat in the cron container, mostly in whether its first run came before the idle window.
What changed. This applies only to an app whose test definition declares a scheduled job, which today means Nextcloud. After the stack has settled by the rule above, the lab waits until three things are true: a 5-minute mark has passed since the stack became healthy, the last mark is at least 60 seconds old, and the next mark is more than 40 seconds away (the 30-second idle window plus 10 seconds), so that no mark falls inside the window. It then takes the idle window as before. The extra wait is capped at 360 seconds; if the cap were reached, the window would start anyway and the run would carry a warning. The marks are computed from the clock, not read from the job, so the lab does not know whether cron.phpreally started or how long it ran. The result file now records the marks, the wait, whether the idle window began at least 60 seconds after a mark, and whether a mark fell inside the load window. In the first batch measured this way (Oct 18, three runs) the extra wait was 63.7 to 91.0 seconds (mean 73.8), the cap was not reached, the idle window began 61.0 to 61.6 seconds after a mark, and no mark fell inside an idle window or a load window.
From this batch the lab also records two diagnostics for every container of a multi-container stack: the cgroup memory.stat (anonymous memory, file cache and related counters) right after the idle window and at the highest sample of the load, and a per-container memory and CPU series for the load window. The tables and charts on this site still show Docker's working set. In the Oct 18 batch, memory.stat showed that the cron container's 34 MiB after its first run is anonymous memory (33.2 to 33.3 MiB; file cache 0.23 MiB), not file cache.
What did not change. The settle rule, the idle window length, the load steps, the median of three runs, the sizing rule and every other app. No other result was re-run. The peak is still the highest simultaneous total in the load window, so it includes cron's memory when cron bursts. The marks do not predict a burst: in the Oct 18 batch two of three upload windows had one (147 and 167 MiB) although no mark fell inside either window, and the bursts began 115 and 128 seconds after the last mark. We do not know what starts them, and we do not know why the container sometimes falls back to about 1 MiB.
Effect on the numbers. Nextcloud's idle and its startup peak are now taken after at least one 5-minute mark (if the cap were reached, the run would say so). The peak and the sizing reading can still be set by a burst, which the marks do not predict. The published figures moved from 265 to 297 MiB idle, from 367 to 452 MiB peak and from 449 to 479 MiB for the highest reading used for sizing; the sizing tier stays 1 GB. The new idle figure is close to the Oct 8 result (299 MiB), which also came after a first run. The Nextcloud guide, the three-way file-server comparison and the Google Drive alternatives page were updated on Oct 18, and each page's update log says so.
2026-10-17: one wording for the editor's role
The article byline, the "How this was made" line, the footer and this page now use the same wording: the site is published under the editorial oversight of Simon Carter, the site editor. The byline reads "Site editor: Simon Carter" (it used to read "Edited by"). Nothing about how we measure changed, and no new information about the editor was added.
2026-10-11: a concurrency ladder measures how many streams the lab host keeps up with
Before. The Jellyfin article had no table of simultaneous transcodes after Oct 9, because the earlier table came from the 4-vCPU sandbox host and did not apply to the runner.
What changed. A new lab app, jellyfin-ladder (same Compose file and Jellyfin version as the Jellyfin app, its own result file), runs a ladder of 1, 2, 3, 4, 6 and 8 simultaneous software transcodes. Each player has its own device ID and plays the same 1080p H.264 film, asking for a 720p, 4 Mb/s HLS stream that cannot be direct played, and fetches the segments in real time for 60 s of playback per level. The keep-up rule is the one the manual ladders of Oct 4 and Oct 6 used: a level kept up when no segment on any player arrived after its playhead needed it. The ladder stops after the first level that clearly fails (at least half of the players late on a quarter or more of their segments), and the levels above it are recorded as not run. The whole ladder is run 3 times from a fresh stack. A level's verdict is "kept up" only if it kept up in every run that measured it; speed, late segments, first-segment delay and RAM are the median over the runs, with the range. The Jellyfin article's stream table is built from that result and is back, with the host and CPU model next to it.
What did not change. The idle and peak RAM method, the sizing rule, the median of three runs, and the Jellyfin app's own result. The ladder result has no Test box of its own, because its idle figure is not published. CPU and speed figures remain valid for the lab host only.
2026-10-10: disk for images is sized for both Docker image stores
Before. "Where to run it" and the Test box showed the unpacked image size Docker reported on the lab host. After the move to the GitHub Actions runner (Docker 28.0.4, overlay2) that was the smaller of two possible figures.
Why we changed it. Docker Engine 29 and later uses the containerd image store on fresh installs, and Docker's documentation says it keeps images in both compressed and unpacked form and uses more disk space than overlay2. Someone who installs Docker today would very likely need more disk than the runner's number suggested. On our earlier host (Docker 29.6.2) the same image digests were reported at up to 1.07 times unpacked plus compressed size.
What changed. The disk recommendation for images is now unpacked plus compressed size, plus 10%, rounded up (the containerd store), and the overlay2 figure is shown next to it. The rule is described above. Image disk figures went up by 43% to 62% on every page (Nextcloud 1.9 GB to 2.9 GB, Immich 3.3 GB to 4.9 GB, Jellyfin 1.7 GB to 2.8 GB, Home Assistant 2.4 GB to 3.4 GB, Vaultwarden with Caddy 366 MB to 530 MB). The articles that quoted an image size were updated, and each page's update log says so.
What did not change. The measured sizes in the result files, the RAM sizing rule, and the advice on data volumes and growth. Nothing was re-run.
2026-10-09: results come from a GitHub Actions runner (2 vCPU, 8 GB)
Before. Results came from a cloud sandbox host with 4 vCPUs and 15.7 GiB of RAM. It was not the same machine from session to session: the CPU model changed from 2.80 GHz to 2.10 GHz, and some figures moved with it (see above).
Why we changed it. Consistency. A fixed, documented machine type that every app runs on makes the figures comparable with each other and repeatable, and it removes the sandbox host as a variable. The lab harness already ran on any Docker host, so the procedure did not change.
What changed. On Oct 8, 2026 we re-ran all 11 lab apps three times each on a standard GitHub Actions runner. All runs succeeded and settled, and none carried a variance flag. Immich (v3.2.4 to v3.3.0) and Home Assistant (2026.9.4 to 2026.10.0) were updated in the same step, so for those two the new figures mix a version change with the host change. The six articles were updated to the new numbers on Oct 9; each page's update log says so.
Effect on the numbers. CPU-bound figures changed a lot. Jellyfin's one-stream software transcode ran at 3.07x real time against 6.2x, and used both vCPUs; the table of simultaneous transcodes came from the old host and was removed until it was re-measured on the runner (it came back on Oct 11, see below). Immich's job queues took 54 s against 39 s. RAM figures mostly stayed close: idle RAM moved by about 15% or less for every app (Home Assistant 311 to 358 MiB with a version change, Nextcloud 343 to 299 MiB, Immich 1,321 to 1,380 MiB, Jellyfin 225 to 233 MiB, Vaultwarden with Caddy 24 to 25 MiB). Load peaks moved more where the load depends on how fast the host serves requests: Vaultwarden with Caddy peaked at 70 MiB against 91. The memory-cap tests for Immich also came out differently (the 4 GiB cap killed no machine-learning worker in three runs, against eight of eight on the old host); the Immich articles give the counts and say what we can and cannot conclude. Unpacked image sizes are lower on the runner. The compressed (registry) sizes of the unchanged images did not move, so the images did not shrink; the difference comes from the image store: Docker 28 with overlay2 reports the unpacked size only, and Docker 29 with the containerd image store keeps the compressed layers as well (see the 2026-10-10 entry).
What did not change. The settle rule, the idle window, the load steps, the median of three runs and the sizing rule. Manual checks quoted in the articles (restores, updates, permission and proxy tests) were made on the earlier sandbox host and were not repeated; the articles say which version and host each one was observed on.
2026-10-06: results are the median of three runs, with the range
Before. Each result was one run, printed as an exact number.
Why we changed it. On Oct 6, 2026 we found that the lab host differs between sessions (CPU model 2.10 GHz instead of 2.80 GHz) and that the same Jellyfin 12.1 idled at 189 MiB on Oct 4 and at 270 MiB on Oct 6. Vaultwarden is bimodal: idle 7.7 MiB in one run, 15 to 17 MiB in others. A single run presented as exact misleads.
What changed. The lab now does three full fresh runs per app. Idle RAM is the median of the runs, peak RAM is the median of the run peaks, and every figure shows its range and the number of runs. A result is flagged "varies between runs" when the highest run is more than 1.5 times the lowest for idle or peak. The sizing rule uses the highest run (the higher of the load peak and the startup peak, over all runs), times 1.5, so sizing stays conservative. Result files carry schema_version 3 with every run recorded in full. We re-ran every published result with three runs on 2026-10-06 and updated every page that quotes them; each page's update log says so.
What did not change. The settle rule, the idle window, the load steps, image sizes and the version pinning are unchanged. A result we have not re-run yet shows "1 run" and no range.
2026-10-05: every article says how it was made
Each article now carries a short "How this was made" line near the top, and the section above states the same facts. Article markup now names the site as author and Simon Carter as editor. Nothing about how we measure changed.
2026-10-02: idle is measured after memory settles, not after 60 seconds
Before. The lab waited a fixed 60 seconds after the app was healthy and set up, then sampled for about 30 seconds and published that as idle RAM.
Why we changed it. On Oct 1, 2026 we watched Immich's server container for ten minutes after a start. It held 1.4 to 1.5 GiB for the first minutes, then dropped to about 0.7 GiB roughly two minutes after the stack became healthy. A 60-second wait plus a 30-second window lands before that drop, so our idle figures for Immich described a start-up phase and overstated the steady-state RAM. We publish these numbers as measurements, so they had to describe steady state.
What changed. The settle rule above, plus the startup peak and the memory timeline in every result. We re-ran every lab result under the new rule on 2026-10-02 and updated every page that quotes them; each page's update log says so. Result files now carry schema_version 2.
What did not change. Peak RAM under load is measured as before: the highest simultaneous total during the load step. Image sizes, startup time and the load itself are unchanged.
Prices and affiliate links
Prices come from vendor pages and carry the date we read them. Affiliate relationships never change a result or a ranking; see the affiliate disclosure.