InferenceDoctor: autotuning LLM serving configs, verified

Your GPUs are slow and expensive, and the fix is usually a config change nobody tried. Tensor parallelism, max batch size, sequence length: these get set once from vendor defaults or by one heroic engineer, and then every model change silently invalidates the tuning. I built InferenceDoctor to make this systematic, and I just open-sourced the whole thing, history and all.

v0.1: see what's slow

The first version only diagnoses. A benchmark CLI talks to any OpenAI-compatible endpoint (vLLM, TensorRT-LLM) and measures p50/p95 end-to-end latency, aggregate output tokens per second, and dollars per million output tokens when you supply a GPU hourly cost. It writes a portable JSON report, and a client-side Next.js dashboard renders it. No backend, no uploads, API keys never touch the report. This part is Apache-2.0 and always was.

v0.2: stop guessing, start searching

Diagnosis tells you where you are, not where to go. v0.2 added an experimental autotuning engine behind the CLI's planning API: it fingerprints your workload from numeric traces (input, output, and arrival distribution summaries), screens candidate configs with a conservative memory model, fits Gaussian-process surrogates in log units for latency, throughput, and error, and spends a fixed trial budget using expected improvement plus uncertainty. Then comes the part I cared about most: reserved verification trials run the winning config on fresh data, and if verification fails, the engine withholds the recommendation and rolls back instead of blessing a regression. Every trial lands in a hash-chained SQLite ledger, so the record of what was tried is tamper-evident.

v0.3: spend the budget where you were wrong

Before building further, I tore down six prior-art systems at the mechanism level: AIConfigurator, SLO-Guard, TUNA, CherryPick, Triton Model Analyzer, and BurstGPT. That teardown verified seven capabilities none of them provide, and v0.3 rebuilds the search thesis around those gaps. The core bet: verification residuals are the scarce signal. Prediction errors from held-out verification trials steer where the next experiment goes, weighted toward regions near the Pareto frontier, with verification residuals carrying double weight. A dollar value-of-information stop rule halts the search when the next experiment's expected saving drops below its cost (verification of the current best still runs). And the surrogate finally conditions on the workload itself, through a configuration-by-workload product kernel, instead of a fingerprint that never influenced a decision.

What it is not

Honest scope matters more than a demo. There is no hardware validation yet, no H100 numbers, no claim of beating anything on real silicon. The prior-art field is crowded, and I make no novelty or patentability claim; no patent has been filed. The repo history shows the full arc, v0.1 through v0.3, including the teardown notes, so you can see exactly what was tried and what remains unproven.

What's next

The next milestone is a reproducible real-staging experiment on one small model, followed by an independent workload and hardware holdout. If you serve LLMs on your own GPUs, the most useful thing you can give me is skepticism: what would make this trustworthy enough to try? The repo is at github.com/sriranga13/inferencedoctor.