Skip to main content
Lab notebook · services/flexinfer

flexinfer climb

Where flexinfer started, the summit it is working toward, the rules it may not break, and every merge of the last 30 days, each tied to the roadmap waypoint it served or marked unplanned.

Latest leg: 17 changes merged on 2026-11-06

Now climbingchecking

Checking the operator for recent events…

Merge-queue moves and narrated legs appear here when the operator is reachable.

Base, summit and ropes

Base

The problem the climb started from

Self-hosted inference on a mixed homelab fleet (two gfx1100 cards, a Radeon VII and a GTX 980 Ti) breaks in ways nobody measures: lanes that hog a card, crash on load or time out behind a cold start. FlexInfer is the Kubernetes operator and OpenAI-compatible proxy that serves the lab's LLM, embedding, rerank, speech, image, audio and video lanes with scale-to-zero, breakers, reroute and per-GPU certification, climbing toward a fleet where every request the proxy routes is answered and every backend change is certified before it serves.

Source: ROADMAP.md#current-status · climbing since 2026-07-02

Summit

Where it is trying to get to, as measured

Proxy request success ratio (rolling 1d)
99.5%
target ≥ 99.5%
−0.4 pts since Oct 30, away from target
Roadmap gates met (Now list)
50%
target ≥ 100%
2 of 4 checked, 1 verified (1 claimed, unverified)
±0 pts since Nov 7
Master pipeline success ratio (7d)
67.1%
target ≥ 90%
+67.1 pts since Oct 9, toward target
Merges per week
109 merges
target ≥ 40 merges
+101 merges since Oct 9, toward target

Source: .loom/climb.yaml · as of 2026-11-07

Ropes

What must hold while climbing

Read from the roadmap and spec · as of 2026-11-07

Altitude

Summit metrics as progress toward their targets: the top line is the target met, and the legend turns each metric’s line on or off. Every merge is a tick on the time axis, coloured by how it was joined to the route; inferred joins are dashed because they are a similarity match, not a key. When merges are too dense to tell apart they become one bar per day, split the same way, and a bar opens that day’s merges. Focus or hover a tick or bar for detail; the arrow keys move along the walk. Shaded bands are weather (main red, outages, breaker and storm holds) merged into episodes; review holds, the queue waiting on a person, run as the thin lane along the top. An inverted red marker stands on a merge that was later reverted (a false summit); flags are release tags. Lines and merges co-move here; nothing on this chart claims one caused the other.

Main red, Oct 8 15:27 – Oct 23 16:20 UTC, 360.9 h, flexinfer: main pipeline red.Main red, Oct 24 17:42–18:11 UTC, 0.5 h, flexinfer: main pipeline red.Main red, Oct 24 19:11–19:44 UTC, 0.5 h, flexinfer: main pipeline red.Main red, Oct 26 02:50–07:07 UTC, 4.3 h, flexinfer: main pipeline red.Main red, Oct 26 21:19 – Oct 27 05:42 UTC, 8.4 h, flexinfer: main pipeline red.Main red, Oct 28 06:18–15:49 UTC, 9.5 h, 2 bands, flexinfer: main pipeline red.Main red, Oct 30 15:35–15:57 UTC, 0.4 h, flexinfer: main pipeline red.Main red, Oct 30 23:46 – Oct 31 16:14 UTC, 16.5 h, flexinfer: main pipeline red.Main red, Oct 31 17:28–21:01 UTC, 3.6 h, flexinfer: main pipeline red.Main red, Nov 1 18:58–20:49 UTC, 1.8 h, flexinfer: main pipeline red.Main red, Nov 2 14:46–15:38 UTC, 0.9 h, flexinfer: main pipeline red.Main red, Nov 2 16:58–18:28 UTC, 1.5 h, flexinfer: main pipeline red.Main red, Nov 3 05:39–05:51 UTC, 0.2 h, flexinfer: main pipeline red.Main red, Nov 3 18:08–20:17 UTC, 2.2 h, flexinfer: main pipeline red.Main red, Nov 3 21:39 – Nov 4 00:40 UTC, 3 h, flexinfer: main pipeline red.Main red, Nov 5 01:25–03:09 UTC, 1.7 h, flexinfer: main pipeline red.Main red, Nov 5 05:12–05:32 UTC, 0.3 h, flexinfer: main pipeline red.Main red, Nov 5 07:06–07:43 UTC, 0.6 h, flexinfer: main pipeline red.Main red, Nov 5 17:20–17:38 UTC, 0.3 h, flexinfer: main pipeline red.Main red, Nov 5 18:20–18:34 UTC, 0.2 h, flexinfer: main pipeline red.Main red (still holding), Nov 6 23:19 – Nov 7 15:27 UTC, 16.1 h, flexinfer: main pipeline red.
targetrowroadmap textinferredunplannedowner mergesMain red

Weather, false summits and base camps: tab to the first mark or hover one for its detail; the arrow keys move between marks.

2026-11-07 · 1 merge: 1 unplanned (1 owner merges).

2026-11-07MR !1328join: unplannedoperator lane

docs(loom): declare the flexinfer climb in .loom/climb.yaml (loom-core .loom/209 C.18)

Served no written waypoint (off route: owner merge). Cost not recorded.

Commit 810b628fc7bd on GitHub →

Weather

21 episodes from 22 recorded bands (same kind, gaps of 30 minutes or less merged), about 433.4 hours held in the window (overlaps counted once); one is still holding.

  • Main red433.2 h21 episodes (22 bands), holding

The 10 longest episodes

  • Main red Oct 8 15:27 – Oct 23 16:20 UTC · 360.9 h · flexinfer's condition main pipeline red
  • Main red Oct 30 23:46 – Oct 31 16:14 UTC · 16.5 h · flexinfer's condition main pipeline red
  • Main red (still holding) Nov 6 23:19 – Nov 7 15:27 UTC · 16.1 h · flexinfer's condition main pipeline red
  • Main red Oct 28 06:18–15:49 UTC · 9.5 h · 2 bands · flexinfer's condition main pipeline red
  • Main red Oct 26 21:19 – Oct 27 05:42 UTC · 8.4 h · flexinfer's condition main pipeline red
  • Main red Oct 26 02:50–07:07 UTC · 4.3 h · flexinfer's condition main pipeline red
  • Main red Oct 31 17:28–21:01 UTC · 3.6 h · flexinfer's condition main pipeline red
  • Main red Nov 3 21:39 – Nov 4 00:40 UTC · 3 h · flexinfer's condition main pipeline red
  • Main red Nov 3 18:08–20:17 UTC · 2.2 h · flexinfer's condition main pipeline red
  • Main red Nov 1 18:58–20:49 UTC · 1.8 h · flexinfer's condition main pipeline red
Show the other 11 episodes
  • Main red Oct 24 17:42–18:11 UTC · 0.5 h · flexinfer's condition main pipeline red
  • Main red Oct 24 19:11–19:44 UTC · 0.5 h · flexinfer's condition main pipeline red
  • Main red Oct 30 15:35–15:57 UTC · 0.4 h · flexinfer's condition main pipeline red
  • Main red Nov 2 14:46–15:38 UTC · 0.9 h · flexinfer's condition main pipeline red
  • Main red Nov 2 16:58–18:28 UTC · 1.5 h · flexinfer's condition main pipeline red
  • Main red Nov 3 05:39–05:51 UTC · 0.2 h · flexinfer's condition main pipeline red
  • Main red Nov 5 01:25–03:09 UTC · 1.7 h · flexinfer's condition main pipeline red
  • Main red Nov 5 05:12–05:32 UTC · 0.3 h · flexinfer's condition main pipeline red
  • Main red Nov 5 07:06–07:43 UTC · 0.6 h · flexinfer's condition main pipeline red
  • Main red Nov 5 17:20–17:38 UTC · 0.3 h · flexinfer's condition main pipeline red
  • Main red Nov 5 18:20–18:34 UTC · 0.2 h · flexinfer's condition main pipeline red

Route as written and walked

The waypoints the roadmap lists, in written order, against the merges that served them. A waypoint walked before one written ahead of it is a slip; one with no merge in the window is dimmed.

12 written, 6 served, 195 off route

WrittenWalkedUnplannedwritten ordermerge time123456789101112Oct 9Oct 15Oct 21Oct 27Nov 2

12 written, 6 served, 195 off route: 12 written waypoints, 6 with a step in the window, 36 steps that served one and 195 off the written route. 5 waypoints were walked out of written order and 6 have no step in the window. The written rail is in the order the roadmap lists its waypoints, not on the time axis; the walked rail is.

Route as written

Now 1 waypoint

  1. 6.

    Now

    walking · walked out of written order · ROADMAP.md:114 @ 810b628f

    • MR !12432026-11-04join: roadmap text

Next 4 waypoints

  1. 1.

    Backend currency and gfx1100 vLLM gaps

    Next · walking · walked out of written order · ROADMAP.md:135 @ 810b628f

    • MR !11392026-10-25join: inferred 0.83
    • MR !12822026-11-04join: inferred 0.79
    • MR !12852026-11-05join: inferred 0.85
    • MR !12882026-11-05join: inferred 0.87
    • MR !12902026-11-05join: inferred 0.86
    • MR !13082026-11-05join: inferred 0.83
  2. 2.

    Runtime images built and promoted by CI, no pin drift

    Next · planned · no step in this window · ROADMAP.md:149 @ 810b628f

  3. 3.

    Put the idle gfx1100 cards to work for loom-core and Mills

    Next · done · walked out of written order · ROADMAP.md:158 @ 810b628f

    • MR !12032026-11-01join: roadmap text
  4. 4.

    Python 3.14 model-tools promoted through a ModelCache publisher canary

    Next · unverified · no step in this window · ROADMAP.md:168 @ 810b628f

Plans 3 plans, 4 slices; open a plan for its slices

  • mills bl flexinfer proxy lane breaker fails fast on upstream timeouts 20260 done · 0 walking · 1 planned
    1. 8. bl-flexinfer-proxy-lane-breaker-fails-fast-on-upstream-timeouts-202610 planned · no step in this window
  • mills bl flexinfer proxy lane breaker fails fast on upstream timeouts 2026100 done · 0 walking · 0 planned
    1. mills bl flexinfer vllm engine stall marks lane unready 2026100 done · 0 walking · 3 planned
      1. 10. bl-flexinfer-vllm-engine-stall-marks-lane-unready-202610 planned · no step in this window
      2. 11. vllm-readiness-probe-fails-while-stalled planned · no step in this window
      3. 12. controller-engine-stalled-condition-restart-gpuunresponsive planned · no step in this window

    Issues, specs and other waypoints 1 waypoint

    1. 5.

      Current Status

      walking · walked out of written order · ROADMAP.md:20 @ 810b628f

      • MR !11042026-10-22join: roadmap text
      • MR !11102026-10-23join: roadmap text
      • MR !11322026-10-24join: roadmap text
      • MR !11452026-10-24join: roadmap text
      • MR !11552026-10-25join: roadmap text
      • MR !11612026-10-25join: roadmap text
      • MR !11662026-10-25join: roadmap text
      • MR !11742026-10-26join: roadmap text
      • MR !11772026-10-26join: roadmap text
      • MR !11782026-10-26join: roadmap text
      • MR !11822026-10-26join: roadmap text
      • MR !11652026-10-27join: roadmap text
      • MR !11982026-10-30join: roadmap text
      • MR !12002026-10-30join: roadmap text
      • MR !12062026-10-30join: roadmap text
      • MR !12082026-10-31join: roadmap text
      • MR !12522026-11-02join: roadmap text
      • MR !12072026-11-02join: roadmap text
      • MR !12642026-11-03join: roadmap text
      • MR !12682026-11-03join: roadmap text
      • MR !12702026-11-03join: roadmap text
      • MR !12752026-11-04join: roadmap text
      • MR !12772026-11-04join: roadmap text
      • MR !12782026-11-04join: roadmap text
      • MR !12792026-11-04join: roadmap text
      • MR !12342026-11-04join: roadmap text

    Off the written route

    195 steps: 195 owner merges; the latest first.

    • MR !13282026-11-07owner mergesdocs(loom): declare the flexinfer climb in .loom/climb.yaml (loom-core .loom/209 C.18)
    • MR !13252026-11-06owner mergesops(gfx906): prewarm the master-rebuilt runtime 13e8d6aa on the Radeon VII
    • MR !13182026-11-06owner mergesfeat(runtime): RDNA3 W4A16 persistent split-K tickets (opt-in plugin) + GPTQ-RDNA3 pre-registration
    • MR !13232026-11-06owner mergesops(debug): pin the Q2 canary to rocm-gfx1100-serving-v033-5d815c0d (0dc5cd0b)
    • MR !13222026-11-06owner mergesdocs(loom): owner-window log for B1, B2, C1, C2 (C2 reverted)
    • MR !13212026-11-06owner mergesrevert(models): back out the pyannote KT-DIAR lane (!1262); the gfx906 runtime has no pyannote backend
    • MR !12622026-11-06owner mergesops(models): pyannote diarization lane + provisional certificate for KT-DIAR [LIVE-IMPACTING, owner window G3]
    • MR !13192026-11-06owner mergesfix(plugin-common): segmented winners compiled against the live KV cache (Q2) + S5 stage A result (fail)
    • MR !13172026-11-06owner mergesops(scheduler): pin flexinfer-scheduler by digest in values-k3s
    • MR !12612026-11-06owner mergesops(gfx906): repin the Radeon VII runtime to the pyannote layer [LIVE-IMPACTING, owner window G2]
    • MR !12572026-11-06owner mergesops(models): give the gte CPU twin 12 threads and a 14-core limit [LIVE-IMPACTING, owner window]
    • MR !13162026-11-06owner mergesdocs(loom): owner-window execution log, phase A done
    Show the next 30 steps
    • MR !12552026-11-06owner mergesops(proxy): turn on the InferenceJob API and roll the proxy to master 51e6bd81 [LIVE-IMPACTING, owner window]
    • MR !12892026-11-06owner mergesops(controller): roll the k3s controller to master 51e6bd81 so vllmVersion takes effect (!1281, !1291) [LIVE-IMPACTING, owner window]
    • MR !13152026-11-06owner mergesdocs(loom): runbook D3 is outside S9's failure pattern (no CPU offload); add an offload log check
    • MR !13142026-11-06owner mergesdocs(loom): S9 result: the runtime base fails T2V, VACE and Stable Audio; expandable_segments with offload is the cause
    • MR !13132026-11-06owner mergesdocs(loom): gate runbook D3 (Qwen-Image in-pod) on a multi-render check after S9's finding
    • MR !13072026-11-06owner mergesdocs(loom): pre-register S9, the Wan decode A/B on the 0.32 base and two opt-in kill-tests
    • MR !13092026-11-05owner mergesdocs(loom): S7 embed host is conditional, G3 fp8 segmented is killed; record both verdicts
    • MR !13062026-11-05owner mergesci(runtime): publish the gfx1100-serving-v033 persona from CI
    • MR !13052026-11-05owner mergesdocs(loom): pre-register KT-QOS2, a smaller prefill budget behind the seat reserve
    • MR !13042026-11-05owner mergesdebug(models): S7 + G3 pool canary on the XTX (runtime@316f8188), not applied
    • MR !13022026-11-05owner mergesdocs(loom): KT-QOS fails bar 1 and KT-SO passes on vLLM 0.33; record both verdicts
    • MR !13012026-11-05owner mergesdocs(loom): record the owner's window answers Q1-Q7; XTX phase stops at KT-SWAP
    • MR !13002026-11-05owner mergesops(harbor): pin the control-plane digests for retention via deploy/pins
    • MR !12992026-11-05owner mergesdocs(loom): ordered runbook for the Drafts held for the 2026-11-06 owner window
    • MR !12982026-11-05owner mergesfeat(vllm-plugins): embeddings in mapped host memory (S7) and fp8 segmented KV on gfx11 (G3), opt-in
    • MR !12972026-11-05owner mergesdebt(vllm-plugins): delete the vLLM 0.23/0.26 compat code (B6)
    • MR !12962026-11-05owner mergesops(models): retire the vLLM 0.23 Qwen3.5-35B MoE lanes (B6 prep)
    • MR !12952026-11-05owner mergesdocs(loom): KT-OCTAVE fails on the pool; record the S6 verdict, G3 next
    • MR !12942026-11-05owner mergesdocs(loom): S4 step 0 decode-step anatomy: G1 go, Q1/G5 no-go, Q2/Q3 fail
    • MR !12932026-11-05owner mergesops(canary): KT-OCTAVE canary on the XTX with a pre-registered row (S6)
    • MR !12922026-11-05owner mergesdocs(loom): S8 gfx906 llama.cpp b11847 canary is conditional; record the evidence
    • MR !12912026-11-05owner mergesfix(backend): vllm-omni vllmVersion, CDNA3-only enableAiter, never certify LMFE
    • MR !12862026-11-05owner mergesops(gfx906): llama.cpp b11847 layer on the pyannote runtime, prewarm and canary (S8)
    • MR !12842026-11-05owner mergesfeat(runtime): vLLM 0.33.0-rocm72 serving profile; ISA check lists compressed offload bundles
    • MR !12832026-11-04owner mergesdocs(loom): XTX runtime on vLLM 0.32 (527e2714) passes; record the verdict
    • MR !12812026-11-04owner mergesfix(backend): render vLLM flags and env for the declared vLLM release
    • MR !12802026-11-04owner mergesdocs: refresh ROADMAP.md and IMPLEMENTATION_STATUS.md to 2026-11-04
    • MR !12762026-11-04owner mergesdocs(loom): Wan verified on the VAE bypass build; KT-FLEX waits only on the 24 h window
    • MR !12742026-11-04owner mergesfeat(runtime): prewarm and certify 527e2714, the v032 build with creative-lane deps, on the XTX
    • MR !12732026-11-03owner mergesci(diffusers-video): give the gfx1100 video publish a 2h timeout

    153 older off-route steps are in the climb’s JSON.

    Legs

    One chapter per day, narrated from the day’s merges, roadmap changes, metric readings and weather. Every sentence carries its evidence: a chip opens the merge, the roadmap line as written, the metric point or the weather band it rests on, and a sentence whose evidence could not be published is left out and counted. A day narrated again says “revised” and keeps its earlier versions; a day the model could not narrate within the rules is written from a fixed template and marked as one.

    1 chapter, newest first. Arrow keys move between chapters; Tab moves through a chapter’s citations.

    1. Leg for ascendingtemplate: written from the deterministic template, not by the model

      17 changes merged on 2026-11-06

      Moved
      Merges per week · Nov 6: Summit metric “Merges per week” on 2026-11-06: 100 merges. Jumps to the altitude chart.Proxy request success ratio (rolling 1d) · Nov 6: Summit metric “Proxy request success ratio (rolling 1d)” on 2026-11-06: 99.4%. Jumps to the altitude chart.
      Slipped
      Master pipeline success ratio (7d) · Nov 6: Summit metric “Master pipeline success ratio (7d)” on 2026-11-06: 64.5%. Jumps to the altitude chart.
      • Merges per week 81 merges → 100 merges +27 merges rising over 7 days, toward target
      • Master pipeline success ratio (7d) 66% → 64.5% −17.1 pts falling over 7 days, away from target
      • Proxy request success ratio (rolling 1d) 99.5% → 99.5% +0.8 pts rising over 7 days, toward target

      Co-moved: On 2026-11-06, 17 changes merged while Merges per week rose from 81 to 100 (toward its target), Master pipeline success ratio (7d) fell from 0.66 to 0.6452 (away from its target) and Proxy request success ratio (rolling 1d) rose from 0.9946 to 0.9954 (toward its target). This is co-movement on the same day only; the record does not show that any change moved a metric.

    Where this comes from

    Committed snapshotgenerated 2026-11-07 15:31 UTC ()

    This page is built from the committed snapshot of the operator document. Served from the snapshot because: operator not configured. Sanitized from the Mills chronicle read model: run, backlog and roadmap ids are hashed; titles, headlines and narrated sentences are scrubbed of paths, URLs, credentials, addresses and unpublished repository names; only climbs on published repositories are listed. Merge request numbers and short commit hashes are kept because they are public; so is the name of the model that narrated each leg, while its backend, prompts and traces are not. Merge request numbers are shown as text because the GitLab they live on needs a sign-in.

    flexinfer climb | Cody Blevins