Performance And Automation Updates – Shogun Exodus Devlog #32

Raw Patch Notes

- Redesigned website and updated it with more information.

UPDATE 2026-07-31 #156 - Automated X Devlog Posts and Website Alt Tags
- The devlog automation now posts to X after the WordPress publication and Discord posting attempt complete. Gemini 3.6 Flash with minimal thinking generates platform-specific copy from the WordPress summary and raw patch notes; the script then adds one unique random hashtag from each configured pool and the live WordPress link.
- X receives a normal text-and-link post immediately, with no image upload, delayed runner, queue, or native X scheduling.
- Files: `.github/scripts/post_devlog.py`, `.github/workflows/devlog-automation.yml`, `DeveloperFiles/update-log.txt`.
- Website Devlog script has been updated to include Alt-tags in the images uploaded.

UPDATE 2026-08-01 #157 - Stutter Diagnostics: Frame Gap Instrumentation
- Diagnostic-only change (no gameplay/perf fixes) for the recurring stutter investigation. Reports from 2026-08-01 showed ~45-52ms hitches in which `physics_total` + `process_total` accounted for only ~13% of the frame (e.g. `t=15.82s -> 46.0ms [physics_total=4.2 process_total=1.9]`), so the cause sits in code neither bracket can reach. Per the AGENTS.md rule ("if a hitch's labels do not sum to its frame time, add a label before guessing"), the blind spot is now measured rather than theorised.
- `FrameProbe` gains a third DISJOINT bucket `frame_gap`: the span from the end of the last `_process` to the first probe point of the next frame. This is render submission, texture uploads, the `call_deferred`/MessageQueue flush, and queued node deletion — everything the physics and process brackets structurally cannot see. It is sampled at the start of frame N+1, which is where Godot's frame N+1 `delta` accounts for it, so `physics_total + process_total + frame_gap` now sums against a single delta.
- `FrameProbe` also gains `render_submit` (`RenderingServer.frame_pre_draw` -> `frame_post_draw`), connected only if the build exposes those signals. It is a SUBSET of `frame_gap`, never an extra bucket, and exists to split rendering out of the gap. It must never be added to the accounted total or rendering is double-counted.
- `TickProfiler` big-hitch lines now print `acct` (the three disjoint buckets), `unacct` (frame time minus acct, deliberately unclamped so a negative reveals bracket overlap), plus per-hitch node count and video memory. Node count per hitch fingerprints spawn/free churn versus rendering.
- `MAX_BIG_FRAMES` raised 40 -> 200. The 40 cap was silently truncating real 30s windows — one report's hitch list ended at t=11s of 30s — so previous conclusions about rhythm were drawn from a censored sample. The report now prints an explicit TRUNCATED warning when the cap is hit.
- New "hitch periodicity" report section: interval mean/median/min/max between consecutive hitches, a coefficient of variation with a REGULAR/SEMI-REGULAR/SCATTERED verdict, and a bucketed histogram so a clean beat buried under scattered noise stays visible. Reported separately for all hitches >=20ms and severe hitches >=40ms, because a player typically only perceives the large ones. This exists to settle a direct conflict between the reported symptom (a regular ~2s beat) and the logs (1-3 irregular hitches per second).
- `Performance.get_monitor()` polling moved off the every-frame path (three driver queries per frame, ~7800/sec at the 2600fps seen in menu windows) to hitch frames plus a periodic refresh, so the profiler perturbs its own measurement less.
- Files: `Autoload/FrameProbe.gd`, `Autoload/TickProfiler.gd`, `DeveloperFiles/update-log.txt`.
- Follow-up (#157, frame attribution correctness): the first instrumented run exposed a long-standing bug in the profiler itself, not in the game. Godot's `_process(delta)` for iteration N reports the duration of iteration N-1, but the accumulator read at that moment holds iteration N's samples — so every big-hitch line paired a frame's TOTAL with the NEXT frame's LABELS. A 39ms `pawn_physics` spike therefore printed as an innocent frame with `unacct` ~36ms, and when two slow iterations landed back to back the same spike printed twice 0.01s apart. This silently misdirected the stutter investigation toward "uninstrumented code" that did not exist.
- Fixed by moving frame ownership into `FrameProbe`: it now measures the wall-clock duration of each main-loop iteration across its own probe points and calls the new `TickProfiler.end_frame(total_usec)` at the START of the following iteration — the earliest moment a frame's trailing `frame_gap` is knowable. The accumulator is flushed there, so a frame's total and its labels always describe the same frame. `TickProfiler._process` no longer samples frames; it only owns the 30s report cadence. `FrameProbe` also drops the `Engine.get_process_frames()` dependency (substep detection now uses its own start marker) and calls the new `TickProfiler.discard_frame()` when a frame is interrupted by a pause, instead of letting a partial frame merge into the next one.
- Fixed saturated spike counters: `record_channel_tick` / `record_named_tick` retain only the most recent `MAX_SPIKES_STORED` samples, but the report printed that array's SIZE as the spike count — so every "spikes>5ms: 20" actually meant "20 or more, true count unknown", across both the channel and per-listener sections. Records now carry a separate `spike_count` that is never capped; the report prints the true count and notes when it is only listing the most recent N.
- `unacct` is now documented in-report as an INSTRUMENTATION SELF-CHECK rather than a finding: near zero means the decomposition is complete and the biggest bucket names the cause; persistently large means a span is unmeasured and a probe should be added rather than a cause guessed.
- Files: `Autoload/FrameProbe.gd`, `Autoload/TickProfiler.gd`, `DeveloperFiles/update-log.txt`.
- Follow-up (#157, pawn_physics decomposition): `pawn_physics` was a single bracket around the entire per-pawn step, with only `ai_decide` / `nav_refresh` / `collision_refresh` measured inside it — together roughly a third of the average, leaving the rest unattributed. Added DISJOINT sub-brackets covering the remainder: `pawn_slide` (instrumented inside `_apply_gravity_and_slide`, the single home of `move_and_slide()` that all seven call sites route through — also the only per-step call into the physics server, so a dirty broadphase or heavy collision resolve surfaces there), `pawn_carry_visual` (`_refresh_carried_item_visual_if_needed`, which rebuilds a state string via ItemVisuals every step while carrying), `pawn_needs_state` (`_tick_rest_states` + `_update_need_activity_state_flags`), and `pawn_free_time` (`_update_free_time_buff`, which touches status effects every step).
- Sub-brackets now report through `_profiler_sub_sample()`, which accumulates into `_pawn_sub_usec` so `_physics_process` can emit `pawn_other` = the part of the step no bracket covered (the move-state pathing logic in `_process_move_to_point` / `_process_move_to_stairs` / `_process_climb_stairs`, minus their nested slide). This mirrors the frame-level design: the decomposition is exhaustive, so a spike always lands in a named bucket instead of a silent remainder. Verified no sub-bracket nests inside another (all `_apply_gravity_and_slide` call sites live in `_physics_process_impl` or the move-state processors, never inside the AI/needs/carry/free-time brackets), which is what keeps `pawn_other` non-negative.
- Files: `Scripts/BasicWanderAI.gd`, `DeveloperFiles/update-log.txt`.

UPDATE 2026-08-01 #158 - Room Lookup Optimization (Wander Destination Stutter Fix)
- Fixes the recurring 40-115ms stutter at 25 pawns in a 47-room castle (FPS stayed high at ~325; the complaint was periodic hitches). Diagnosed via the #157 instrumentation, which localized it precisely: `pawn_other` spiked >5ms exactly 41 times against 41 frames >16ms, reproducing across both windows, with `pawn_slide` max 0.14ms and `frame_gap` max 2.56ms exonerating movement, rendering and the deferred flush. Hitch magnitudes were quantised at ~35.5ms (observed 35.4 / 35.4 / 35.8 / 70.6 = one, one, one, then two pawns doing the same operation on the same physics step). Periodicity CV 1.12 = SCATTERED, so the reported "every 2 seconds" was the subset of spikes that happened to cluster, not a tick channel. Full analysis and scope in `DeveloperFiles/room-lookup-optimization-brief.txt`.
- Root cause: `BasicWanderAI._pick_new_destination()` evaluated EVERY candidate walk point in the castle (47 rooms x 3 points = 141 candidates) and then picked one at random. Each candidate made 3 calls that reached `WorldIndex.get_room_at_position()`, and `WorldBuildingRegistry._as_room_array()` `duplicate(true)`d ALL 47 room dictionaries on every one of them — roughly 20,000 deep dictionary copies so a single pawn could choose a single random destination. Cost is linear in room count, which is why the stutter reappeared when the castle roughly doubled (~24 -> 47 rooms) with no new feature to blame, and why an earlier 100-pawn test passed on the smaller castle.
- Fix 1 (behaviour-neutral, benefits every consumer): removed the redundant deep copying from the registry room lookups. `_find_room_at_position_in_rooms` / `_find_actor_room_in_rooms` now take the provider's LIVE room array and iterate it read-only; they already returned `duplicate(true)` of the winning room, so that remains the single isolation boundary and every public getter still hands out an isolated copy. Dropped the redundant SECOND duplicate in `get_room_at_position` / `get_actor_room` and the THIRD in `BasicWanderAI._get_navigation_room_at_point`. `_as_room_array` had exactly three call sites and is now deleted. `is_actor_in_room` was deliberately left as a linear scan (not switched to `rooms_by_id`): with duplicate room ids the scan returns true when ANY match overlaps, whereas the dictionary retains only the last one written, so the scan is the semantics-preserving choice. Beneficiaries with no code change: BasicWanderAI, JobManager, JobsHUD, ConstructionScaffold, PlaceableLightComponent, PlaceablePowerGeneratorComponent, RoomClippedLight, BuildingPortraitHUD, InitialSpawnTool, LightingSystem, SmokeSystem.
- Fix 3 (behaviour-neutral): `_pick_new_destination` resolved the same adjusted point twice (once inside `WorldIndex.can_actor_enter_point`, once inside `_point_has_actor_clearance`). Added `_can_stand_at_navigation_point_with_room()`, which takes the pre-resolved room, cutting 3 lookups per candidate to 2. The clearance test was extracted into a shared `_point_has_clearance_in_room()` used by BOTH the fast path and the original `_point_has_actor_clearance`, so the two cannot drift about what counts as standable. The room-lock contract is preserved exactly: a point outside every room is rejected, and a locked room stays enterable only by the builder that locked it. Deliberately NOT collapsed further to a single lookup: clamping x can move a point into the x-span of a different overlapping room on another floor with a closer `walk_y`, so the raw and adjusted lookups are not provably identical.
- Gameplay unchanged by design: the candidate set is identical, so wander destination distribution is unchanged; GOAP, jobs, eating, sleeping, socialising and corpse hauling were never involved (they route through `_request_move_to_world()`, which validates a single point, and are untouched); room locking, save/load, mod merge, node paths and data schemas are untouched. `_pick_new_destination` has only two callers — idle-timer expiry and the spawn-time `_refresh_nav_snapshot(true)` in `_ready()`.
- Known accepted side effect: the wander fast path bypasses `WorldIndex.can_actor_enter_point()`, so its two `_debug_world` rejection lines no longer fire for wander candidate rejection. Goal-directed movement still uses the original path and still logs.
- Diagnostics: added a `pawn_pick_dest` sub-bracket around `_pick_new_destination` (routed through `_profiler_sub_sample`, so it is carved OUT of `pawn_other` rather than double counted). Destination picking is occasional but heavy and linear in room count, so a spike there was previously indistinguishable from ordinary movement logic. Verified disjoint: no other sub-bracket runs inside that call.
- NOT changed (deliberately deferred): the `LightingSystem` night-time debuff cost (~99ms per lighting tick, to be handled after this round); `pawn_free_time` (avg 1.91ms every step, 34% of average `pawn_physics` — `_update_free_time_buff` calls apply/remove_status_effect unconditionally per pawn per step, a constant tax that never spikes); candidate-room sampling in `_pick_new_destination` (would alter wander distribution and idle-failure behaviour, expected unnecessary now); physics tick rate.
- Files: `Scripts/World/WorldBuildingRegistry.gd`, `Scripts/BasicWanderAI.gd`, `AGENTS.md`, `DeveloperFiles/current-game-setup.txt`, `DeveloperFiles/room-lookup-optimization-brief.txt` (new), `DeveloperFiles/update-log.txt`.

*This is an automated post, using Python and LLMs to summarize game updates. Mistakes, while rare, may happen.