yj"-dZddlmZddlZddlZddlmZddlmZddl m Z m Z m Z m Z mZmZdZdZd Zd Zd Zed GddZd+dZd,dZd-dZd-dZd.dZd/dZd-dZd0d Zdd!d1d*ZdS)2uMemory-pressure bounds for the gateway's per-session AIAgent cache. The gateway caches one ``AIAgent`` per session so a long-lived conversation reuses its prompt prefix instead of rebuilding the system prompt every turn. Each cached agent also pins ``_session_messages`` — the full live transcript, tool outputs included, which is tens of MB on a tool-heavy session. ``gateway/run.py`` bounds that cache two ways, and both are blind to how much memory it actually holds: * the LRU cap counts *entries*, not bytes, and 128 warm transcripts is several GB; * the idle TTL only sheds agents that went quiet for an hour, and it deliberately defers eviction for a finalizable session that has not expired yet, so a busy gateway hoards every transcript all day. This module supplies the missing signal: the process's own anonymous RSS, compared against a budget derived from the cgroup limit the gateway actually runs under. ``GatewayRunner._sweep_agent_cache_under_pressure`` uses it to shed LRU transcripts through the existing soft-eviction path, which rebuilds from the persisted session on the next turn (#80764). Everything here is pure or read-only so it can be tested without a gateway. Config lives under ``agent.agent_cache`` in ``config.yaml``. ) annotationsN) dataclass)Path)AnyCallableIterableListOptionalTupleg?iiT)frozencZeZdZUdZdZded<dZded<dZded<eZ ded <e Z ded <dS) AgentCacheBoundsa!Operator-facing bounds for the per-session agent cache. ``max_size`` and ``idle_ttl_secs`` are ``None`` when the operator did not set them, so ``gateway/run.py`` keeps using its module-level defaults. ``memory_high_mb`` is ``None`` when pressure eviction is switched off. N Optional[int]max_sizeOptional[float] idle_ttl_secsmemory_high_mbintmax_evictions_per_passprotect_recent) __name__ __module__ __qualname____doc__r__annotations__rr_DEFAULT_MAX_EVICTIONS_PER_PASSr_DEFAULT_PROTECT_RECENTrB/home/ragecks/.hermes/hermes-agent/gateway/agent_cache_pressure.pyrr6st#H""""%)M))))$(N(((("AAAAA1N111111r!rvaluerreturnrct|ts|dS t|}n#ttf$rYdSwxYw|dkr|ndSNr) isinstanceboolr TypeError ValueErrorr#parseds r" _positive_intr-Fsh%%-tU z "ttaZZ66T)+AArct|ts|dS t|}n#ttf$rYdSwxYw|dkr|ndSr&)r'r(floatr)r*r+s r"_positive_floatr1Psh%%-tu z "ttaZZ66T)r.ctjdkrdSg} ddlm}|}n#t$rd}YnwxYw|r%|dkr|d|dd|df|d |D]} t |d }n#t$rYEwxYw|r|d krR t|}n#t$rYowxYw|dks|d kr|cSdS)uReturn the memory limit this process runs under, if it is cgroup-capped. Prefers cgroup v2 ``memory.high`` (the throttling point — passing it is what stalled the reported shutdown) over ``memory.max``, and falls back to cgroup v1. ``max`` / absurd sentinel values mean "unlimited". Checks the process's *own* cgroup first (where a systemd unit's ``MemoryHigh=``/``MemoryMax=`` actually lands — the root files read ``max`` on those deployments), then walks up to the root for container-style limits. linuxNr)_own_cgroup_path/z/sys/fs/cgroupz /memory.highz /memory.max)z/sys/fs/cgroup/memory.highz/sys/fs/cgroup/memory.maxz+/sys/fs/cgroup/memory/memory.limit_in_byteszutf-8)encodingmaxl) sysplatformgateway.cgroup_cleanupr4 Exceptionextendr read_textstripOSErrorrr*) candidatesr4own candidaterawlimits r"_cgroup_limit_bytesrEZs |wtJ;;;;;;     sczz22221111        y//++W+==CCEECC    H  cUll  HHEE    H  A::7++  4s0' 66:5B00 B=<B= C C'&C'c< ttjdttjdzS#ttt f$rYnwxYw ddl}t|jS#t$rYdSwxYw)N SC_PAGE_SIZE SC_PHYS_PAGESr) rossysconfr?r*AttributeErrorpsutilvirtual_memorytotalr;)rLs r"_total_memory_bytesrOs 2:n--..RZ5P5P1Q1QQQ Z 0      6((**0111 tts$AAAA#)B BBsettingct|trB|}|dkr|dvrdnt |Sn)t|t r|sdSnt |St p t}|sdSt|tztz }|tkr|ndS)axResolve the ``memory_high_mb`` setting into an absolute MB budget. ``"auto"`` derives a budget from the cgroup limit the gateway runs under (or total RAM when uncapped), which is what makes this fix work out of the box on the containerised/systemd deployments where the leak bites. A positive number is taken literally; anything falsy disables the pass. auto)offnonefalsedisabledN) r'strr>lowerr-r(rErOr_AUTO_BUDGET_FRACTION _BYTES_PER_MB_AUTO_BUDGET_FLOOR_MB)rP normalizedrDbudgets r"resolve_memory_high_mbr_s'3 &]]__**,,   !III":..  GT " "& 4 W%%%  ! ! :%8%:%:E t ..> ? ?F44466$>r!configc d}t|tr?|d}t|tr|d}t|tsi}t|d}|d}t|}|2t|trt|t s|dkrd}t t|dt|dt|d d ||nt||nt S) a Read ``agent.agent_cache`` out of a raw config mapping. Reads the *raw* user config (the gateway's loader does not deep-merge ``DEFAULT_CONFIG``), so an absent key stays absent and the caller can tell "operator chose 128" from "operator said nothing". Nagent agent_cacherrrrrrrR)rrrrr) r'dictgetr-rr(rr1r_rr)r`section agent_cfg max_evictionsrprotect_parseds r"resolve_agent_cache_boundsrjsXG&$3JJw'' i & & 3mmM22G gt $ $!'++.F"G"GHHM[[!122N">22N ~s + + >400  a    w{{:6677%gkk/&B&BCC-gkk:JF.S.STT*6MM<[-8NN>U    r!c ddlm}|}|d}t|tr |dkr|dzS|d}t|tr |dkr|dzSn#t $rYnwxYw ddl}t |tj  j tz S#t $rYdSwxYw)uReturn the process's anonymous resident memory in MB, or None. Anonymous pages are the ones cached transcripts live in — the reported incident measured 11.0 GB of anon out of 11.0 GB total, so file-backed pages are noise here. ``collect_memory_snapshot`` already reads ``/proc/self/status`` without a dependency; psutil covers everything else, where only total RSS is available. r)collect_memory_snapshot rss_anon_kibirss_kibN) hermes_cli.mem_trimrlrer'rr;rLProcessrIgetpid memory_inforssr[)rlsnapshotanon_kibrnrLs r"read_anon_rss_mbrvs  ??????**,,<<// h $ $ $At# #,,y)) gs # # #! d? "       6>>")++..::<<@=PQQQ tts+AA=4A== B  B AC$$ C21C2rbr(ct|dd}t|tsdSt|dd}t|trt|trdS|t |kS)uxTrue when the agent's live transcript is fully on disk. Soft eviction drops ``_session_messages`` and rebuilds it from the persisted session next turn, so it is only safe once persistence has caught up. ``_last_flushed_db_idx`` is advanced to ``len(messages)`` by ``AIAgent._flush_messages_to_session_db`` and only on a fully successful write — the same divergence the FTS write-corruption guard reacts to when it preserves live history over a lagging transcript. Unknown shapes are treated as *not* caught up: a skipped eviction costs memory, a wrong one costs the user their conversation. _session_messagesNF_last_flushed_db_idx)getattrr'listrr(len)rbmessagesflusheds r" transcript_persistence_caught_uprsxu1488H h % %ue3T::G gs # #z'4'@'@u c(mm ##r!)rordered_entriesIterable[Tuple[str, Any]] is_evictableCallable[[str, Any], bool]rhrrList[Tuple[str, Any]]c2t|}|dks|sgStt|dt|dz}|r |d| }g}|D]=\}}t||krn$|||r|||f>|S)uChoose which cached sessions to shed, least-recently-used first. ``ordered_entries`` must be in LRU→MRU order (the cache is an ``OrderedDict`` kept in that order by ``move_to_end`` on every hit). The batch is capped so one pass cannot stall the gateway tearing down clients. ``protect_recent`` is an upper bound, clamped to half the cache: a handful of sessions can be big enough to exhaust the budget on their own (a single tool-heavy transcript runs to hundreds of MB), and a fixed guard would then protect the entire cache and leave the gateway climbing toward the OOM killer with nothing it is willing to shed. rN)r{minr7r|append) rrrhrentriesprotectplankeyrbs r"plan_pressure_evictionsrs&?##G #na((#g,,!*;<rs4#""""" !!!!!!AAAAAAAAAAAAAAAA"$  $ 2 2 2 2 2 2 2 2********2222j    ????:''''T<$$$$4        r!