yjxCUdZddlmZddlZddlZddlZddlZddlZddlZddl m Z ddl m Z m Z mZmZiZded<dD]ZeeedZeeeee< d)d Zd*dZd+dZd,dZd-d.dZddd/d Zd0d"Zd0d#Zd1d&Zd2d(ZdS)3uShutdown forensics — capture context when the gateway receives SIGTERM/SIGINT. The gateway's ``shutdown_signal_handler`` runs synchronously inside the asyncio event loop. We can't safely block it for long, but we DO want a durable record of who/what triggered the shutdown so that "the gateway keeps dying" incidents can be diagnosed after the fact. This module exposes :func:`snapshot_shutdown_context`, a fast (<10ms), non-blocking probe that returns a structured dict the signal handler can log immediately, plus :func:`spawn_async_diagnostic`, a fire-and-forget ``ps`` walk that runs as a detached subprocess so it can't block teardown even if /proc is wedged. Anything that needs to wait (e.g. shelling out to ``ps aux``) belongs in the async helper, never in the synchronous probe. ) annotationsN)Path)AnyDictListOptionalzDict[int, str]_SIGNAL_NAME_BY_NUM)SIGTERMSIGINTSIGHUPSIGQUITSIGUSR1SIGUSR2sigrreturnstrc|dS t|}n&#ttf$rt|cYSwxYwt|d|S)zBReturn a human-readable signal name (or ``str(sig)`` as fallback).NUNKNOWNzsignal#)int TypeError ValueErrorrr get)rsig_ints @/home/ragecks/.hermes/hermes-agent/gateway/shutdown_forensics.py _signal_namer%sl {yc(( z "3xx  " "7,?g,?,? @ @@s  99pidrkey Optional[str]cR td|dd5}|D]V}||dzr<|dddccdddSW dddn #1swxYwYn#tt t f$rYnwxYwdS)zIRead a single field from /proc//status. Linux only; None elsewhere./proc/z/statusutf-8encoding:N)open startswithsplitstripFileNotFoundErrorPermissionErrorOSError)rrfhlines r_read_proc_fieldr/0s( '3'''' : : : 9b 9 9??39--9::c1--a0668888 9 9 9 9 9 9 9 99 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9  8     4sAB A A># B 0A>2 B >BB BB B$#B$cD td|dd5}|}dddn #1swxYwYn#tttf$rYdSwxYw|sdS|dddd S) zLRead /proc//cmdline as a printable string. Linux only; None elsewhere.r z/cmdlinerbN r!replace)errors)r&readr*r+r,r4decoder))rr-datas r_read_proc_cmdliner9<s (3((($ / / 27799D                 8tt t << & & - -gi - H H N N P PPs,A7 A;A;AAADict[str, Any]cd|i}|dkr|St|d}|||d<t|d}|||d<t|d}|$ t||d <n#t$rYnwxYwt|d }|!|r|dn||d <t |}|r |dd |d <|S)zCompact /proc/ snapshot: pid, ppid, state, uid, cmdline. Best-effort. Missing fields are simply omitted rather than raising. rrNameNnameStatestatePPidppidUiduid,cmdline)r/rrr(r9)rsummaryr=r?rArCrEs r _proc_summaryrGIs %clG axx C ( (D  S' * *E   C ( (D  !$iiGFOO    D  3 & &C +.7QC %%G+$TcT]  NsA!! A.-A.received_signalc ftj}tj}tj}tj}||t ||t |nd||t|t|d}tj d}|r||d<tj d}|r||d<t|p|dk|d< tj d |d <n#ttf$rYnwxYw t|d }|c|d kr]|rt |n||d <|rtt |nd|d<n#t t"f$rYnwxYw tj d} | rt%| dz } | rF | d} | dd|d<d|| vpd|| v|d<n#t$rYnwxYwt%| dz } | r5 | d} | dd|d<n#t$rYnwxYwn#t*$rYnwxYw|S)a'Fast (<10ms) snapshot of who/what is asking us to shut down. Captures: * The signal number/name (so SIGINT vs SIGTERM is visible) * Our own PID/ppid + parent process info from /proc (Linux) * Whether systemd is our parent (``ppid==1`` or ``INVOCATION_ID`` set) * Whether takeover/planned-stop markers exist (consumed lazily by the caller) * /proc/self limits + load average (1-min) * Wall-clock and monotonic timestamps for cross-correlating later phases Pure stdlib, never raises, never blocks on subprocesses. N)ts ts_monotonicsignal signal_numrrAparentself INVOCATION_IDsystemd_invocation_idJOURNAL_STREAMsystemd_journal_streamr% under_systemdr loadavg_1m TracerPid0 tracer_pidtracer HERMES_HOMEz.gateway-takeover.jsonr!r"rDtakeover_markerz"target_pid": z'target_pid': takeover_marker_for_selfz.gateway-planned-stop.jsonplanned_stop_marker)time monotonicosgetpidgetppidrrrGenvironrbool getloadavgr,AttributeErrorr/isdigitrrrexists read_text Exception) rHnowr_rrActx invocation_idjournal_streamrYhermes_home_str takeover_pathrawplanned_stop_paths rsnapshot_shutdown_contextrshs0 )++C  I )++C :<2s to walk /proc, during which the asyncio loop is frozen and adapter teardown can't begin. T)parentsexist_okNwin32z echo '=== shutdown diagnostic @ z ==='; echo '--- date ---'; date -u +%Y-%m-%dT%H:%M:%SZ; echo '--- ps auxf (top 60 by cpu) ---'; ps auxf --sort=-pcpu 2>/dev/null | head -60; echo '--- pstree of self ---'; pstree -plau a 2>/dev/null | head -40 || true; echo '--- /proc/loadavg ---'; cat /proc/loadavg 2>/dev/null || true; echo '--- recent dmesg (oom/killed) ---'; dmesg -T 2>/dev/null | tail -20 || journalctl --user -n 20 --no-pager 2>/dev/null | tail -20 || true; echo '=== end ==='itimeoutz.0fbashz-c)stdoutstderrstdinstart_new_session close_fds)rNmkdirr,sysplatformr`rar&rO_WRONLYO_CREATO_APPEND subprocessPopenSTDOUTDEVNULLr*closer)rurvrtscriptfdprocs rspawn_async_diagnosticrs3*dT:::: tt  |wt ;       WS]]BK"*$ >G$}}}CHF ww !!-77566 H'Fvvw H H    ww$%%1 7888 ww|9 7C $577888-3;#((((J )# ) )& ) )  ) )# ) )  )   ) ) %  ) )cl tj|tdS#ttf$rYdSwxYw)zFJSON-serialise a context dict for structured ingestion. Never raises.T)default sort_keysz{})jsondumpsrrr)rls rcontext_as_jsonr:sEz#sd;;;; z "tts 33 drain_timeoutOptional[Dict[str, Any]]c tjd}|sdSd} tdd5}|D]\}d|vrV|d}t |D]}|dr|}n|rn]dddn #1swxYwYn#ttf$rYnwxYw|sdSd}dggfD]} tj d g|d |d d d dd d} n"#ttj tf$rYEwxYw| j dkrU| jD]|}|dre|ddd} | rt%| }nt'| }|n}|n|dS|dz } d} || z} || || | | kdS)u.At startup, sanity-check that systemd's TimeoutStopSec >= drain_timeout. When the gateway is run under a stale systemd unit file (e.g. the user upgraded hermes-agent but never re-ran ``hermes setup`` to regenerate the unit), ``TimeoutStopSec`` can be smaller than the configured ``restart_drain_timeout``. Result: SIGTERM arrives, the drain starts, and systemd SIGKILLs the cgroup mid-drain — looks like a phantom kill in the journal because the journal only logs ``code=killed status=9``. Returns ``None`` when the alignment is fine OR we can't determine it (not running under systemd, ``systemctl`` unavailable, etc.). Returns a dict with ``timeout_stop_sec`` + ``drain_timeout`` + ``mismatch`` bool when we have data to report. Best-effort. Never raises. rPNz/proc/self/cgroupr!r"z.service/z--user systemctlshowz--property=TimeoutStopUSecTr4g@)capture_outputtextr#r5r}rzTimeoutStopUSec==r%g.Ag>@)unittimeout_stop_secr expected_minmismatch)r`rcrr&r)r(reversedendswithr,r*rrunTimeoutExpired returncoder splitlinesr'rgr_parse_systemd_duration_to_us)rrm unit_namer-r.partsp timeout_usflagresultvaluerheadroomexpecteds rcheck_systemd_timing_alignmentrBs"JNN?33M t $I % 8 8 8 B  %% JJLL..s33E%e__""::j11"()I!E"!                & '     t !%JR  ^UtUVUYU8TU#$\_FF":#u!E!EJ)E  ! E "t!K/Hx'H,& $x/   sGB0A B$ B0$B((B0+B(,B00CC#C99DDrqc|sdSdddddddd}d}d }d }|d zD]O}|s|d krp|rh||}||sdS |tt ||zz }n#t $rYdSwxYwd }d }||z }|r||z }|rj|rh||}|dS |tt ||zz }n#t $rYdSwxYwd }d }|r:|s8 |tt |dzz }n#t $rYdSwxYwd }Q|dkr|ndS) zParse 'TimeoutStopUSec=1min 30s' / '90s' style values to microseconds. systemd accepts a wide grammar; we cover the common cases (s, ms, min, h) and return None on anything unexpected. Never raises. Nr%ii@Bil$'-)usmsssecminhhrrrr.)rgrlowerrrwrisalpha)rqunitstotal_ustokendigitsch multipliers rrrs t    EH E FCi   ::<< 299 "YYu{{}}55 %V%44 E&MMJ$> ? ??HH!   444  bLFF ZZ\\  RKEE   5;;==11J!tt Cf :;;;   ttt FEE  E  Cf 9:::   ttt F!||88-s6("B  BB2"D D$#D$2"E E$#E$)rrrr)rrrrrr)rrrr)rrrr:)N)rHrrr:)rurrvrrtrwrrx)rlr:rr)rrwrr)rqrrrx) __doc__ __future__rrr`rLrrr^pathlibrtypingrrrrr __annotations___namegetattr_valrrr/r9rGrsrrrrrrrrs"#"""""  ,,,,,,,,,,,,')(((( M//E 765$ ' 'D ).CCII&AAAA     Q Q Q Q>ZZZZZB! QQQQQQhBTTTTn5.5.5.5.5.5.r