yj+~UdZddlZddlZddlmZddlmZddlZddlZddl Z ddl Z ddl Z ddl Z ddl Z ddlZddlZ ddlZn #e$rdZYnwxYw ddlZn #e$rdZYnwxYwddlmZmZddlmZddlmZddlmZmZmZmZmZmZm Z m!Z!ej"e#Z$dd l%m&Z'dd l(m)Z)m*Z*da+da,ee-e.d <d e-fd Z/e0Z1e1dz Z2e2dz Z3e2dz Z4e2dz Z5dZ6e j7Z8e j9Z:dZ;e2dz ZeddZ?eee>e.d<e>e2e3e<Z@d e>fdZAejBde eCeffdZDd efd ZEd!ZFd"ZGd#ZHd eIfd$ZJd%eCd e-fd&ZKd efd'ZLejBd(ZMeNd)hZOd%eCd efd*ZPdd+eeCd,eed eeCfd-ZQd.eeCefd eeCeffd/ZRdd1ed2eCd eCfd3ZSd.eeCefd eCfd4ZTd.eeCefd eeCeffd5ZUd.eeCefd e-fd6ZVd.eeCefd e-fd7ZWd.eeCefd eCfd8ZXd9efd:ZYd9efd;ZZd9edZ]d?eCd e^fd@Z_dAeCd eeCeffdBZ`dCed efdDZadEedFed e-fdGZbdEedFed e-fdHZcddIdAeeCefdJedKeeCd eeCfdLZddAeed e^fdMZfddAeeCefdKeeCd eeCfdNZgd9ed dfdOZhd9ed1e^d dfdPZiddRe-d dfdSZjd9ed eeIfdTZkd eeIfdUZld eeIfdVZmddWZndXeCd dfdYZod e^fdZZpdd[Zqd eeCfd\Zrd]ed eee-ffd^Zsd eeeCeffd_Ztd eeeeCeffd`Zud]ed eee^e^e^ffdaZvdbeee^e^e^fd dfdcZwddddeeeeCefdfee!eCd eeeCeffdgZxddQdhdeeeeCefdfee!eCdie-fdjZyddQdhdeeeeCefdfee!eCdie-fdkZzdleeCd eeCfdmZ{d eeCfdnZ|dQdod1edpe-d eeCfdqZ}dredsedtedued eeeCeeCff dvZ~d.eeCefd eeeCeeCeeCe-ffdwZdxeeCdyeeCdue-dzeeCd df d{Z dd|eeCdAeCd}eeCd~ee^deeCdeeeCefd+eeCd,eeeCdseeCdreeCdteeCdzeeCdee eCeeCfdeeeCdleeCdue-dee-dxeeCdyeeCd eeCeff(dZd%eCd eeeCeffdZGddeZdeCd eeeCeffdZdde-d eeeCeffdZd%eCdeeCefd eeeCeffdZdd%eCdeeCd eeeCeffdZd%eCd eeeCeffdZd%eCd eeeCeffdZd%eCd e-fdZd%eCd1e-d e-fdZd%eCd e-fdZd%eCd dfdZ dd%eCdRe-deeCdeeCdeeCf dZd.eeCefd dfdZd%eCd e-fdZd%eCdeCd e-fdZd e^fdZd%eCd e-fdZd eCfdZddd%eCde^d e-fdZdZd eIfdZddddeeeCefdJedfeeeCd e-fdZd eeeCeffdZd eeeCeffdZdZd e^fdZdede^d e^fdZd%eCdeCfdZded eCfdZd eeCfdZ ddeeeCeCfdeeeCd eeCeffdZdS)z Cron job storage and management. Jobs are stored in ~/.hermes/cron/jobs.json Output is saved to ~/.hermes/cron/output/{job_id}/{timestamp}.md N) ContextVar) dataclass)datetime timedelta)Path)get_hermes_home)OptionalDictListAnySetTupleUnion Collection)now)atomic_replaceatomic_write_text HAS_CRONITERreturnctt ddlm}|adan#t$rdaYnwxYwttS)zDImport croniter on first use; honor a pre-set HAS_CRONITER override.Nr)croniterTF)rr ImportErrorbool) _croniters //home/ragecks/.hermes/hermes-agent/cron/jobs.py_ensure_croniterr4s_ ! 6 6 6 6 6 6 HLL ! ! ! LLL !   s  ##cron jobs.jsonticker_heartbeatticker_last_success<g>@outputxT)frozenc.eZdZUeed<eed<eed<dS)_CronStorePathscron_dir jobs_file output_dirN)__name__ __module__ __qualname__r__annotations__rr&r&ws0NNNOOOr/r&cron_store_override)default_cron_store_overridec(t}||Stttt }|t kr|St}|tkr|S|dz }t||dz |dz S)uReturn paths pinned to this execution context's profile. Precedence, most explicit first: 1. an active use_cron_store() override (ContextVar); 2. deliberately re-pointed module constants — if CRON_DIR/JOBS_FILE/ OUTPUT_DIR no longer match their import-time values, someone chose the documented process-wide compatibility surface; honor it; 3. the ACTIVE profile home, resolved fresh via get_hermes_home() (context-local override, then the HERMES_HOME env var) — so a test or embedder that re-points HERMES_HOME after this module was imported reads/writes ITS OWN store, not whatever jobs.json the import happened to freeze (the filed incident: fixtures that patched the env too late silently rewrote the user's real jobs file); 4. the import-time constants (home unchanged since import — the common path, returned unchanged). Nrrr") r2getr&CRON_DIR JOBS_FILE OUTPUT_DIR _IMPORT_STORErresolve HERMES_DIR)overridelive_constantshomer's r_current_cron_storer>s$$''))H$Xy*EEN&&    $ $ & &D zf}H 8X %;X=P Q QQr/r=c#RKt|dz }tt ||dz |dz } dVt|dS#t|wxYw)z@Route cron storage to ``home`` without mutating process globals.rrr")r'r(r)N)r expanduserr9r2setr&reset)r=r'tokens ruse_cron_storerDsDzz$$&&..0069H $ $,(*     E* ""5)))))""5))))s *B B&c(tjS)z>Return the output directory for the active cron store context.)r>r)r.r/rget_cron_output_dirrFs  ++r/ig@cHtjdd}t}|r/ t |}n#t t f$r t}YnwxYw|dkrt tSt|tzt tS)ugResolve the one-shot running-claim stale-recovery TTL. Derived from ``HERMES_CRON_TIMEOUT`` (the cron inactivity timeout the scheduler enforces on each run) so the safety valve tracks how long a run is actually allowed to go quiet, instead of a magic constant: - unset / invalid → default 600s inactivity limit → TTL = 1800s - ``0`` (unlimited runs) → no finite bound to derive from → fall back to ``ONESHOT_RUN_CLAIM_TTL_SECONDS`` - positive N → ``max(N * headroom, ONESHOT_RUN_CLAIM_TTL_SECONDS)`` so a tiny configured timeout can never expire a claim mid-run. HERMES_CRON_TIMEOUTr) osgetenvstrip _DEFAULT_CRON_INACTIVITY_TIMEOUTfloat ValueError TypeErrorONESHOT_RUN_CLAIM_TTL_SECONDSmax_ONESHOT_RUN_CLAIM_TTL_HEADROOM)rawtimeouts r_oneshot_run_claim_ttl_secondsrWs ))2 . . 4 4 6 6C.G 7 7CjjGGI& 7 7 76GGG 7!||2333 11 +,,  sAAAjob_idc ddlm}||vS#t$r!td|dYdSwxYw)u|Return True when the scheduler in THIS process is still running ``job_id``. Direct liveness signal for stale-entry recovery (#62002): the run_claim TTL alone cannot distinguish "the claiming tick died" from "the run is alive but slow" — a run stalled on network I/O (or a laptop that slept mid-run) legitimately outlives the TTL. The in-process ticker and the run share this process, so the scheduler's running set settles the common single-gateway case without any claim-age guesswork. Imported lazily: the scheduler imports this module at load, so a module-level import here would be circular. r)get_running_job_idszsCron running-set liveness check failed for job %r; keeping the entry to avoid deleting a possibly live one-shot runTexc_info)cron.schedulerrZ Exceptionloggerwarning)rXrZs r_job_running_in_this_processrasy 666666,,....  C     tts '??c.tjdz S)z=Return the advisory lock path for the current cron directory.z .jobs.lock)r>r'r.r/r_jobs_lock_filerc s  )L 88r/c #Kttdd}|rF|dzt_ dVtxjdzc_n#txjdzc_wxYwdSt5dt_dt_d} t t tdd}|dttj tz} tj |tjtjzn#t t"f$rtj |krWt$d tt |n#t $rYnwxYwd}Yntjd YnwxYwnLt,Ett,d |tt,d dn9#t t"f$r%}t$d |Yd}~nd}~wwxYw dV| t tj |tjnLt,Ett,d |tt,ddn#t t"f$rYnwxYw|n#|wxYwn#| t tj |tjnLt,Ett,d |tt,ddn#t t"f$rYnwxYw|w#|wxYwwxYwdt_dt_n#dt_dt_wxYw ddddS#1swxYwYdS)uSerialize a load_jobs→modify→save_jobs critical section. Combines the in-process threading lock (cheap mutual exclusion between the gateway's parallel tick threads) with a cross-process advisory file lock on ``/.jobs.lock`` (mutual exclusion between the gateway process and standalone ``hermes`` CLI invocations, which previously shared no lock at all — a `cron pause` could be silently clobbered by a concurrent gateway write, leaving a "paused" job still firing). The flock is blocking, but every critical section that uses it is short (field updates only — no agent execution), so contention resolves in milliseconds. If neither fcntl nor msvcrt is available the manager still provides in-process locking, matching the historical behaviour. Nested calls in the same thread reuse the held lock so legacy callers that invoke save_jobs() inside a broader mutation section don't deadlock or try to reacquire the advisory file lock. depthrNza+utf-8encodingTuTimed out after %.0fs waiting for the cron jobs lock (%s) — another process is holding it. Proceeding with in-process locking only so the scheduler stays alive (#60703).g?lockingLK_LOCKzSjobs.json cross-process lock unavailable (%s); proceeding with in-process lock onlyLK_UNLCK)getattr_jobs_lock_statere_jobs_file_lock load_stamp ensure_dirsopenrcseekfcntltime monotonic_JOBS_LOCK_TIMEOUT_SECONDSflockLOCK_EXLOCK_NBOSErrorIOErrorr_errorclosesleepmsvcrtfilenor`LOCK_UN)relock_fd _deadlinees r _jobs_lockrs( $gq 1 1E !& ( EEE  " "a ' " " "  " "a ' " " " " " " I/I/!" '+#@ // J 00$III Q$!% 0 03M MI,,!K1NOOO! '1,,,#~//9<< & %M%?$3$5$5 !"!"!"!)$+MMOOOO'.!)!)!)$(D!)*. % JsOOOOO!, ,*'.GFI..w~~/?/?QZA[A[]^___W% J J J FGHJJJJJJJJ J (&( ,!K????#/6GFI66w~~7G7GQWYcIdIdfghhh#W-  '7&( ,!K????#/6GFI66w~~7G7GQWYcIdIdfghhh#W-  '&'  "*.  ' '&'  "*.  ' . . . . 'SI/I/I/I/I/I/I/I/I/I/I/I/I/I/I/I/I/I/s6AA(O?A$G?*,DG?AF--FF- F F-FF-G?F-*G?,F--AG?>O?H5H0+O0H55O9K:=OA3J43K 4KK KK  O K66O:N8>A3M21N2N NN N N8N4 4N88O;O?O..O??PPidcBt|pd}|r |dvsd|vsd|vrtd|t|st|jrtd|t j|z S)adResolve a job's output directory, rejecting any path-escape attempt. Job IDs are filesystem path components under ``OUTPUT_DIR``. A legacy or crafted ID containing ``..``, absolute paths, or nested separators would allow output writes/deletes to escape the cron output sandbox. Reject anything that isn't a single safe path component. rJ>.../\z%Invalid cron job id for output path: )strrMrPr is_absolutedriver>r))rXtexts r_job_output_dirr}s v|   " " $ $D M4;&&#++KKKLLL DzzM4::#3MKKKLLL  +d 22r/skillskillsc||r|gng}n(t|tr|g}nt|}g}|D]@}t|pd}|r||vr||A|S)zPNormalize legacy/single-skill and multi-skill inputs into a unique ordered list.NrJ) isinstancerlistrMappend)rr raw_items normalizeditemrs r_normalize_skill_listrs ~$,UGG" FC !H LL J$$4:2$$&&  $D **   d # # # r/jobct|}t|d|d}||d<|r|dnd|d<|S)zLReturn a job dict with canonical `skills` and legacy `skill` fields aligned.rrrN)dictrr4)rrrs r_apply_skill_fieldsrs[cJ ":>>'#:#:JNN8>D      8}} 3r/c~t|}t|dd}t|d}||d<||d<t|d}|st|d}|p*|dr|ddndp|p|pd }|d d pd }||d<t ||d <t ||d <|S)a,Return a read-safe cron job shape for UI/API/tool/scheduler consumers. Older or hand-edited jobs can have nullable fields like ``prompt``, ``name``, or ``schedule_display``. Keep storage untouched on read, but ensure consumers never crash while formatting or running those records. runknownpromptnamescriptrrrJcron jobN2rstate)rrr4rMreffective_job_state)rrrXrrr label_sources r_normalize_job_recordrsO%S))J jnnT22I > >F jnnX66 7 7FJt!Jx JNN622 3 3 9 9 ; ;D  7!*..":":;;AACC  +5>>(+C+CK 8$Q''    CRC &&((6JJv%>z%J%JJ!" .j99Jw r/ct|ddkrdSt|dS)z>True when the record carries any operator-facing pause signal.rpausedT paused_at)rr4rMrrs r_has_pause_markerrsL(())//11X==t  $$ % %%r/cX|ddsdSt|rdSdS)zTrue iff the scheduler may fire this job. ``enabled`` is the scheduler-honoured flag. Pause markers (``state`` / ``paused_at``) are a second gate so a contradictory half-paused record never fires even before self-heal runs. enabledTF)r4rrs ris_job_runnablers9 779d # #uu 4r/ct|d}|dvr|S|ddst|s|dkrdS|pdS|dks|drdS|pdS)u&Operator-facing state derived from the scheduler-honoured flag. A job with ``enabled=true`` must never display as paused — that was the 07-30 outage failure mode (list looked frozen, fleet kept merging). Terminal states (completed/error) are preserved regardless of enabled. r>r} completedrTrr scheduled)rr4rMr)rstoreds rrrscggg.. / / 5 5 7 7F ''' 779d # #" S ! ! Vx%7%78!! SWW[11{  [ r/pathcb tj|ddS#ttf$rYdSwxYw)z > > F v}fm44444     0 &-           s$BB8 B C$(CCct}|jdd|jddt |jt |jdS)z6Ensure cron directories exist with secure permissions.Tparentsexist_okN)r>r'mkdirr)rstores rrqrqBsk  ! !E N555 4$777 !!!!!r/sc>|}tjd|}|st d|dt |d}|dd}dddd }|||zS) u Parse duration string into minutes. Examples: "30m" → 30 "2h" → 120 "1d" → 1440 zD^(\d+)\s*(m|min|mins|minute|minutes|h|hr|hrs|hour|hours|d|day|days)$zInvalid duration: 'z''. Use format like '30m', '2h', or '1d'rfrr!i)mhd)rMlowerrematchrPintgroup)rrrunit multiplierss rparse_durationrOs  A H\^_ ` `E [YqYYYZZZ  A  E ;;q>>! D..K ;t$ $$r/rc|}|}|}|dr5|dd}t|}d|d|ddS|}t |dkr|t d|ddDr[tstd  t|n'#t$r}td |d |d}~wwxYwd ||d Sd|vstj d|r tj|dd}|j)t#j}||}d|d|ddS#t$r}td|d |d}~wwxYw t|}t#t)|z} d| d|dS#t$rYnwxYwtd|d)uM Parse schedule string into structured format. Returns dict with: - kind: "once" | "interval" | "cron" - For "once": "run_at" (ISO timestamp) - For "interval": "minutes" (int) - For "cron": "expr" (cron expression) Examples: "30m" → once in 30 minutes "2h" → once in 2 hours "every 30m" → recurring every 30 minutes "every 2h" → recurring every 2 hours "0 9 * * *" → cron expression "2026-02-03T14:00" → once at timestamp zevery Nintervalr)kindminutesrc3@K|]}tjd|VdS)z ^[\d\*\-,/]+$N)rr).0ps r z!parse_schedule..s@*+!1%%r/zOCron expressions require 'croniter' package. Install with: pip install croniterzInvalid cron expression 'z': r)rrrTz^\d{4}-\d{2}-\d{2}Zz+00:00tzinfooncezonce at z%Y-%m-%d %H:%M)rrrzInvalid timestamp 'rzonce in zInvalid schedule 'z'. Use: - Duration: '30m', '2h', '1d' (one-shot) - Interval: 'every 30m', 'every 2h' (recurring) - Cron: '0 9 * * *' (cron expression) - Timestamp: '2026-02-03T14:00:00' (one-shot at time))rMr startswithrsplitlenallrrPrr^rrr fromisoformatreplacer _hermes_now isoformatstrftimer) roriginalschedule_lower duration_strrpartsrdt hermes_tzrs rparse_scheduler ds$~~HH^^%%N  ** |))++  ..****    NN  E 5zzQ3/4RaRy !! pnoo o K X     K K KIIIaIIJJ J K    h"(#8(CC E'(8(8h(G(GHHBy 'MM0 ZZyZ11,,..Ebkk2B&C&CEE   E E EC8CCCCDD D E   **7!;!;!;;&&((,(,,          CX C C C  sDC++ D5D  D2BF88 GGG AH)) H65H6r ctj}|jStjj}|||S||S)aReturn a timezone-aware datetime in Hermes configured timezone. Backward compatibility: - Older stored timestamps may be naive. - Naive values are interpreted as *system-local wall time* (the timezone `datetime.now()` used when they were created), then converted to the configured Hermes timezone. This preserves relative ordering for legacy naive timestamps across timezone changes and avoids false not-due results. Nr)rrrr astimezoner)r  target_tzlocal_tzs r _ensure_awarersf $I y<>>,,..5zzz**55i@@@ == # ##r/rcurrentcv|j|jdS||kS)u Return True when a stored aware timestamp uses a different UTC offset. Naive stored timestamps return False: they carry no offset to compare, and are normalized by ``_ensure_aware`` instead — they intentionally never take the offset-repair path. NF)r utcoffsetrrs r_timezone_offset_mismatchrs;} 6u     !2!2!4!4 44r/c^|d|dkS)aReturn True when the stored local wall-clock time has not arrived yet. Cron schedules express local wall-clock intent. If Hermes/system local time changes after next_run_at was persisted, an old offset can make a future wall-clock run look due at the converted absolute time (for example 21:00+10 becomes 13:00+02). Comparing naive wall-clock values lets us distinguish that migration case from a genuinely missed run whose scheduled wall time has already passed. Nr)rrs r_stored_wall_clock_is_futurers+ >>> & &)E)E EEr/ last_run_atrrcDt|tr|ddkrdS|rdS|d}|sdS tt j|}n#t $rYdSwxYw||ttz kr|SdS)aReturn a one-shot run time if it is still eligible to fire. One-shot jobs get a small grace window so jobs created a few seconds after their requested minute still run on the next tick. Once a one-shot has already run, it is never eligible again. rrNr)seconds) rrr4rrrr^rONESHOT_GRACE_SECONDS)rrrr run_at_dts r_recoverable_oneshot_run_atrs h % %f)=)=)G)Gtt \\( # #F t!("8"@"@AA ttC),ABBBBBB 4s!A11 A?>A?cfd}d}|d}|dkr<|dddz}|dz}t|t||S|d krtr|d }|r t }t ||}|t} |t} t| | z }|dz}t|t||S#t$rYnwxYw|S) a(Compute how late a job can be and still catch up instead of fast-forwarding. Uses half the schedule period, clamped between 120 seconds and 2 hours. This ensures daily jobs can catch up if missed by up to 2 hours, while frequent jobs (every 5-10 min) still fast-forward quickly. r#i rrrrfr!rrr) r4rSminrrrget_nextrr total_secondsr^) r MIN_GRACE MAX_GRACErperiod_secondsgracerrrfirstseconds r_compute_grace_secondsr*s;II <<  D z!i33b8!#9c%33444 v~~*,,~||F##   !mmc** h//x00!$fun%C%C%E%E!F!F&!+9c%&;&;<<<     sBD!! D.-D.cvt}t|tsdS|d}|dS|dkrt |||S|dkr|d}|dS|rY t t j|}|t|z}n6#t$r|t|z}YnwxYw|t|z}| S|dkr|d }|sdStst d |dS|}|r5 t t j|}n#t$r|}YnwxYwt||} | t }| SdS) zo Compute the next run time for a schedule. Returns ISO timestamp string, or None if no more runs. NrrrrrrrrzCannot compute next run for cron schedule %r: 'croniter' is not installed. croniter is a core dependency as of v0.9.x; reinstall hermes-agent or run 'pip install croniter' in your runtime env.)rrrr4rrrrrr^rrr_r`rr") rrrrrlastnext_runr base_timers rcompute_next_runr/4s --C h % %t <<  D |t v~~*8SkRRRR   ,,y)) ?4  8 <$X%;K%H%HII)G"<"<"<< < < <7!;!;!;; <Yw7777H!!### ||F## 4!!  NN    4    )(*@*M*MNN      i((==**!!### 4s$94B..C C!E)) E87E8ctt|ttjddS)aRAtomically write the current epoch time to ``path``. Delegates to :func:`utils.atomic_write_text` (tmpfile + fsync + ``atomic_replace``, same pattern as ``save_jobs``) so a concurrent reader in another process (``hermes cron status``) never sees a torn/truncated file. Best-effort: failures are swallowed by callers. z.hb_) tmp_prefixN)rqrrrurs r_atomic_write_epochr2vs6MMMdC ,,@@@@@@r/c 8ttjt|jdd\}} t j|dd5}|ttd|| t j | dddn #1swxYwYt||dS#t$r( t j|n#t$rYnwxYwwxYw) z2Atomically persist a non-negative integer counter..tmpz.count_dirsuffixprefixwrgrhrN)rqtempfilemkstemprparentrKfdopenwriterSflushfsyncrr BaseExceptionunlinkr{)rrfdtmp_pathfs r_atomic_write_counterrFsTMMM#DK(8(8PYZZZLB  Yr3 1 1 1 !Q GGCAu && ' ' ' GGIII HQXXZZ  ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! x&&&&&   Ih        D  sTC'A+C > C' CC'CC'' D2DD DDDDFsuccessct} t|jdz n#t$rYnwxYw|r+ t|jdz dS#t$rYdSwxYwdS)u?Record a ticker liveness signal, and optionally a successful-tick signal. The ticker calls this once per loop iteration. ``success=True`` additionally bumps the *last successful tick* marker. We track two distinct signals so `hermes cron status` can tell a thread that is merely *alive and looping* (heartbeat fresh, success stale) from one that is actually *firing jobs* (both fresh) — a ticker stuck failing every tick would otherwise keep the plain heartbeat fresh and falsely report healthy (#32612, #32895). Resolution uses ``_current_cron_store()`` so the heartbeat is correctly scoped to the active profile's store — critical under multiplex_profiles where each profile needs its own liveness signal (#69377). Best-effort: a write failure must never disrupt the tick loop. rr N)r>r2r'r^)rGrs rrecord_ticker_heartbeatrIs ! !E EN-??@@@@        1F F G G G G G    DD s( 55A A"!A"c |d}tdtjt |z S#t $rYdSwxYw)Nrgrhg) read_textrMrSrurOr^)rrUs r_epoch_file_agerLsinngn..44663 eCjj0111 ttsAA A)(A)cLt}t|jdz S)uSeconds since the ticker loop last iterated, or None if unknown. None = heartbeat file missing/unreadable (older build, never ran, or a torn read). Callers treat None as "cannot determine", not "dead". Resolution uses ``_current_cron_store()`` so the heartbeat is correctly scoped to the active profile — critical under multiplex_profiles where ``hermes cron status`` must report per-profile liveness (#69377). rr>rLr'rs rget_ticker_heartbeat_agerOs% ! !E 5>,>> ? ??r/cLt}t|jdz S)u-Seconds since the ticker last completed a tick WITHOUT raising, or None. Resolution uses ``_current_cron_store()`` so the heartbeat is correctly scoped to the active profile — critical under multiplex_profiles where ``hermes cron status`` must report per-profile liveness (#69377). r rNrs rget_ticker_success_agerQs% ! !E 5>,AA B BBr/c8tjdz } t|d}n#t t f$rd}YnwxYwt|td|dzdS#t$rYdSwxYw)zIIncrement the profile-local stale-schedule catch-up counter, best effort.catch_up_occurrencesrgrhrrfN) r>r'rrKrMr{rPrFrSr^)rrs rrecord_catch_up_occurrencerTs  ),B BD  88>>@@AAEE$   EEE dC5MMA$566666      s/5AB A%"B $A%%$B BBmessagect}|jdz } ttjt |jdd\}} tj|dd5}| tj d| d| tj |d d d n #1swxYwYt||d S#t $r( tj|n#t$$rYnwxYwwxYw#t&$rYd SwxYw) aOPersist the most recent tick failure so other processes can surface it. The ticker thread lives inside the gateway process; ``hermes cron status``/``list`` run in a separate process and previously could only infer "ticks may be failing" from marker staleness, with no clue WHY. A root-owned ``jobs.json`` (#68483) failed every tick for ~14h with the reason visible only in the gateway's errors.log. Writing the last error next to the heartbeat markers gives the CLI something concrete to show. Best-effort: a write failure must never disrupt the tick loop. ticker_last_errorr4z.terr_r5r9rgrh N)r>r'rqr:r;rr<rKr=r>rurMr?r@rrrArBr{r^)rUrrrCrDrEs rrecord_ticker_errorrYs ! !E >/ /D  'DK     H 2sW555 %49;;=='--//===>>> $$$ % % % % % % % % % % % % % % % 8T * * * * *     (####             sl:ED,A9C1% D1C55D8C59D ED.-E. D;8E:D;;EE EEctjdz } tdt|dS#t tf$rYdSwxYw)z7Return the profile-local stale-schedule catch-up count.rSrrgrh)r>r'rSrrKrMr{rPrs rget_catch_up_occurrence_countr[st  ),B BD1c$..'.::@@BBCCDDD Z qqsAAA0/A0c~t} |jdz dS#t$rYdSwxYw)zGRemove the last-tick-error marker after a successful tick. Best-effort.rWN)r>r'rBr{rs rclear_ticker_errorr] sT  ! !E - -5577777      s . <<c8t} |jdz d}n#t$rYdSwxYw|}t |dkrdSd|dd}|pdS)zr'rKr^ splitlinesrjoinrM)rrUlinesrUs rget_ticker_last_errorrbs  ! !E~ 33>>>PP tt NN  E 5zzA~~tiiabb ""((**G ?ds / ==r(ct|dd5}|}dddn #1swxYwY tj|dfS#tj$rtj|ddfcYSwxYw)aTolerantly parse jobs.json; shared by load_jobs and the save-path peek. Returns ``(data, used_strict_fallback)``. utf-8-sig absorbs a Windows BOM; a strict parse failure is retried with ``strict=False`` to survive bare control characters in string values. IO errors from the open and parse errors from the fallback propagate to the caller, which decides between repair (load_jobs) and bail-out (peek). rz utf-8-sigrhNF)strictT)rrreadjsonloadsJSONDecodeError)r(rErUs r_parse_jobs_filerj%s i{ 3 3 3qffhh3z#%%  333z#e,,,d22223s488A'B?Bctj}tt|}|st dgS t |\}}n{#t$r3}t d|td||d}~wt$r3}t d|td||d}~wwxYwt|trT|dg}|r+|r)t|tdt ||St|t"r<|r)t|tdt ||Std t%|j) zLoad all jobs from storage.NzIOError reading jobs.json: %szFailed to read cron database: z#Failed to auto-repair jobs.json: %sz*Cron database corrupted and unrepairable: jobsz8Auto-repaired jobs.json (had invalid control characters)z3Auto-repaired jobs.json (bare list wrapped as dict)z7Cron database corrupted: expected {'jobs': [...]}, got )r>r(rq_jobs_file_stampr_record_load_stamprjr|r_r} RuntimeErrorr^rrr4 save_jobsr`rtyper*)r(pre_read_stampdata _strict_retryrrls r load_jobsru6s#%%/IMMM&i00N     4    T.y99mm HHH 4a888?A??@@aG TTT :A>>>KKKLLRSST$xx##  WT W dOOO NNU V V V>*** $  R dOOO NNP Q Q Q>*** YDJJDWYY  s$A** C"4.B"" C"/.CC"cXtj}|sgS t|\}}n#t$rYdSwxYwt |t r/|dg}t |tr|ndSt |tr|SdS)uBest-effort read of on-disk jobs without repair side-effects. Caller must hold ``_jobs_lock()``. Returns ``[]`` when the file is missing, ``None`` when the payload is unreadable/corrupt (caller should not attempt a shrink-merge against an unknown baseline). Never calls ``save_jobs`` — the repair-free property is what keeps the save path re-entrancy-safe (a repairing read here would recurse through ``_save_jobs_unlocked``). Nrl) r>r(rrjr^rrr4r)r(rs_rls r_peek_jobs_unlockedrxes$%%/I      "9--aa tt$8xx##!$--7tt47$ 4s> A  A cv |}|j|j|jfS#t$rYdSwxYw)u^Cheap change-detection stamp for jobs.json: ``(mtime_ns, size, ino)``. ``None`` means the file is missing/unstatable. Used as a fast-path gate in front of the shrink-merge so the healthy no-race save costs one ``stat()`` instead of a full read+parse (the ``advance_next_runs`` batching exists because this path is hot — see its docstring). ``st_ino`` is included because every legitimate writer goes through mkstemp+rename (new inode), so even a same-size write inside one mtime quantum on a coarse-clock filesystem (ext4 jiffies, network mounts) cannot false-match. N)stat st_mtime_nsst_sizest_inor{)r(sts rrmrm~sK ^^   BI66 tts '* 88stampcNttddsdS|t_dS)ueRemember jobs.json's stamp for the enclosing _jobs_lock() section. No-op outside a critical section. Lets the save path skip the shrink-merge parse when the file provably hasn't changed since this section loaded it (#80703's fast-path). The caller must capture the stamp BEFORE reading the file: a sibling landing mid-read then leaves the recorded stamp OLDER than disk — a mismatch, so the merge runs (fail-safe direction). Stamping after the read would let that sibling's write be certified as "seen" without being in the loaded payload, wrongly suppressing the recovery. rerN)rmrnrp)rs rrnrns- #Wa 0 0"'r/ removed_idsrlrcttdd}|&ttj|kr|St }||Sd|pdD}t }|D]T}t|tr=| dr(| t|dUg}|D]r}t|ts| d} | s0t| } | |vs| |vrH| || | s|s|Stdt|d|D||zS)uReturn *jobs* plus any on-disk jobs missing from the save payload (#80624). Under ``_jobs_lock()``'s degraded flock-timeout path (#60703), two processes can both believe they own the store. A writer that loaded an older/smaller snapshot then calls ``save_jobs`` and would otherwise clobber concurrent creates (the filed ``no_agent`` watchdog pattern: CLI/tool create succeeds, then a gateway tick/remove rewrites ``jobs.json`` empty or without the new id). Intentional deletes pass ``removed_ids``. Any other id present on disk but absent from *jobs* is treated as a concurrent create and merged back before the atomic write. The caller's list is never mutated — a new list is returned when anything was recovered. Fast path: when the enclosing critical section recorded a load stamp and the file's ``(mtime_ns, size)`` still matches, nothing can have changed underneath us, so the read+parse is skipped entirely — one ``stat()`` on the healthy no-race save. rpNc0h|]}|t|Sr.rris r z._merge_unexpected_disk_jobs..s#@@@!a@s1vv@@@r/r.rzPreserved %d cron job(s) present on disk but missing from the in-memory save payload (concurrent create under degraded lock or stale writer) (#80624): %sc8g|]}|dSrr4rjs r z/_merge_unexpected_disk_jobs..s"(((t(((r/)rmrnrmr>r(rxrArrr4addrrr_r`r) rlrr disk_jobsintended_removenew_idsr recovereddisk_jobdisk_ids r_merge_unexpected_disk_jobsrs0 $lD 9 9E -.A.C.C.MNNRWWW #%%I @@ (9r@@@OG(( c4  (SWWT]] ( KKCI ' ' '&(I  (D))  ,,t$$  g,, g  O!;!; """ G   NN ( I((i(((  ) r/rrrc tj}t tj|}n=#t $r0 tj|j}n#t $rd}YnwxYwYnwxYwd} tdD]}|st||}tj t|jdd\}} tj |dd 5}tj|td |d d |tj|dddn #1swxYwYn7#t($r* tj|n#t $rYnwxYwd}wxYw|st-t.dd} | duot1|| k} | rdn t3} | `d|D d|pdD t5 fd| Dr* tj|n#t $rYnwxYwd}t7||d}t9|t;||t=ddS|st||}tj t|jdd\}}tj |dd 5}tj|td |d d |tj|dddn #1swxYwYt7||d}t9|t;||dS#t($r*|& tj|n#t $rYnwxYwwxYw)aSave all jobs to storage. Caller must hold _jobs_lock(). ``removed_ids`` lists job ids this mutation intentionally deleted. ``replace=True`` skips the shrink-merge guard (tests / disaster recovery that mean to rewrite the store wholesale). Nrrr4z.jobs_r5r9rgrh)rl updated_atrF)indent ensure_asciirpch|]A}t|t|d,t|dBSrrrr4rrs rrz&_save_jobs_unlocked..(sV###%a..#4555;;#AdG ###r/c0h|]}|t|Sr.rrs rrz&_save_jobs_unlocked..-s#III1qIAIIIr/r.c3K|]\}t|toB|do-t|dvot|dvV]dSrNr)rdjintended payload_idss rrz&_save_jobs_unlocked...s  #2t,,:FF4LL:4MM<: 4MM9 r/)r>r(rqrKrzr{r<rangerr:r;rr=rgdumprrr?r@rrArBrmrnrmrxanyrrrrn)rlrrr( _stat_beforerD_attemptrCrE_stamp _unchangedrrrs @@r_save_jobs_unlockedrs6$%%/IMMM wy))     79#344LL   LLL  H`aD D H R24[QQQ#+ ())&LB Yr3999)QI!%[]]5L5L5N5NOO %*  GGIIIHQXXZZ((()))))))))))))))!   Ih''''D   ! !!1<FF$&P+;I+F+F&+P%/IDD4G4I4I (##!%###K JI1BIIIH #,  !!Ih////&!!! D!#' 8Y / / /H  # # # $Y = = = t $ $ $ FF N.tMMMD'I$%%fX   HYr3 1 1 1 !Q I[]]-D-D-F-FGG"      GGIII HQXXZZ  ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! x+++Y L99999     (####     s: A4AA4 A.+A4-A..A43A4:AN E/#A4E# E/#E' 'E/*E' +E/.N / F#:FF# FF#FF##A>F$NN " " "!NNN " n ,,s%A%B.. B=<B=C C#"C#ct|dt|dt|ddt|dfS)zBReturn the stored inference-routing fields in their semantic form.rrrTrr)rr4rrs r_normalized_inference_axesrso %SWWZ%8%899$SWWW%5%566$SWWZ%8%8tTTT SWWZ !!  r/monitor_script monitor_urlrc|r|rtd|s|r|rtd|r|stddSdS)a.Shared create/update validation for job execution-mode invariants. ONE owner for the class: create_job and update_job both call this so an invariant enforced at create time cannot be violated through the update door (monitor jobs silently degrading when no_agent is flipped on, etc.). uamonitor_script and monitor_url are mutually exclusive — a job can only have one monitor source.umonitor_script/monitor_url cannot be combined with no_agent=True — the whole point of a monitor job is to suppress or wake the AGENT based on source changes. Use a plain no_agent script job instead.udno_agent=True requires a script — with no agent and no script there is nothing for the job to run.N)rP)rrrrs r_validate_job_mode_invariantsrs +  0     + 8  P      3       r/rrrepeatdeliverorigin context_fromenabled_toolsetsattach_to_sessionc*t|}||dkrd}|ddkr|d}||rdnd}tjjdd}t }t ||}t|}t| }t| d }t| tr!t|  nd}|pd}| r d | Dnd}|pd}t|}t|}t|tr|nd}t|tr!t| nd}|pd}t|tr!t| nd} | pd} t|| ||t| tr,| r| gnd} n&t| trd | Dpd} nd} t|}!dd lm}"|"|!||!p|r|dndp|r|ndpd}#t%||||\}$}%t'|}&|ddkrr|&p|dp|}'t*d|p|#dd |'t.t1d|'dt.did|d|p|#dd d|!d|d|r|dndd|d|d|$d|%d|d |d!|d"|d#| d$dd%| d&|id'|d(|d)|dd*d+d d,d-d.dd/dd0|d1|&d2dd3dd4dd5dd6|d|d7|d8|}(|||(d9<t35t5})|)|(t9|)dddn #1swxYwY|(S):u! Create a new cron job. Args: prompt: The prompt to run (must be self-contained, or a task instruction when skill is set). Ignored when ``no_agent=True`` except as an optional name hint. schedule: Schedule string (see parse_schedule) name: Optional friendly name repeat: How many times to run (None = forever, 1 = once) deliver: Where to deliver output ("origin", "local", "telegram", etc.) origin: Source info where job was created (for "origin" delivery) skill: Optional legacy single skill name to load before running the prompt skills: Optional ordered list of skills to load before running the prompt model: Optional per-job model override provider: Optional per-job provider override base_url: Optional per-job base URL override script: Optional path to a script whose stdout feeds the job. With ``no_agent=True`` the script IS the job — its stdout is delivered verbatim. Without ``no_agent``, its stdout is injected into the agent's prompt as context (data-collection / change-detection pattern). Paths resolve under ~/.hermes/scripts/; ``.sh`` / ``.bash`` files run via bash, anything else via Python. context_from: Optional job ID (or list of job IDs) whose most recent output is injected into the prompt as context before each run. Useful for chaining cron jobs: job A finds data, job B processes it. enabled_toolsets: Optional list of toolset names to restrict the agent to. When set, only tools from these toolsets are loaded, reducing token overhead. When omitted, all default tools are loaded. Ignored when ``no_agent=True``. workdir: Optional absolute path. When set, the job runs as if launched from that directory: AGENTS.md / CLAUDE.md / .cursorrules from that directory are injected into the system prompt, and the terminal/file/code_exec tools use it as their working directory (via TERMINAL_CWD). When unset, the old behaviour is preserved (no context files injected, tools use the scheduler's cwd). With ``no_agent=True``, ``workdir`` is still applied as the script's cwd so relative paths inside the script behave predictably. no_agent: When True, skip the agent entirely — run ``script`` on schedule and deliver its stdout directly. Empty stdout = silent (no delivery). Requires ``script`` to be set. Ideal for classic watchdogs and periodic alerts that don't need LLM reasoning. monitor_script: Optional path to a cheap monitor source script (same resolution/containment rules as ``script``: relative to ~/.hermes/scripts/, .sh/.bash via bash, else Python). Each tick the script runs FIRST and its output is hashed as exact bytes: unchanged output suppresses the agent run entirely (recorded as a silent 'no_change' tick); changed output injects a MONITOR CHANGE DETECTED block (unified diff + new output) into the prompt before a normal agent run. Scripts should emit stable output (no timestamps). Mutually exclusive with ``monitor_url``; incompatible with ``no_agent=True``. monitor_url: Optional http(s) URL used as the monitor source instead of a script — fetched with a bounded GET each tick. Same hash-suppression semantics as ``monitor_script``. Returns: The created job dict Nrrrrfrlocal Trcg|]D}t|#t|ESr.rrM)rts rrzcreate_job..s9VVVas1vv||~~V3q66<<>>VVVr/cg|]D}t|#t|ESr.rrs rrzcreate_job..s9OOO1A OA OOOr/)check_gateway_lifecyclerrrrrrzKRejecting one-shot cron job '%s': run_at %s is outside the %ss grace windowrRequested one-shot time  is more than &s in the past and cannot be scheduled.rrrrrrrrrrrrrr monitor_staterrrrr)timesrrrrr paused_reason created_at next_run_atr last_status last_errorlast_delivery_errorrrrr)r uuiduuid4hexrrrrrrrMrrrrrcron.lifecycle_guardrrr/r4r_r`rrPrrurrp)*rrrrrrrrrrrrrrrrrrrparsed_schedulerXrnormalized_skillsrrrnormalized_scriptnormalized_toolsetsnormalized_workdirnormalized_no_agentnormalized_attachnormalized_monitor_scriptnormalized_monitor_url prompt_textrrrrrrrrls* r create_jobr!slb%X..Ofkkv&((V^$1((' Z\\ crc "F -- ! ! # #C-eV<<3E::6x@@6xVZ[[[/9&#/F/FPF ))+++D)1TZjtVV3CVVVVpt-5+G44x..-78I4-P-PZ))VZ?I.Z]?^?^ hN 3 3 9 9 ; ; ;dh 9 AT9CKQT9U9U_S--33555[_3;t"! ,$$1=1C1C1E1EO **,,--4 L$ ' 'OO OOOWSW  "6**K=<<<<<K):;;;Q ?+ @ cA+ B {C+ D tE+ F tG+ H dI+ J tK+ N 7O+ P &Q+ R /S+ T %U+ + C^$#4  {{ C$ Js 3PP P cft}|D]}|d|krt|cS dS)zGet a job by ID.rN)rur)rXrlrs rget_jobrsG ;;D.. t9  (-- - - -  4r/cHeZdZdZdedeeeefffd ZxZ S)AmbiguousJobReferencez1Raised when a job name matches more than one job.refmatchesc ||_||_dd|D}td|dt |d|ddS)N, c3&K|] }|dV dSrr.)rrs rrz1AmbiguousJobReference.__init__..s&11A$111111r/z Job name 'u' is ambiguous — matches z jobs: z. Use the job ID instead.)r r r`super__init__r)selfr r ids __class__s rrzAmbiguousJobReference.__init__s ii1111111  ' ' 'W ' 'c ' ' '     r/) r*r+r,__doc__rr r r r __classcell__)rs@rr r  s^;; C $tCH~*>          r/r r c<|sdSt}|D]}|d|krt|cS |fd|D}|sdSt|dkrt |d|Dt|dS)aUResolve a job reference (ID or name) to a job record. - Exact ID match wins (works even if a different job's name equals this ID). - Otherwise, case-insensitive name match. - If a name matches more than one job, raises AmbiguousJobReference so the caller can surface the matching IDs rather than silently picking one. Nrcng|]1}|dpdk/|2S)rrJ)r4r)rr ref_lowers rrz#resolve_job_ref..(s>RRR!f (;'B'B'D'D 'Q'QA'Q'Q'Qr/rfc,g|]}t|Sr.rrs rrz#resolve_job_ref..-s!AAAq'**AAAr/r)rurrrr )r rlr name_matchesrs @rresolve_job_refrs t ;;D.. t9  (-- - - -  IRRRRtRRRL t <1# AALAAA    !a 1 11r/include_disabledcdtD}|s d|D} ddlm}|d|D}n#t$ri}YnwxYw|D].}||dd|d</|S) z2List all jobs, optionally including disabled ones.c,g|]}t|Sr.rrs rrzlist_jobs..4s! : : : !! $ $ : : :r/c>g|]}|dd|S)rTrrs rrzlist_jobs..6s+:::a155D#9#9::::r/r)latest_executionsc:g|]}|ddS)rrJr)rrs rrzlist_jobs..:s&#F#F#F#CGGD"$5$5#F#F#Fr/rrJlatest_execution)rucron.executionsr!r^r4)rrlr!latestrs r list_jobsr&2s : :ikk : : :D ;::4:::555555""#F#F#F#F#FGG @@"(**SWWT2->->"?"?  KsA AAupdatesc $ t|pi}|r2tddt |t 5t }t|D]y\}}|d|krd|vr$|d}|dvrd|d<nt||d<dD]M}||vrG||}t|tr!t| nd}|pd||<Nt|} ti||} hd|r| d } t| tr!t|  nd} t| d pd| d pdt!| d | pdd |v} t!hd|ot| | k} d|vsd|vrJt#| d| d}|| d<|r|dnd| d<| r!| d }t|trt%|}|| d <|d|d| d| d<| ddkrt'|}||ddkrh|dp|}t(d| d||t,td|dt,d|| d<| rlt/| d| d | d!| d "\}}|| d#<|| d$<| d%d&r| ddkr| dsrt'| d }|V| d ddkr7| d dd'}td|d(t,d)|| d<| ||<t1|t3||ccdddS dddn #1swxYwYdS)*zCUpdate a job by ID, refreshing derived schedule fields when needed.z%Cron job field(s) cannot be updated: r rr>FNrJN)rr>rrrrrrrrr>rrrrrrrrrrrrrrzRRejecting one-shot cron job update '%s': run_at %s is outside the %ss grace windowrrrrrrrrrrrrTr is in the past (grace window: zs) and cannot be scheduled.)_IMMUTABLE_JOB_FIELDS intersectionrPr`sortedrru enumeraterrrrMrrr4rrrr r/r_r`rrrpr)rXr' bad_fieldsrlrr_wd _mon_field_mvprevious_inference_axesupdated _upd_scriptschedule_changedinference_fields_changedrupdated_scheduleupdated_next_runrrrr-s r update_jobr9BsB '33GMrBBJ  SDIIfZ>P>P4Q4Q S S    o2o2{{oom 2m 2FAs4yF""G##i(+++)-GI&&);C)@)@GI&@ 6 6 ((!*-C.8c.B.BL#c((..***C*-+GJ'&@&E&E #)*#*:#6 .44;'56F'G'G$*:GJ'.5kk&$((GKK@R4S4STT//*+;;w''833'78H'I'I$)0,0088FBB!1!5!5h!?!?!SCS>#KK77"1 )]v]]4]]].>GM*' ;4U$[[44!++g..$[[44$[[44 5551!> 0A+,,:(){{9d++ 2 G0D0D0P0PY`YdYderYsYs0P+GJ,?@@# (;(?(?(G(G6(Q(Q$Z044XyIIF$]6]]*?]]]*2 &DG dOOO(a11 1 1_o2o2o2o2o2o2o2o2m 2o2o2o2o2o2o2o2o2o2o2o2o2o2o2o2` 4sR TT  T reasonct|}|sdSt|dddt|dS)z:Pause a job without deleting it. Accepts a job ID or name.NrFr)rrrrrr9rr)rXr:rs r pause_jobr=s[ & ! !C t  D $0022#     r/c :t|}|sdSt|d}|V|dddkr7|ddd}td|dtd t |d d d dd|d S)zWResume a paused job and compute the next future run from now. Accepts a job ID or name.NrrrrrzCannot resume: one-shot time r)zs) and will never fire.rTrrrrrr)rr/r4rPrr9)rXrrrs r resume_jobr@s & ! !C t"3z?33Ks:226::fDDZ$$Xy99 MF M M3 M M M     D  !&      r/c t|}|sdSt|dddddtdS)zKSchedule a job to run on the next scheduler tick. Accepts a job ID or name.NrTrr?r<)rXrs r trigger_jobrBs^ & ! !C t  D  !&==2244      r/c6t|}|sdS|dt5t}t|}fd|D}t||krt }t |h|rtj| ddl m }|n-#t$r t dd YnwxYw d d d dS d d d n #1swxYwYdS) zRemove a job by ID or name.Frc,g|]}|dk|Srr.)rr canonical_ids rrzremove_job..s';;;a1T7l#:#:#:#:#:r/rr) clear_notepadz*Failed to clear notepad for removed job %sTr[N)rrrurrrprshutilrmtree cron.notepadrFr^r_debug)rXrrl original_lenjob_output_dirrFrEs @r remove_jobrMs & ! !C ut9L {{4yy ;;;;4;;; t99| # #-\::N d 7 7 7 7$$&& . n--- 666666 l++++    @ 4  1 $ 2 5s6BD4CD'C0-D/C00DDDct5t}t|D]{\}}|d|krjt|d}|rd|d<n|dd||kr|||<t ||ccdddS| dddn #1swxYwYdS)aSet/clear the preflight alert-dedup marker; return the PRIOR value. The marker records that the operator was already alerted about this job's blocked configuration, so the scheduler alerts exactly once and stays silent on subsequent ticks until the config heals (same alert-once shape as the dead-pin auto-pause in #73506). Persisted on the job record so the dedup survives gateway restarts. rpreflight_alertedTNF)rrur-rr4poprp)rXrrlrrpriors r_set_preflight_alertedrRs6   {{oo  FAs4yF""SWW%899::7/3C+,,GG/666E>>!DGdOOO         #                 5sB B6(B66B:=B:c"t|dS)zAMark the job as preflight-alerted; return True if it already was.TrRrXs rmark_preflight_alertedrV5s !&$ / //r/c&t|ddS)z@Clear the preflight alert-dedup marker (config validates again).FNrTrUs rclear_preflight_alertedrX:s65)))))r/r}delivery_errorstatusc bt5t}t|D]M\}}|d|kr:t}||d<|p|rdnd|d<|s|nd|d<|r|dd||d <d|d <|d d|d <|d r|d } | d } | dd} |did} | dko| duo | dko| dk} | s | dz } | | d<| 8| dkr2| | kr,d|d<d|d<d|d<t|ddddSt|d||d<|d|did} | dvrdd|d<|dsd|d<t d|d|dd| n)d|d<d|d<n|ddkrd|d<t|ddddSOt d|ddddS#1swxYwYdS) u Mark a job as having been run. Updates last_run_at, last_status, increments completed count, computes next_run_at, and auto-deletes if repeat limit reached. ``delivery_error`` is tracked separately from the agent error — a job can succeed (agent produced output) but fail delivery (platform down). ``status`` overrides the derived ``last_status`` ("ok"/"error") with a specific terminal status for this run — e.g. ``"blocked_config"`` when the pre-dispatch configuration validation refused to run the agent (T1-26), so `cronjob list` distinguishes "your config is broken" from "the run itself failed". rrokr}rNrrOr fire_claim run_claimrrrrrrrrfFrrr>rrztFailed to compute next run for recurring schedule (is the 'croniter' package installed in the gateway's Python env?)zyJob '%s' (%s) could not compute next_run_at; leaving enabled and marking state=error so the job is not silently disabled.rrrrz0mark_job_run: job_id %s not found, skipping save) rrur-rrrPr4rpr/r_r}r`)rXrGr}rYrZrlrrrrrrrpreclaimed_oneshots r mark_job_runr`?s$ aSaS{{oo] ] FAs4yF""!mm--//%(M"%+%M0LWM"18$BEEdL!7GG/666-;)*%)L!77;''3'+C $778$$ ]F"JJw//E & ; : :I77:r2266v>>D*!-*!AI*&M ' .8!Q .7{+(UQYY9;M;M*/I'2G -1M*!$yaSaSaSaSaSaSaSaS~&6c*os%K%KM"}%-77:r2266v>>D333'.G "ww|44!J -  < GGFCGGD#,>,>?? */I'2G WWW%%11#.CL$aSaSaSaSaSaSaSaS#| I6RRRCaSaSaSaSaSaSaSaSaSaSaSaSaSaSaSaSaSaSsE&J$C+J$:J$$J(+J(cF|ddS |dpi}|dpi}d|dd|dd |d d d |d d d|ddd|dddtd}t|dd|td|d|dd dS#t $r:}td|d|Yd}~dSd}~wwxYw)uLeave an operator-visible trace when a wedged one-shot is removed. A finite one-shot whose dispatch was claimed (``repeat.completed`` >= ``repeat.times``) but which never reached ``mark_job_run`` (``last_run_at`` is null) was interrupted mid-run — scheduler restart, gateway kill, or a non-Exception escape (#73973). The recovery guards remove such jobs so they stop appearing due, but a silent removal leaves the user with no output, no error, and no job record. Write a small diagnostic file into the job's output directory so the removal is observable and debuggable. Best-effort: diagnostics must never break the removal itself. rNrr^z7# Cron job removed without producing output - job id: rz - name: rz - dispatch claimed: rrrrz - run claimed at: atrz by byz - removed at: u This one-shot job's dispatch was claimed, but the run never completed (`last_run_at` was never written) — the scheduler process was most likely killed or restarted mid-execution. The job has been removed to stop it re-firing; recreate it to run again. rJuXJob '%s': removed without a completed run — diagnostic written to its output directoryz8Failed to write wedged-oneshot diagnostic for job %r: %s)r4rrsave_job_outputr_r`r^rJ)rrclaimrrs r _write_wedged_oneshot_diagnosticrfs ww}) ""(b $$*   wwv  $*::k3#?#?  CI**WVYBZBZ  "'4!;!;  BG4QZA[A[   )]]4466     b))4000 # GGFCGGD#.. / /         F GGDMM1          sEE F &/FF c :t5t}t|D]8\}}|d|kr|diddkrddddS|d}|sddddS|d}||d krddddS|d d }||kr|d qd |d <d |d<d|d<t |t d|d|dd||dddd S||t ||ht|t d|d|dd||dddd S|dz|d <t |t d|d|dd|d |ddddSt d| ddddS#1swxYwYdS)upAtomically claim a finite one-shot job dispatch BEFORE execution. Increments ``repeat.completed`` under the cross-process jobs lock and persists the claim immediately, so that if the tick dies mid-execution (gateway kill, OOM, segfault, hard-timeout) the dispatch is not lost. This converts finite one-shot jobs from *at-least-once* to *at-most-times* semantics — a job that self-destructs fires at most ``repeat.times`` times instead of infinitely (issue #38758). Returns ``True`` if the caller may proceed to run the job, ``False`` if the dispatch limit is already reached (in which case the stale job is removed). Only claims jobs with ``schedule.kind == "once"`` and ``repeat.times > 0``. Recurring jobs (they use ``advance_next_run``) and infinite-repeat / no-repeat jobs are left unchanged and always allowed to proceed. rrrrNTrrrrrFrrru>Job '%s': dispatch limit reached (%d/%d) — marking completedrrru5Job '%s': dispatch limit reached (%d/%d) — removingrfz Job '%s': claimed dispatch %d/%du|claim_dispatch: job_id %s not in store — proceeding without claim (handed-in job dict; nothing to persist a claim against)) rrur-r4rpr_inforPrfrJ)rXrlrrrrrs rclaim_dispatchris" >>{{oo5 5 FAs4yF""wwz2&&**622f<< >>>>>>>>WWX&&F >>>>>>>>JJw''E} >>>>>>>> ;22IE!!77=))5 &+C N#.CL)-C &dOOOKKXc(:(:;;!  !?>>>>>>>>H $VH55550555 KGGFCGGD#$6$677  [>>>>>>>>^#,a-F;  dOOO LL2c 2 233{#     o>>>>>>>>r   G    }>>>>>>>>>>>>>>>>>>s9AJ;J J BJ.ArrrrfN)rArrurrr4r/rp)job_idsrrlradvancedrrnew_nexts radvance_next_runsrqS sY g,,C q {{mm%%''  C4y##77:r**..v66D///'J==H H (>(>>>%-M"A    dOOO!sBrrrT)rrurrr4rrrr#r^rr|r/rp) rXr}rlrrexisting claimed_at_agernxts rclaim_job_for_firer s]* !!{{  C4yF""#3'' !!!!!!!!--Cww|,,H  !.x/Ehtn/U/U!V!VJ *,;;==DD4444#444444$-!!!!!!!!.!D'*}}kmm L LC 77:r**..v66D+++&s: HH-),C & dOOOA!!!!!!!!BC!!!!!!!!!!!!!!!!!!sI1F  %F 3ACF F  CF CBF ?F  FFc ddlm}|pi}t|tr|dini}t |dt S#t$rt t cYSwxYw)aResolve the completed one-shot retention window from config. ``cron.completed_retention_days`` (number, default ``COMPLETED_ONESHOT_RETENTION_DAYS``). A non-positive value disables the sweep, retaining completed one-shot records indefinitely. r load_configrcompleted_retention_days)rrrrr4rO COMPLETED_ONESHOT_RETENTION_DAYSr^rrrs r!_completed_oneshot_retention_daysr s 7111111kmm!r*4S$*?*?G37762&&&R LL*,L     7775666667sA&A))B  B raw_jobsc t}|dkrdS|t|z }d}t|D]} |ddkr|d}t |t r|dnd}|d krf|d } t | t s ttj | } n#t$rYwxYw| |kr| |d }|d } |$| r"| t | td |d|d d| |d#t$r5td|d dd YwxYw|S)u{Prune terminal ``state == "completed"`` one-shot records past retention. Mutates *raw_jobs* in place; returns True when anything was removed (the caller persists). Ids removed are added to *removed_ids* when provided so ``save_jobs``'s shrink-merge guard (#80624) allows the intentional delete. Only one-shot (``schedule.kind == "once"``) records in the terminal completed state are candidates; recurring jobs and non-terminal one-shots are never touched. Age is measured from ``last_run_at`` — a completed record without a parseable ``last_run_at`` is kept (never guess a record into deletion). rF)daysrrrrNrrTrzNJob '%s': pruning completed one-shot record (finished %s, retention %.1f days)rrz/Retention sweep skipped malformed job record %rr[)rrrr4rrrrrrr^removerr_rhrJ) rrrretention_dayscutoffremovedrjrrlast_run last_run_dtrids r_sweep_completed_oneshotsr s"788Nu 9.111 1FG8nn""! vvg+--vvj))H+5h+E+EO8<<'''4Dv~~vvm,,Hh,,  +H,B8,L,LMM     f$$ OOB   G&&,,C&3&C))) KK5vrvvdC0011         LLAtS!!        NsIFAF!*F !C/.F/ C<9F;C<< FBF;GGcnt5tcdddS#1swxYwYdS)uGet all jobs that are due to run now. For recurring jobs (cron/interval), if the scheduled time is stale (more than one period in the past, e.g. because the gateway was down OR because a long-running previous execution overran the interval), the accumulated missed runs are collapsed — ``next_run_at`` is fast-forwarded to the next future occurrence so a backlog does NOT burst-fire on restart — but the job still fires ONCE now. This prevents the perpetual-defer loop (#33315) where a job whose runtime exceeds ``interval + grace`` would be skipped forever. Note: firing once on catch-up flows through ``mark_job_run``, so a job with a ``repeat.times`` limit consumes one of its runs on that catch-up fire. N)r_get_due_jobs_lockedr.r/r get_due_jobsr) s{ &&#%%&&&&&&&&&&&&&&&&&&s *..c t}td}t}D]R}|ds;|ddpt jjdd|d<d}Sdtj D}g}|D]1}t|dtsi|d<d}2D]1}t|dtsi|d<d}2|D]}|d }|lt|ts|d dd}G tj|]#t$r|d dd}YwxYwD]}|d }|lt|ts|d dd}G tj|]#t$r|d dd}YwxYw|D]}|d }|.t|ts|d dd}Gt|tr> tj|r#t$r|d dd}YwxYwD]}|d }|.t|ts|d dd}Gt|tr> tj|r#t$r|d dd}YwxYwt!} t#|| rd}fd |D}|D]} | d dst%| r| d} t&d| d| | D]n}|d| krd|d <d|d<|ds||d<|dsd|d<d}| d} | r| diddkro t-tj| d} || z }d|cxkr| krnnvn#t0t2t4f$rYnwxYw| d }|s| di}|d}t7||| d }|rdnd}|s*|dvr&t9||}|r|}|s1|| d <|}t&d| d| dd||D]}|d| dkr ||d <d}ntj|}| di}|d}t-|}|dkr||krt=||rt?||rt9||}|rt&d | d| dd| | |D]}|d| dkr ||d <d}n||krtC|}|dvr||z |krt9||}|rtt&d!| d| dd|||D]}|d| dkr ||d <d}ntE|dkrl| d"}|rT|d#}|d$d}|&|dkr||krtG| dd%rGt&d&| d| dd||Yt&d'| d| dd||D]U}|d| dkrA$||%t|dd}nVtM| |dkrH|tOd(}|| d<D]}|d| dkr ||d<d}n|(| l#t$rIt&)d)| dp| dpdYwxYw|rtU|pd |S)*zLInner implementation of get_due_jobs(); must be called with _jobs_lock held.FrrXNrTc,g|]}t|Sr.)rrs rrz(_get_due_jobs_locked..Q s! D D Dq  " " D D Dr/rrrrcLg|]tfdD S)c3pK|]0}|ddkV1dSrr)rrrs rrz2_get_due_jobs_locked... s<&V&Vrrvvd||quuT{{'B&V&V&V&V&V&Vr/)r)rrrs @rrz(_get_due_jobs_locked.. s<WWWa3&V&V&V&VX&V&V&V#V#VWWWWr/rzsJob '%s' (%s) has pause markers while enabled=true; self-disabling so it cannot fire (pause must be authoritative).rrrrrz+auto-disabled: enabled+paused contradictionr^rrrbrrzone-shot>rrz4Job '%s' had no next_run_at; recovering %s run at %srrzlJob '%s' next_run_at offset changed (%s -> %s). Recomputing cron run to preserve local wall-clock intent: %sz~Job '%s' missed its scheduled time (%s, grace=%ds). Running now; next run provisionally set to: %s (re-anchored on completion)rrrrJuiJob '%s': dispatch limit reached (%d/%d) but its run is still in flight in this process — keeping entryuNJob '%s': one-shot dispatch limit reached (%d/%d) — removing stale due entryrz.Skipping malformed cron job %r during due scan)+rrurAr4rPrrrcopydeepcopyrrrrrr^rWrrr_r}rrr#KeyErrorrPrQrr/rhrrrr*rTrarrrfr|r exceptionrp)r needs_saveintentionally_removedrrlduernrlr_run_claim_ttlrjidexisting_claimrrr-rrrecovered_next recovery_kindraw_next_run_dt next_run_dtrpr'rrrrers @rrr; s --C{{HJ&)eevvd|| vvh--F1A#2#1FBtHJ D DDM(,C,C D D DD C!%% ++T22 AjMJ"&&,,d33 BzNJ & & UU= ! ! >b#&& &mT***! &*2.... &&&EE-...!%JJJ&  & & VVM " " >b#&& &}d+++! &*2.... &&&FF=$///!%JJJ&  " " UU= ! ! >*R"5"5> EE- & & &JJ C  " "&r**** " " "mT***!  " "  " " VVM " " >*R"5"5> FF=$ ' ' 'JJ C  " "&r**** " " "}d+++!  " "455N!311111  *i8Dww}--H% 77:r22||F++"= # 6 6""" /= F $ &-$2F*F*F%5h %P%PN%-(, %%3M") JGGFCGGD#$6$677!"  #B$x3t9,,,:=)%) - '4X>>Owwz2..H<<''D'88K"3&&-osCC'0#FF',HcmmooFFKKWc(:(:;;'1133  '""d8s4y0008B}-)-J!E1c!! /x88///S;5F4U4U4W4WZ_4_4_ 0#--//JJH5 : GGFCGGD#,>,>??$!$"#+&&B!$x3t9444<= 1-1 % 534446>> WWX..F)% & 7 3 3$*JJ{A$>$>  ,yE?Q?Q >#&==??+--HHE',C $&""d8s4y00.3B{O)-J!E1  3      @73774==7C    H  G((=(EFFFF JsE**"FFG22"HH:J"J43J4L,,"MM g3"Cg3(Ag3-AS>=g3>Tg3TBg38Fg3Fg3 B,g3A#g33Aiirc ddlm}|pi}t|tr|dini}t |dt S#t$r t cYSwxYw)zVResolve the per-job output-file retention cap from config (``cron.output_retention``).rrroutput_retention)rrrrr4r_CRON_OUTPUT_DEFAULT_KEEPr^rs r_cron_output_keepr s)111111kmm!r*4S$*?*?G37762&&&R8<< 24MNNOOO )))(((()sA&A))A=<A=rLkeepc^|dkrdS td|dDdd}n#t$rYdSwxYwd}||dD]T} ||dz }#t$r+}td |j|Yd}~Md}~wwxYw|S) aRemove the oldest ``*.md`` run-output files beyond *keep*. Returns count deleted. Mirrors the quick-snapshot retention in ``hermes_cli.backup._prune_quick_snapshots``: output filenames are timestamp-based (``%Y-%m-%d_%H-%M-%S.md``) so a reverse lexical sort orders newest-first, and everything past *keep* is the tail to drop. A non-positive *keep* disables pruning. Pruning failures are swallowed so they can never break output saving. rc3BK|]}||VdSN)is_file)rrEs rrz$_prune_job_output.. s/ C C1qyy{{ CQ C C C C C Cr/z*.mdc|jSr)r)rEs rz#_prune_job_output.. s!&r/T)rreverseNrfz"Failed to prune cron output %s: %s)r,globr{rBr_rJr)rLrfilesdeletedstaleexcs r_prune_job_outputr s qyyq C C++F33 C C C      qqGtuuPP P LLNNN qLGG P P P LL=uz3 O O O O O O O O P Ns'0; A A A55 B*?!B%%B*c tt|}|ddt|t d}||dz }t jt|dd\}} tj |dd 5}| || tj |d d d n #1swxYwYt||t!|n5#t"$r( tj|n#t&$rYnwxYwwxYwt)|t+|S) zSave job output to file.Trz%Y-%m-%d_%H-%M-%Sz.mdr4z.output_r5r9rgrhN)rqrrrrrr:r;rrKr=r>r?r@rrrrArBr{rr)rXr"rL timestamp output_filerCrDrEs rrdrd sMMM$V,,N555 &&':;;I i#4#4#44K#N(;(;FS]^^^LB  Yr3 1 1 1 !Q GGFOOO GGIII HQXXZZ  ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! x---[!!!!   Ih        D  n&7&9&9::: sUD3,AD< D3D  D3D "D33 E%>EE% E E%E  E%rUc"t|pd}|sdS ddlm}||p|}n-#t$r t d|dYnwxYw|dS)uReduce one job skill reference to the bare name the curator matches on. A job may store an absolute path under ``HERMES_HOME/skills`` or an external skills dir; the scheduler resolves those through ``normalize_skill_lookup_name`` before handing them to ``skill_view``. The curator compares this set against bare skill names, so it has to resolve them the same way — otherwise a path-referencing job's skill looks unreferenced and gets archived out from under it. Best-effort: if the resolver is unavailable or rejects the value, fall back to the plain cleanup so a broken import can never lose a name. rJr)normalize_skill_lookup_namez8referenced_skill_names: could not normalize skill ref %rTr[r)rrMagent.skill_utilsrr^r_rJlstrip)rUrrs r_canonical_skill_refr s  rNN " "E r AAAAAA++E22;e     F       ;;==   $ $$s='A'&A'c t}n:#t$r-tddt cYSwxYwt }|D]w}t |t st|d|dD](}t|}|r| |)x|S)uReturn the set of skill names referenced by ANY cron job. Includes paused and disabled jobs deliberately: a paused job never fires, so its skills never get a ``bump_use`` from the scheduler, yet resuming it must still find its skills present. The curator uses this set to protect referenced skills from inactivity archival — a skill a live job depends on is "in use" regardless of when it was last loaded. Names are canonicalized the way the scheduler resolves them at load time, so a job that stores an absolute skill path is protected too. Best-effort: a corrupt/unreadable jobs store returns an empty set rather than raising, so a cron issue can never break the curator. z0referenced_skill_names: failed to load cron jobsTr[rr) rur^r_rJrArrrr4rr)rlnamesrrcleaneds rreferenced_skill_namesr& s{{  GRV WWWuu eeE###t$$  )#'''*:*:CGGH umbrella_skill_name``. pruned: list of skill names that were archived with no forwarding target. Returns a report dict:: { "rewrites": [ { "job_id": ..., "job_name": ..., "before": [...], "after": [...], "mapped": {"old": "new", ...}, "dropped": ["old", ...], }, ... ], "jobs_updated": N, "jobs_scanned": M, } Best-effort: exceptions from loading/saving propagate to the caller so tests can assert behaviour; the curator invocation site wraps this call in a try/except so a failure here never breaks the curator. r)rewrites jobs_updated jobs_scannedFrrNTrr)rXjob_nameraftermappeddroppedz2Curator rewrote skill references in %d cron job(s)) rrAkeysrrurr4rrrpr_rhr) rr pruned_setrlrchangedr skills_beforerr new_skillsrtargets rrewrite_skill_refsrF sd *++LV\r""J#l''))***J F F1EEE 3 3 {{)+" " C1#'''2B2BCGGHDUDUVVM  %'F!#G$&J% , ,<'')$/F#)F4L2& ":":"))&111Z''NN4((((++%%d+++ ' &CM,6@:a==DCLG OO''$--GGFOOcontextmanagerrrDrFrRrTrNrOrWrarcr frozensetr*rrrrrrrrrrr stat_resultrrqrrr rrrrrr*r/r2rFrIrLrOrQrTrYr[r]rbrjrurxrmrnrrrprrrrrrrr LookupErrorr rr&r9r=r@rBrMrRrVrXr`rfrirlrqrsr|rrrrrrrrrrdrrrr.r/rrs """"""!!!!!!   LLLL EEEMMMM FFF((((((((,,,,,,KKKKKKKKKKKKKKKKKKKK  8 $ $******33333333 # htn### $    8_   & & ( (    { " !#55!66  ")/##"9?$$"   $ ?Ij ???j/!:; )Z@@ R_RRRR>  *sDy) * * * * ,T,,,,!%#$#( :499999  e/e/e/V" 4&)) 3C 3D 3 3 3 3 #x}X\]`Xa"T#s(^S#XC34S>c" tCH~ $sCx.    F&4S>&d&&&& c3h D    !T#s(^!!!!!( d     t    # 4# ".1I# d# # # # L"""%c%c%%%%*aSaT#s(^aaaaH$h$8$$$$& 5h 5 5d 5 5 5 5 F FH F F F F F""& 38n #  c] <TcD;;tCH~;HSM;U]^aUb;;;;D Ad At A A A AST$Td8$8E? @(5/ @ @ @ @CCCCC              Fs     x}    $33sDy)93333",4S#X',,,,^Xd4S>&:;2%S# 2F)G&(huS#s]';<((((((.2<<< tCH~ <*S/*< $sCx. <<<<D.2 CCC tCH~ C*S/*C CCCCR.2 L L L tCH~  L*S/* L L L L L (3-B)#))))XNSdW_`cWd*-*- *- *-  *-  8C=(3- '( *-*-*-*-ZDcNuXc]HUXM[cdg[hjn=n7o SM #   SM       B !'+"&"" 48,0!(,$(!%'__ SM__ 3-_ SM _ c] _ T#s(^ $ _ C=_ T#Y _ C=_sm_sm_ SM_5d3i01_tCy)_c]_ !_" ~#_$SM%_&#'_( #s(^)____DCHT#s(^4      K    22$sCx.!922222   d38n1E     {s{T#s(^{c3h8P{{{{|  c 8C= HT#s(^"9 9#c(# 9  9999x&d4S>*&&&&$zd4S>2zzzzD )3))))d##:CF%c%c%%%%6CB.2"&n n 4S>*n T#Y n  #s(^n n n n n n s#AA  A AAA