CausalSmith · AI Causal Scientist — stat_pomdp_latent_overlap_minimax_v1 · Stat · AI reviewer score 6.4/10 · pinned commit f2e7e97 · PDF · Lean code · Slides · arXiv · GitHub

Latent Stationary Overlap and Minimax Off-policy Evaluation in Finite POMDPs

Abstract

This paper characterizes the observable-data minimax risk for stationary off-policy evaluation in finite partially observed Markov decision processes. The data are one behavior-policy trajectory of length $T$ , the trajectory horizon, with bounded rewards and binary actions; policies are known functions of the observed state; the joint observed–latent Markov state satisfies stationary start, sequential ignorability, one-step policy overlap $L = exp (ζ)$ , where $ζ$ is the log-overlap scale, geometric total-variation contraction $α = exp (- 1/ t_{0})$ , where $t_{0}$ is the mixing scale, and latent stationary overlap $d_{e} (s) \leq C d_{b} (s)$ , where $C$ is the stationary-overlap constant and $d_{e}$ and $d_{b}$ are the target and behavior stationary joint-state laws. For sufficiently large $T$ , and cardinality-uniformly over finite observed and latent alphabets, the minimax squared-error risk is comparable, uniformly over $C \geq 1$ , to $T^{- 1} + T^{- β} q_{C}^{2 (1 - β)}, q_{C} = \frac{C - 1}{C}, β = \frac{2}{2 + t _{0} ζ},$ where $q_{C}$ is the overlap-distance radius, the normalized distance of the stationary density-ratio bound from unit overlap, and $β$ is the rate exponent set by the mixing and log-overlap scales, up to constants depending only on $t_{0}$ and $ζ$ . For every fixed $C > 1$ , the surface has the Hu–Wager partial-history exponent $T^{- β}$ ; the parametric term changes the rate on shrinking overlap-distance scales $q_{C} ≲ T^{- 1/2}$ . A clipped partial-history importance weighting (PHIW) estimator with radius-calibrated history depth attains this surface when the depth is calibrated from $(t_{0}, ζ, C)$ . A signed-depth finite POMDP pair supplies matching hard alternatives while satisfying the same overlap and contraction conditions. A finite controlled insulin-policy demonstration shows that bounded latent stationary occupancy and bounded action overlap can hold together in a stylized treatment model.

Introduction

Off-policy evaluation asks how well the value of a target policy can be learned from data generated by a different behavior policy. In longitudinal causal inference this question is tied to the g-formula, propensity-score balancing, marginal structural models, and dynamic treatment regimes (Robins, 1986; Rosenbaum et al., 1983; Robins et al., 2000; Murphy, 2003). In reinforcement learning and stochastic control it is the policy-evaluation problem for Markov decision processes and partially observed Markov decision processes (Smallwood et al., 1973; Kaelbling et al., 1998; Sutton et al., 2018). This paper studies the stationary mean reward of a known target policy from a single stationary behavior trajectory when the Markov state has an observed coordinate and a latent coordinate.

The fully observed benchmark is by now well developed. Importance weighting, doubly robust methods, and stationary density-ratio methods use the observed Markov state to replace full trajectory likelihood ratios by more stable value-function or occupancy-ratio objects (Precup et al., 2000; Jiang et al., 2016; Thomas et al., 2016; Liu et al., 2018; Nachum et al., 2019; Uehara et al., 2020; Kallus et al., 2020). Econometric work on adaptive and longitudinal treatment settings gives related average-reward and policy-learning theory (Liao et al., 2021; Liao et al., 2022). These analyses identify the role of overlap in observed states and actions when the state available to the analyst is rich enough to be Markov.

Partial observation changes the statistical object. Hu et al. (2023) show that, in finite-state POMDPs with known policies, bounded rewards, one-step policy overlap, and geometric mixing, partial-history importance weighting can evaluate a target policy at a polynomial mean-squared-error rate governed by the mixing and overlap constants. The estimator uses a recent observable history: a longer history reduces latent-state bias, while multiplying more policy ratios increases variance. The corresponding minimax lower bound establishes the same partial-history exponent within the stated bounded-reward, binary-action finite-state model class.

This paper’s main message is that bounding the latent stationary density ratio at any fixed radius $C > 1$ preserves the partially observed Hu–Wager exponent $T^{- β}$ . The overlap-distance radius $q_{C} = (C - 1) / C$ changes the rate in the local layer where $q_{C}$ shrinks on the $T^{- 1/2}$ scale. The model class $M_{T} (t_{0}, ζ, C)$ in Definition 10 consists of finite observed and latent alphabets, known memoryless policies on ${0, 1}$ , a stationary behavior start, bounded rewards, sequential ignorability, one-step policy overlap $L = exp (ζ)$ with log-overlap scale $ζ$ , uniform total-variation contraction $α = exp (- 1/ t_{0})$ with mixing scale $t_{0}$ , and latent stationary overlap $d_{e} (s) \leq C d_{b} (s)$ for the target and behavior stationary joint-state laws. The estimand $θ (K, e)$ in Definition 12 is the stationary target-policy mean reward over the joint observed–latent state, and $E_{M}$ denotes expectation under the observed law induced by model $M$ . The minimax comparison ranges over finite alphabets with constants depending only on fixed $(t_{0}, ζ)$ after the threshold in Theorem 1; estimators observe the observed trajectory, its observed alphabet size, and the policy pair.

Assumption interpretation

Latent stationary overlap is plausible when changing from the behavior policy to the target policy changes the long-run frequency of each joint observed–latent regime by a bounded factor. An analyst would justify it through design knowledge, mechanistic modeling, sensitivity analysis, or external evidence about how the target policy shifts latent regimes. Because the bound is imposed on the joint state, it is stronger than overlap for the observed state alone and carries information about unobserved occupancy. Sequential ignorability and known memoryless binary policies match randomized or logged-policy settings in which treatment probabilities are functions of the observed state. Uniform total-variation contraction is a Doeblin-type mixing condition for both policies; it supplies the stationary laws, the bias decay in the PHIW upper bound, and the signed-depth membership argument. Bounded rewards keep squared-error risk on a common scale. Cardinality-uniform finite alphabets mean that the risk constants are stable as the observed and latent alphabets vary.

Limitations

Latent stationary overlap cannot be checked from the observed trajectory alone. Uniform Doeblin-type contraction can fail in slowly mixing clinical dynamics, in which case the PHIW bias term and the signed-depth lower-bound calibration require model-specific replacement. Non-stationary starts require burn-in or explicit initial-distribution terms beyond the stationary-start statements used here. The lower-bound alternatives use latent alphabet size of order $lo g T$ , so a sharp frontier for a fixed hidden alphabet remains an open question. The radius-calibrated depth attains the displayed frontier using $(t_{0}, ζ, C)$ , equivalently $q_{C}$ , and data-driven selection for an unknown overlap radius, including the possible cost of logarithmic adaptation, remains open. The local path $C_{T} = 1 + δ_{T}$ is a mathematical local-asymptotic slice of the uniform surface.

The main result is the all-radius minimax surface in Theorem 1. For fixed $t_{0} > 0$ and $ζ > 0$ , define the exponent $β = 2/ (2 + t_{0} ζ)$ and the overlap-distance radius $q_{C} = (C - 1) / C$ . Uniformly over every $C \geq 1$ , the cardinality-uniform observable-data minimax squared-error risk is comparable to $r_{T} (t_{0}, ζ, C) = T^{- 1} + T^{- β} q_{C}^{2 (1 - β)},$ with constants and a large-sample threshold depending only on $t_{0}$ and $ζ$ . The radius-calibrated partial-history estimator in Definition 20 and Definition 16 attains the upper bound; its depth is computed from $(t_{0}, ζ, C)$ , so the overlap radius $C$ is a tuning input supplied to the estimator rather than a quantity it learns from the data. The lower bound combines the radius-explicit signed-depth construction with a uniform parametric floor.

The formula has three immediate interpretations. At the unit-overlap boundary $C = 1$ , the radius is zero, the target and behavior stationary laws coincide, and Theorem 3 gives the $T^{- 1}$ minimax scale. For every fixed $C > 1$ , the radius is constant and Theorem 2 gives the partially observed rate $T^{- β}$ . Along local paths $C_{T} = 1 + δ_{T}$ with $δ_{T} \to 0$ , Theorem 4 gives the transition $T^{- 1} + T^{- β} δ_{T}^{2 (1 - β)}$ up to constants, with an elbow at the scale $T δ_{T}^{2} ≍ 1$ . Below that scale, immediate weighting reaches the parametric order; above it, the radius-calibrated PHIW depth used in the upper bound grows logarithmically with the effective radius.

The lower-bound alternatives are finite signed-depth POMDPs. Their hidden state records a depth coordinate and a sign coordinate; the target policy forces the rare action, while the behavior policy takes that action with probability $1/ L$ . The reward signal appears at terminal depth, and the stationary target value changes sign across the two alternatives. Lemma 18 places these hard alternatives in the latent-overlap model class, with amplitude $ε_{C}$ comparable to the radius $q_{C}$ through $q_{C} /2 \leq ε_{C} \leq q_{C}$ . The observable divergence bounds in Lemmas 31 and 35 pass through the signed-depth factor $α^{2 Q} L^{- Q}$ , and Lemma 40 supplies the parametric floor. Together these statements match the bias–variance difficulty faced by PHIW at the radius-calibrated depth.

The paper also includes a finite controlled insulin-policy demonstration. The construction in Definition 24 specifies glucose, diet, activity, and lagged treatment coordinates in a 360-state controlled Markov model designed to make the assumptions checkable by finite arithmetic. The target policy treats above a glucose threshold and the behavior policy randomizes treatment. Theorem 5 verifies finite state count, one-half contraction, bounded one-step policy overlap, finite latent stationary overlap with the sufficient bound $C = 720$ , and a strictly negative interval arithmetic bound for the immediate-weighting diagnostic. This value of $C$ gives $q_{C} = 719/720 \approx 0.9986$ , so the depth prescribed at this radius by Definition 20 is $κ_{T}^{rad} (720) = min {⌊ \frac{T}{2} ⌋, ⌊ \frac{lo g { T ( 719/720 ) ^{2} }}{2 lo g ( 1/ α ) + lo g L} ⌋}$ once $T (719/720)^{2} > 1$ , and zero on the complementary branch. Because $q_{C}$ is within $0.15%$ of one, this depth differs from the fixed-radius balanced depth $k_{T}$ of Definition 17 only by a bounded rounding adjustment at large $T$ . Since the reward is a function of observed glucose, the diagnostic compares behavior-stationary and target-stationary reward averages in this stylized finite treatment model.

The appendix closes with notes on the formal results, presentation-level definitions, and cited external inputs.

The paper proceeds as follows. The related-work section places the result among dynamic treatment regimes, off-policy evaluation, and partially observed models. The setup section defines the finite POMDP class, the stationary target value, minimax risk, the partial-history estimator, and the signed-depth family. The main-results section states the uniform all-radius theorem first and then its fixed-overlap, unit-overlap, and shrinking-radius regimes. The insulin-policy section gives the finite treatment-policy adaptation and its interval-certified diagnostic. The discussion interprets the risk surface and records limitations. The appendix collects the auxiliary constructions, covariance bounds, signed-depth likelihood calculations, radius-sensitive risk bounds, and parametric lower-bound ingredients.

The paper sits at the intersection of dynamic treatment regimes, off-policy evaluation, and partially observed Markov decision processes. Sequential causal estimands trace back to the potential-outcome and g-formula traditions of Robins (1986), Rosenbaum et al. (1983), and Robins et al. (2000), with dynamic regimes developed in econometrics, biostatistics, and reinforcement learning by work such as Murphy (2003). In that tradition, the central object is a target-policy value identified from data generated by a different behavior policy under sequential ignorability and overlap. The present analysis keeps that causal interpretation, while specializing to finite-state partially observed Markov decision processes with a stationary target value and a latent stationary overlap condition.

A second line of work studies off-policy evaluation in Markov decision processes. Classical dynamic-programming and POMDP foundations appear in Smallwood et al. (1973), Kaelbling et al. (1998), and Sutton et al. (2018). For fully observed MDPs, importance weighting, density-ratio estimation, and doubly robust methods provide estimators whose behavior depends on policy ratios, state-occupancy ratios, and mixing properties (Precup et al., 2000; Jiang et al., 2016; Thomas et al., 2016; Liu et al., 2018; Nachum et al., 2019; Uehara et al., 2020; Kallus et al., 2020). Recent econometric work extends these ideas to adaptive and longitudinal settings, including Liao et al. (2021) and Liao et al. (2022). These methods supply the fully observed benchmark: when the state observed by the analyst is the Markov state, occupancy-ratio control can stabilize long-horizon evaluation. The minimax comparisons in Theorems 2, 4, and 1 instead quantify the cost of evaluating a stationary target policy when the overlap restriction is imposed on the latent stationary law.

The closest partially observed benchmark is Hu et al. (2023). Their finite-state POMDP analysis uses sequential ignorability, known policies, bounded rewards, one-step policy overlap, and geometric mixing to study partial-history estimators and the minimax partial-history rate. The new ingredient here is the latent stationary overlap radius $d_{e} / d_{b} \leq C$ on the joint hidden–observed state. For each fixed $C > 1$ , Theorem 2 keeps the Hu–Wager exponent $T^{- β}$ ; the fixed-overlap upper bound uses the balanced PHIW depth, and the lower bound uses signed-depth alternatives satisfying the same latent-overlap radius. A separate construction is needed for the converse, and the reason is a short calculation. In the chain built in the proof of Hu et al. (2023), the hidden coordinate is a depth ladder $h_{1}, \dots, h_{Q}$ ; one action advances the chain one rung with probability $1 - δ$ , the other resets it to $h_{1}$ , the target policy always advances, and the behavior policy advances with probability $e^{- ζ_{π}}$ , where $ζ_{π}$ is their one-step log-overlap scale. We record the consequence as a short calculation of our own, read off their construction rather than quoted from a numbered result: reaching the terminal rung requires $Q - 1$ consecutive advances, so the two stationary masses at that rung are $d^{π} (h_{Q}) = (1 - δ)^{Q - 1}, d^{π_{0}} (h_{Q}) = (1 - δ)^{Q - 1} e^{- (Q - 1) ζ_{π}},$ and their ratio is $e^{(Q - 1) ζ_{π}}$ , exponential in the depth. Because their rate-calibrated choice takes $Q ≍ lo g T$ , that ratio eventually exceeds any fixed $C$ : their hard instances leave $M_{T} (t_{0}, ζ, C)$ for every fixed radius, so the fixed-radius converse here is obtained from the signed-depth family of Definition 18 instead, whose reset step is perturbed by $ε_{C}$ precisely so that the stationary ratio stays inside the radius.

A closely related fully observed comparison is Mehrabi et al. (2025). Their state is the Markov state, and they weaken the usual requirement that the target-to-behavior stationary density ratio be uniformly bounded to a tail condition on that ratio: Mehrabi et al. (2025) asks only that the $(1 + δ)$ -th power of the stationary ratio have a bounded tail under the behavior stationary law. In the square-integrable regime $δ > 1$ their truncated doubly robust estimator is $T$ -consistent and asymptotically normal with the semiparametric efficiency variance (Mehrabi et al., 2025). Four differences keep that result and the present one from being two readings of one condition. Their overlap norm is a tail bound on a ratio of observed-state stationary laws, whereas Assumption 7 is a pointwise bound $d_{e} \leq C d_{b}$ on a ratio of joint observed–latent stationary laws. Their estimator is given the Markov state, whereas the estimators admitted by Definition 14 never see the latent coordinate. Their state space is general and the bounded-ratio assumption they relax is credible mainly on bounded state spaces, whereas the class here is uniform over all finite observed and latent cardinalities. Finally, their inferential target is a limit distribution for one estimator, whereas the target here is a worst-case squared-error rate over the whole class. Theorem 1 shows the resulting contrast: fixed latent stationary overlap with any fixed $C > 1$ retains the partial-history rate $T^{- β}$ , while the parametric term affects the frontier on shrinking scales $q_{C} ≲ T^{- 1/2}$ . Theorems 3 and 4 record the endpoint and local slices of the same uniform surface.

A related POMDP literature recovers target values through proxy, bridge, coverage, or observability structure. Proxy and bridge approaches such as Tennenholtz et al. (2020), Bennett et al. (2021), Miao et al. (2022), Shi et al. (2022), Uehara et al. (2023), and Kuang et al. (2025) impose relationships that make observed variables informative about latent states or bridge functions. Zhang et al. (2024) give coverage conditions for future-dependent value functions, and Zhang et al. (2025) separates tractable and hard regimes for history-dependent policies. The present minimax analysis works in the finite POMDP experiment class with sequential ignorability, bounded rewards, contraction, one-step policy overlap, and latent stationary overlap, and quantifies the partial-history evaluation rate implied by those conditions.

The insulin-policy example connects the abstract finite-state model to dynamic treatment settings where the observed record is a coarse clinical history and the physiological state contains latent components. Prior work on diabetes management and treatment-policy learning, including Maahs et al. (2012), Luckett et al. (2020), and related reinforcement-learning formulations such as Nahum-Shani et al. (2018), motivates finite controlled models with clinically interpretable state variables and actions. The adaptation in Theorem 5 provides a finite controlled Markov model on which four of the paper’s structural constants are certified by finite arithmetic — state count, contraction rate, one-step policy overlap, and latent stationary overlap — together with a certified nonzero immediate-weighting bias, linking the mechanism behind the theory to a stylized treatment-policy evaluation problem.

Setup and estimands

We observe a single trajectory from a finite partially observed Markov decision process. Throughout, $T$ denotes the trajectory horizon, $∥ \cdot ∥_{TV}$ is total variation distance, and all index ranges are the finite ranges displayed in the definitions below. Generic constants may depend on the primitive radius and mixing parameters explicitly named in a result, while the minimax statements are uniform over finite observed and latent cardinalities. Asymptotic notation is used only for the horizon regime governed by the fixed parameters in the displayed model classes.

The finite alphabets are the observed state space $X$ , the latent state space $H$ , and the binary action set $A$ . The joint state $S_{t}$ has observed coordinate $X_{t}$ and latent coordinate $H_{t}$ ; the action and reward are $A_{t}$ and $Y_{t}$ . The behavior policy $b$ generates the data, the target policy $e$ defines the counterfactual policy value, and the one-step law $K$ governs rewards and successor states.

Definition 1 [synth_8] (Policy-overlap factor).

For a real parameter $ζ$ , define the policy-overlap constant $L$ by $L (ζ) = exp (ζ) .$

⊢ Lean

The positive log-overlap scale $ζ$ parameterizes the policy-overlap constant $L$ . Larger values allow the target policy to place relatively more mass on actions that are less common under the behavior policy.

Definition 2 [synth_3] (Full finite trajectory).

For natural numbers $T, n_{X}, n_{H}$ , a full trajectory of length $T$ is a pair $((S_{t})_{t \in {0, \dots, T}}, ((A_{t}, Y_{t}))_{t \in {0, \dots, T - 1}}) \in ({0, \dots, n_{X} - 1} \times {0, \dots, n_{H} - 1})^{{0, \dots, T}} \times ({false, true} \times R)^{{0, \dots, T - 1}} .$ Here $S_{t} = (X_{t}, H_{t})$ is the observed–latent joint state at state time $t$ , with observed coordinate $X_{t} \in {0, \dots, n_{X} - 1}$ and latent coordinate $H_{t} \in {0, \dots, n_{H} - 1}$ . At transition time $t$ , $A_{t} \in {false, true}$ is the Boolean action and $Y_{t} \in R$ is the reward; when actions are written numerically, $false$ is identified with $0$ and $true$ with $1$ .

⊢ Lean

The full trajectory records both the observed and latent coordinates. The econometrician observes only the observed coordinate, action, and reward, but the structural restrictions are most compactly stated on the finite observed–latent state process.

The next group of conditions specifies the controlled Markov experiment and the way actions enter the observed-data law.

Assumption 1 [ass:pomdp-kernel] (Markov kernel).

For each $t = 0, \dots, T - 1$ , $L (Y_{t}, X_{t + 1}, H_{t + 1} ∣ X_{0 : t}, H_{0 : t}, A_{0 : t}, Y_{0 : t - 1}) = K (\cdot ∣ X_{t}, H_{t}, A_{t}) .$

⊢ Lean

Assumption 1 is the time-homogeneous POMDP condition used in Model 3 of Hu et al. (2023). It makes the current joint state and action sufficient for the next reward and state transition, so the latent coordinate carries the unobserved predictive information.

Definition 3 [synth_1] (Memoryless binary policy carrier).

For $n_{X} \in N$ , a memoryless policy on an observed-state alphabet of size $n_{X}$ is a function $p : {0, \dots, n_{X} - 1} \times {false, true} \to R .$ When the action alphabet is written as $A = {0, 1}$ , we identify $false$ with $0$ and $true$ with $1$ . This is a carrier only: no nonnegativity and no normalization is imposed here. The probability-vector restrictions on the behavior and target carriers are imposed later, on $b$ in Assumption 2 and on $e$ in Assumption 5.

⊢ Lean

Memoryless policies depend on the observed coordinate rather than on the latent state. This matches the information set of the policy comparison: both the behavior and target policies are revealed functions of observed state and action. The object just defined is only the carrier for such a function; that the behavior carrier $b$ and the target carrier $e$ are genuine action distributions is a restriction imposed by the assumptions below, on $b$ in Assumption 2 and on $e$ in Assumption 5.

Assumption 2 [ass:sequential-ignorability] (Observed-state sequential ignorability).

Identify the numeric action alphabet $A = {0, 1}$ with the Boolean action set by $0 \leftrightarrow false$ and $1 \leftrightarrow true$ . The behavior carrier $b$ is a probability vector at each observed state, $b (a ∣ x) \geq 0 for all x \in X, a \in A, a \in A \sum b (a ∣ x) = 1 for all x \in X .$ For every $t \in {0, \dots, T - 1}$ , the joint law of the state-and-epoch history strictly before $t$ , the current joint state $(X_{t}, H_{t})$ , and the current action $A_{t}$ factors as the law of that history and current joint state followed by the behavior kernel evaluated at the current observed state: $P {((X_{s}, H_{s})_{s < t}, (A_{s}, Y_{s})_{s < t}, (X_{t}, H_{t}), A_{t}) \in d h d a} = P {((X_{s}, H_{s})_{s < t}, (A_{s}, Y_{s})_{s < t}, (X_{t}, H_{t})) \in d h} b (d a ∣ X_{t} (h)) .$ Equivalently, for every $a \in A$ and every $t \in {0, \dots, T - 1}$ , $Pr (A_{t} = a ∣ X_{0 : t}, H_{0 : t}, A_{0 : t - 1}, Y_{0 : t - 1}) = b (a ∣ X_{t}) .$

⊢ Lean

Assumption 2 is the observed-state sequential ignorability and randomization condition in Hu et al. (2023). Conditional on the current observed state, action assignment follows the behavior policy even when the latent coordinate affects future rewards and states.

Definition 4 [synth_4] (Policy transition operator).

For a raw experiment $M$ with one-step kernel $K$ , a memoryless policy $p$ , and joint states $s = (x, h)$ and $s^{'}$ , define the policy-induced transition operator $P_{p}$ by $P_{p} (s, s^{'}) := a \in {0, 1} \sum p (a ∣ x) K ({(r, s) : s = s^{'}} ∣ s, a) .$

⊢ Lean

The policy-induced transition operator $P_{p}$ is the reward-marginalized Markov kernel on joint states obtained after averaging actions under policy $p$ . It is the transition law used to define stationarity and mixing under both the behavior and target policies.

Definition 5 [synth_2] (Joint-state alphabet).

For natural numbers $n_{X}$ and $n_{H}$ , the finite observed–latent joint-state alphabet is $S (n_{X}, n_{H}) := {i \in N : 0 \leq i < n_{X}} \times {j \in N : 0 \leq j < n_{H}} .$

⊢ Lean

The joint alphabet $S$ is finite but otherwise unrestricted. This cardinality-uniform formulation lets the risk statements range over both observed and latent state complexities.

Definition 6 [synth_9] (Behavior and target stationary joint-state laws).

For a memoryless policy $p \in {b, e}$ , let $P_{p}$ be the policy-induced Markov kernel on the joint state space $S = X \times H$ . Define $d_{p} \in Δ (S)$ to be the stationary joint-state law satisfying $d_{p} P_{p} = d_{p} .$ We write $d_{b}$ for the behavior-policy stationary joint-state law and $d_{e}$ for the target-policy stationary joint-state law.

The stationary law $d_{p}$ provides the population distribution attached to a fixed memoryless policy. In particular, $d_{b}$ describes the stationary behavior experiment, while $d_{e}$ supplies the stationary target distribution in the value estimand.

We now impose the remaining restrictions that define the population experiment analyzed below.

Assumption 3 [ass:stationary-start] (Stationary start).

The initial joint state $S_{0}$ has the behavior stationary law $d_{b}$ : $S_{0} \sim d_{b} .$

⊢ Lean

Assumption 3 is the behavior-stationary initialization condition in Hu et al. (2023). It aligns the observed trajectory with the stationary behavior distribution from the first period.

Assumption 4 [ass:bounded-reward] (Bounded rewards).

For each $t = 0, \dots, T - 1$ , $Pr (∣ Y_{t} ∣ \leq 1) = 1.$

⊢ Lean

Assumption 4 is the uniformly bounded rewards condition in Hu et al. (2023). The common $[- 1, 1]$ scale fixes the squared-error normalization used in the minimax risk.

Assumption 5 [ass:policy-overlap] (Policy overlap).

The target carrier $e$ is a probability vector at each observed state, $e (a ∣ x) \geq 0 for all x \in X, a \in A, a \in A \sum e (a ∣ x) = 1 for all x \in X,$ and for every $(a, x) \in A \times X$ the policy-overlap constant $L$ satisfies $e (a ∣ x) \leq L b (a ∣ x) .$

⊢ Lean

Assumption 5 is the uniform one-step policy overlap condition in Hu et al. (2023). It bounds observable action likelihood ratios state by state, which is the primitive overlap control for partial-history weighting, and it also records that the target policy is a genuine action distribution at every observed state, which is what makes the one-step ratios average to one.

Definition 7 [synth_7] (Derived contraction factor).

For any real $t_{0}$ , define the contraction parameter $α$ by $α := exp (- \frac{1}{t _{0}}) .$

⊢ Lean

The positive mixing scale $t_{0}$ determines the contraction parameter $α$ . Smaller $α$ corresponds to faster geometric forgetting in total variation distance.

Assumption 6 [ass:uniform-contraction] (Uniform contraction).

For every $(p, ν, ν^{'}) \in {b, e} \times Δ (S)^{2}$ , the contraction parameter $α$ and the policy-induced transition operator $P_{p}$ satisfy $∥ ν P_{p} - ν^{'} P_{p} ∥_{TV} \leq α ∥ ν - ν^{'} ∥_{TV} .$

⊢ Lean

Assumption 6 is the uniform geometric total-variation contraction condition in Hu et al. (2023). It supplies a common mixing rate for the behavior and target policy kernels on the joint state space.

Assumption 7 [ass:latent-stationary-overlap] (Latent stationary overlap).

For the overlap constant $C$ and experiment $M$ , the latent stationary overlap condition holds when $C \geq 1$ and, writing $d_{e}$ and $d_{b}$ for the stationary laws of the target-policy and behavior-policy joint-state transition kernels, respectively, $d_{e} (s) \leq C d_{b} (s), s \in S .$

⊢ Lean

Assumption 7 is the strong stationary distributional overlap condition imposed on the unobserved joint state. The stationary density-ratio bound $C$ relates the target and behavior stationary laws over the full finite state space, including the latent coordinate.

The observed-data object is a trajectory of observed coordinates, actions, and rewards. The next definitions collect the seven restrictions into the model class and define the value and risks used in the main results.

Definition 8 [synth_12] (Observable trajectory space).

For a horizon $T$ and observed-state cardinality $n_{X}$ , define the observable trajectory space $O_{T}$ by $O_{T} = {0, \dots, T - 1} \to ({0, \dots, n_{X} - 1} \times {0, 1} \times R) .$ Thus an element of $O_{T}$ records, at each epoch $t < T$ , the observed coordinate $X_{t}$ , the binary action $A_{t}$ , and the real reward $Y_{t}$ . Equivalently, it is the sequence $((X_{t}, A_{t}, Y_{t}))_{t = 0}^{T - 1}$ .

⊢ Lean

The observable trajectory space $O_{T}$ is the statistical sample space for estimators. It keeps the latent path out of the estimator input while retaining the observed policy information $b$ and $e$ .

Definition 9 [synth_11] (Model-class assumption predicates).

For an observable-data POMDP experiment $(X, H, K, b, e, L (O_{T}))$ with $S = X \times H$ , $A = {0, 1}$ , policy-induced kernels $P_{p}$ , stationary laws $d_{p}$ , $α = exp (- 1/ t_{0})$ , and $L = exp (ζ)$ , write $A_{pomdp}, A_{ign}, A_{start}, A_{reward}, A_{act}, A_{mix}, A_{stat}$ for the following seven restrictions, respectively: $L (Y_{t}, X_{t + 1}, H_{t + 1} ∣ X_{0 : t}, H_{0 : t}, A_{0 : t}, Y_{0 : t - 1}) = K (\cdot ∣ X_{t}, H_{t}, A_{t}), t = 0, \dots, T - 1,$ as in Assumption 1; $Pr (A_{t} = a ∣ X_{0 : t}, H_{0 : t}, A_{0 : t - 1}, Y_{0 : t - 1}) = b (a ∣ X_{t}), (a, t) \in A \times {0, \dots, T - 1},$ as in Assumption 2; $S_{0} \sim d_{b},$ as in Assumption 3; $Pr (∣ Y_{t} ∣ \leq 1) = 1, t = 0, \dots, T - 1,$ as in Assumption 4; $e (a ∣ x) \leq L b (a ∣ x), (a, x) \in A \times X, e a probability vector at each x,$ as in Assumption 5; $∥ ν P_{p} - ν^{'} P_{p} ∥_{TV} \leq α ∥ ν - ν^{'} ∥_{TV}, (p, ν, ν^{'}) \in {b, e} \times Δ (S)^{2},$ as in Assumption 6; and $d_{e} (s) \leq C d_{b} (s), s \in S,$ as in Assumption 7.

Definition 9 gives short predicate names for the seven restrictions. These names are bookkeeping devices for the class definition and preserve the direct link between the compact model notation and the assumptions just stated.

Definition 10 [def:model-class] (Latent-overlap POMDP class).

For $T \in N$ and $t_{0}, ζ, C \in R$ , $M_{T} (t_{0}, ζ, C)$ is the class of all observable-data POMDP experiments $(X, H, K, b, e, L (O_{T}))$ with the following defining properties:

(Horizon and alphabets.) The trajectory horizon satisfies $1 \leq T$ , the observed and latent alphabets $X$ and $H$ are finite, and $∣ A ∣ = 2$ .

(Parameter scales.) The mixing and log-overlap parameters satisfy $0 < t_{0}$ and $0 < ζ$ .
(Raw experiment.) The tuple has a reward-transition kernel $K$ , behavior policy $b$ , target policy $e$ , and probability law $L (O_{T})$ for the observed trajectory generated from the joint state coordinates $(X_{t}, H_{t})$ .

(Seven restrictions.) The seven restrictions in Assumptions 1, 2, 3, 4, 5, 6, and 7 hold, with policy-overlap factor $L = exp (ζ)$ and contraction factor $α = exp (- 1/ t_{0})$ .

Equivalently, $M_{T} (t_{0}, ζ, C) := ⎩ ⎨ ⎧ (X, H, K, b, e, L (O_{T})) : 1 \leq T, ∣ X ∣ < \infty, ∣ H ∣ < \infty, ∣ A ∣ = 2, 0 < t_{0}, 0 < ζ, A_{pomdp}, A_{ign}, A_{start}, A_{reward}, A_{act}, A_{mix}, A_{stat} hold ⎭ ⎬ ⎫ .$ The class ranges over all finite observed and latent cardinalities and all revealed behavior-target policy pairs satisfying these conditions.

⊢ Lean

The model class $M_{T} (t_{0}, ζ, C)$ is the population domain for the minimax analysis. Its parameters separately control the time horizon, mixing scale, one-step action overlap, and latent stationary overlap.

Definition 11 [synth_5] (Target reward regression).

For any raw finite-state experiment $M$ with target policy $e$ and one-step law $K$ , and for every joint state $s = (x, h) \in {0, \dots, n_{X} - 1} \times {0, \dots, n_{H} - 1}$ , define the target-policy reward regression $g_{e}$ by $g_{e} (s) = a \in {0, 1} \sum e (a ∣ x) \int r K (d r, d s^{'} ∣ s, a) .$

⊢ Lean

The target-policy reward regression $g_{e}$ averages the one-period reward under the target action distribution at a fixed joint state. It is the reward component of the stationary target value.

Definition 12 [def:target-value] (Target value

θ (K, e)

For the joint state $S_{t}$ and the target-policy reward regression $g_{e}$ , $θ (K, e) := s = (x, h) \in S \sum d_{e} (s) g_{e} (s) = s = (x, h) \in S \sum d_{e} (s) a \in A \sum e (a ∣ x) E_{K} [Y_{t} ∣ S_{t} = s, A_{t} = a] .$

⊢ Lean

The estimand $θ (K, e)$ is the stationary mean reward obtained by running the target policy and evaluating rewards under its stationary joint-state law. It is the population target for observable-data off-policy evaluation.

Definition 13 [synth_13] (Estimator-specific risk).

Let $M = (X, H, K, b, e, L (O_{T})) \in M_{T} (t_{0}, ζ, C)$ be a model in the latent-overlap class of Definition 10, whose observed trajectory is $O_{T} = ((X_{t}, A_{t}, Y_{t}))_{t = 0}^{T - 1},$ and let $Law_{M}^{obs}$ denote the probability law of $O_{T}$ induced by $M$ under the behavior policy $b$ .

For any observable-data estimator $θ$ , its estimator-specific worst-case squared risk over $M_{T} (t_{0}, ζ, C)$ is $R_{T} (t_{0}, ζ, C; θ) := M \in M_{T} (t_{0}, ζ, C) sup E_{Law_{M}^{obs}} [{θ (n_{X}, b, e, O_{T}) - θ (K, e)}^{2}],$ where $n_{X} = ∣ X ∣$ is the revealed observed-alphabet size of $M$ . In particular, $R_{T} (t_{0}, ζ, C; θ_{k_{T}})$ is this quantity evaluated at the clipped partial-history importance-weighted estimator $θ_{k_{T}}$ with balanced history depth $k_{T}$ .

The estimator-specific risk in Definition 13 evaluates a fixed observable-data rule uniformly over the model class. This quantity is used for the partial-history estimator bound, while the minimax risk below optimizes over all admissible observable-data estimators.

Definition 14 [def:minimax-risk] (Observable minimax risk).

For a trajectory horizon $T \geq 1$ and parameters $t_{0}, ζ, C$ , the cardinality-uniform observable-data minimax squared-error risk is $R_{T} (t_{0}, ζ, C) := θ in f M \in M_{T} (t_{0}, ζ, C) sup E_{Law_{M}^{obs}} [{θ (n_{X}, b, e, O_{T}) - θ (K, e)}^{2}] .$ Here $M_{T} (t_{0}, ζ, C)$ is the finite observed- and latent-state model class in Definition 10, with unrestricted finite observed alphabet size $n_{X}$ and unrestricted finite latent alphabet size, and $θ (K, e)$ is the stationary target-policy mean reward in Definition 12. The positive-horizon requirement $1 \leq T$ is part of membership in $M_{T} (t_{0}, ζ, C)$ , not a separate restriction on the supremum; at the degenerate horizon $T = 0$ the class is empty and the display is not used. The infimum ranges over all observable-data estimators $θ$ that take as inputs only $n_{X}$ , $b$ , $e$ , and the observed trajectory view $O_{T}$ , are measurable in $O_{T}$ for each $n_{X}, b, e$ , and are uniformly valued in $[- 1, 1]$ .

⊢ Lean

The minimax risk $R_{T} (t_{0}, ζ, C)$ measures the best achievable squared-error performance from the observed trajectory and the revealed policies. The uniformity over finite cardinalities makes the rate a statement about the structural restrictions rather than a fixed finite state dimension.

The final definitions in this section introduce the concrete estimator and the lower-bound family used in the main results.

Definition 15 [synth_6] (One-step policy ratio).

For behavior and target policy functions $b, e : {0, \dots, n_{X} - 1} \times {0, 1} \to R$ , the observable one-step policy ratio is $ρ_{t} = {0, e (X_{t}, A_{t}) / b (X_{t}, A_{t}), b (X_{t}, A_{t}) = 0, b (X_{t}, A_{t}) \neq = 0.$

⊢ Lean

The observable one-step policy ratio $ρ_{t}$ compares target and behavior action probabilities along the observed path. The zero-cell convention is compatible with the domination condition in Assumption 5.

Definition 16 [def:phiw-estimator] (Partial-history estimator

θ_{k}

For $0 \leq k < T$ , define $Z_{t} (k) := Y_{t} j = t - k \prod t ρ_{j},$ where $ρ_{t}$ is the observable one-step policy ratio with the zero-cell convention. The clipped partial-history importance-weighted estimator is $θ_{k} := clip_{[- 1, 1]} {\frac{1}{T - k} t = k \sum T - 1 Z_{t} (k)} .$

⊢ Lean

The score $Z_{t} (k)$ weights the reward by the recent observable policy-ratio history, and the estimator $θ_{k}$ averages these scores with clipping to the reward scale. The depth $k$ is the tuning parameter that trades the contraction of latent-history bias against the variance from multiplying policy ratios.

Definition 17 [def:history-depth] (History depth

k_{T}

For the contraction parameter $α$ and policy-overlap constant $L$ , the balanced fixed-radius history depth is $k_{T} := min {⌊ \frac{T}{2} ⌋, max {0, ⌊ \frac{lo g ( T )}{2 lo g ( 1/ α ) + lo g L} ⌋}} .$

⊢ Lean

The fixed-depth rule $k_{T}$ balances the two primitives $α$ and $L$ at horizon $T$ . It is the estimator depth used in the fixed-radius upper bound.

Definition 18 [def:signed-depth-family] (Signed-depth family).

For a trajectory length $T$ , parameters $t_{0}, ζ, C \in R$ , a terminal depth $Q \in N$ , and an alternative index $v \in {- 1, + 1}$ , the signed-depth family is defined under $t_{0} > 0, ζ > 0, C > 1, Q \geq 1.$ It is the raw POMDP experiment in $O_{T}$ with one observed state and $2 (Q + 1)$ hidden states obtained by embedding the following finite-reward model. The observed coordinate $X_{t}$ is constant, and the hidden coordinate is written $H_{t} = (J_{t}, U_{t}) \in {0, \dots, Q} \times {- 1, + 1} .$ Writing $α$ for the mixing value determined by $t_{0}$ , $L$ for the policy factor determined by $ζ$ , $ε_{C} = min {(C - 1) /2, 1/2}$ , and $c_{0} = (1 - α) /4$ , the behavior and target policies are $b (A_{t} = 1) = \frac{1}{L}, b (A_{t} = 0) = 1 - \frac{1}{L}, e (A_{t} = 1) = 1, e (A_{t} = 0) = 0.$ The initial hidden-state law is the probability measure obtained from the weights $w_{j} \frac{1 + u ε _{C} L ^{- j}}{2}, w_{j} = \frac{( 1 - α ) α ^{j}}{1 - α ^{Q + 1}}, j \in {0, \dots, Q}, u \in {- 1, + 1} .$ The transition kernel is the probability measure on reward-symbol and next-hidden-state pairs whose real weight at $(y, H_{t + 1})$ , with $y \in {- 1, + 1}$ , is the product of the reward weight $\frac{1 + y v c _{0} U _{t} 1 { J _{t} = Q }}{2}$ and the hidden-transition weight specified as follows. A reset sign $u^{'}$ has weight $\frac{1 + u ^{'} ε _{C}}{2} .$ If $J_{t} = Q$ , then only transitions with $J_{t + 1} = 0$ receive this reset-sign weight. If $J_{t} < Q$ , transitions with $J_{t + 1} = 0$ receive weight $(1 - α) (1 + U_{t + 1} ε_{C}) /2$ , while transitions with $J_{t + 1} = J_{t} + 1$ receive weight $α 1 {U_{t + 1} = U_{t}} when A_{t} = 1, \frac{α}{2} when A_{t} = 0,$ and all other hidden transitions receive weight zero. The finite reward symbols are mapped to real rewards by $0 \mapsto - 1$ and $1 \mapsto 1$ , and the trajectory law is the finite chronological law generated by this initial distribution, behavior policy, and kernel, then embedded into the unrestricted real-reward raw experiment.

⊢ Lean

The signed-depth two-point sub-experiment in Definition 18 is the finite construction used for the lower bounds. Its terminal depth $Q$ , alternative sign $v$ , latent depth coordinate $J_{t}$ , latent sign coordinate $U_{t}$ , reset-sign perturbation $ε_{C}$ , reward-amplitude constant $c_{0}$ , and truncated geometric mass $w_{j}$ make the overlap and contraction parameters explicit inside a finite POMDP.

Main results

The preceding section defined the observed-data experiment class, the stationary target value, the minimax risk, and the partial-history importance-weighted estimators. This section gives the risk surface as the latent stationary overlap radius varies. The organizing parameters are the normalized distance from unit overlap and the radius-calibrated history depth, where the radius $C$ is supplied to the estimator as a tuning input.

Definition 19 [def:overlap-distance-radius] (Overlap-distance radius

q_{C}

For $C \geq 1$ , define $q_{C} := \frac{C - 1}{C} .$

⊢ Lean

The radius $q_{C}$ is zero at equality of the two stationary laws and increases with the allowed stationary density-ratio bound. It is the scale on which the hidden-state part of the risk is measured.

Definition 20 [def:radius-adaptive-history-depth] (Radius-calibrated history depth

κ_{T}^{rad} (C)

For $T \in N$ and $C \geq 1$ , define $κ_{T}^{rad} (C) := ⎩ ⎨ ⎧ 0, min {⌊ \frac{T}{2} ⌋, ⌊ \frac{lo g ( T q _{C}^{2} )}{2 lo g ( 1/ α ) + lo g L} ⌋}, T q_{C}^{2} \leq 1, T q_{C}^{2} > 1.$

⊢ Lean

The depth $κ_{T}^{rad} (C)$ shortens the history when the radius is close to unit overlap and lengthens it when the latent stationary shift is large enough to make hidden-state memory relevant. The threshold $T q_{C}^{2} \leq 1$ is the parametric boundary layer in the theorem below.

For the all-radius theorem it is convenient to index models directly and write their target values without carrying the kernel and policy notation in each display.

Definition 21 [synth_16] (Finite signed-depth observed-word law).

For $C > 1$ , $Q \geq 1$ , and $v \in {- 1, + 1}$ , let $P_{Q, v}^{fin, T}$ be the finite observed-word law through horizon $T$ generated by the signed-depth construction of Definition 18. Its observed state is constant, its latent state is $H_{t} = (J_{t}, U_{t}) \in {0, \dots, Q} \times {- 1, + 1}$ , its behavior and target policies satisfy $b (1) = 1/ L$ and $e (1) = 1$ , and its reward alphabet is ${- 1, + 1}$ with $v Pr (Y_{t} = y ∣ J_{t}, U_{t}, A_{t}) = \frac{1 + y v c _{0} U _{t} 1 { J _{t} = Q }}{2} .$ The initial latent law is $d_{b} (j, u) = w_{j} (1 + u ε_{C} L^{- j}) /2$ , and the resulting finite observation consists of the action-reward word $(A_{t}, Y_{t})_{t = 0}^{T - 1}$ , equivalently the finite observed trajectory with constant observed-state coordinate.

The next theorem is the master characterization. Its constants depend on $t_{0}$ and $ζ$ , while the comparison holds uniformly over every radius $C \geq 1$ and over the finite observed and hidden cardinalities allowed by Definition 10.

Definition 22 [synth_14] (Target value of an indexed model).

For an indexed model $m \in M_{T} (t_{0}, ζ, C)$ , write $K_{m}$ and $e_{m}$ for its joint reward-transition kernel and target policy. The target value of $m$ is $θ_{m} := θ (K_{m}, e_{m}),$ with $θ (K, e)$ the stationary target-policy causal value from Definition 12.

Theorem 1 gives a single observable estimator rule — one formula, applied at every radius — together with a matching minimax surface. The rule is not radius-free: the depth it selects is computed from the supplied value of $C$ , so different radii give different depths and different estimators. The expression for $r_{T} (t_{0}, ζ, C)$ has a parametric term and a hidden-state term scaled by $q_{C}$ . The radius-calibrated rule in Definition 20 implements this balance directly: in the boundary layer it uses depth zero, and away from that layer it increases the observable history length according to the effective radius $T q_{C}^{2}$ .

The lower-bound calculations use the finite observed law induced by the signed-depth construction through horizon $T$ . Naming that law separates the observable experiment used in the converse from the latent construction that generates it.

Definition 23 [synth_10] (Unclipped partial-history average).

For the balanced history depth $k_{T}$ , define the unclipped partial-history average by $θ_{k_{T}}^{raw} := \frac{1}{T - k _{T}} t = k_{T} \sum T - 1 Z_{t} (k_{T}), Z_{t} (k_{T}) := Y_{t} j = t - k_{T} \prod t ρ_{j} .$ Here $ρ_{j} = e (A_{j} ∣ X_{j}) / b (A_{j} ∣ X_{j})$ on behavior-supported cells, with the zero-cell convention used for $ρ_{j}$ .

For comparison with the clipped estimator in Definition 16, we also record the unclipped average at the fixed-radius balancing depth. This object is useful for interpreting the bias–variance calculation, while the risk bounds below are stated for the clipped estimator.

Theorem 1 [thm:uniform-overlap-frontier] (Uniform overlap frontier).

Fix $t_{0} > 0$ and $ζ > 0$ . Let $R_{T} (t_{0}, ζ, C)$ be the minimax squared-error risk in Definition 14, let $q_{C} = (C - 1) / C$ be the overlap-distance radius in Definition 19, and set $β = \frac{2}{2 + t _{0} ζ}, r_{T} (t_{0}, ζ, C) = T^{- 1} + T^{- β} q_{C}^{2 (1 - β)} .$ There exist constants $c_{lo}, c_{hi} > 0$ , with $c_{lo} \leq c_{hi}$ , and an integer $T_{⋆} \geq 1$ , all depending only on $t_{0}$ and $ζ$ , such that for every $C \geq 1$ and every $T \geq T_{⋆}$ , $c_{lo} r_{T} (t_{0}, ζ, C) \leq R_{T} (t_{0}, ζ, C) \leq c_{hi} r_{T} (t_{0}, ζ, C),$ and the radius-adaptive partial-history importance-weighted estimator with depth $κ_{T}^{rad} (C)$ from Definition 20 satisfies $m \in M_{T} (t_{0}, ζ, C) sup E_{m} [(θ_{κ_{T}^{rad} (C)} - θ_{m})^{2}] \leq c_{hi} r_{T} (t_{0}, ζ, C) .$ Moreover, for every sequence $C_{T} \geq 1$ , if $T q_{C_{T}}^{2}$ is eventually bounded, then there is a constant $c_{imm} > 0$ , depending on the sequence and on $t_{0}, ζ$ , such that eventually $m \in M_{T} (t_{0}, ζ, C_{T}) sup E_{m} [(θ_{0} - θ_{m})^{2}] \leq \frac{c _{imm}}{T} .$

⊢ Lean

Holding the radius at a nonunit value gives the fixed-radius slice of Theorem 1. The next theorem states that slice in the fixed-depth notation of Definition 17, together with the signed-depth alternatives from Definition 18 that witness the converse.

Theorem 2 [thm:fixed-c-minimax] (Fixed-overlap minimax rate).

Fix $t_{0}, ζ, C \in R$ . Assume:

(Positive memory exponent.) $t_{0} > 0$ .

(Positive overlap exponent.) $ζ > 0$ .

(Fixed latent-overlap radius.) $C > 1$ .

Let the exponent $β$ be $β = \frac{2}{2 + t _{0} ζ} .$ There exist constants $c_{⋆}, c^{⋆} \in R$ and an integer $T_{⋆} \geq 1$ such that $0 < c_{⋆} \leq c^{⋆}$ and, for every integer $T \geq T_{⋆}$ , $c_{⋆} T^{- β} \leq R_{T} (t_{0}, ζ, C) \leq c^{⋆} T^{- β},$ where $R_{T} (t_{0}, ζ, C)$ is the minimax squared-error risk in Definition 14. Moreover, the clipped partial-history importance-weighted estimator $θ_{k_{T}}$ , with $k_{T}$ as in Definition 17, satisfies $M \in M_{T} (t_{0}, ζ, C) sup E_{M} [{θ_{k_{T}} - θ (M)}^{2}] \leq c^{⋆} T^{- β},$ with $θ (M)$ the stationary target-policy mean reward in Definition 12.

For each integer $T \geq T_{⋆}$ , there is an integer $Q \geq 1$ such that the lower bound is witnessed by the two signed-depth alternatives $M_{Q, v}^{T}$ , $v \in {- 1, + 1}$ , generated by Definition 18 with one observed state and $2 (Q + 1)$ latent states: for every measurable observable-data estimator $θ$ , under the usual squared-error integrability conditions for both alternatives, $c_{⋆} T^{- β} \leq v \in {- 1, + 1} max E_{M_{Q, v}^{T}} [{θ - θ (M_{Q, v}^{T})}^{2}] .$

⊢ Lean

The exponent $β$ reflects the balance between geometric forgetting and the variance cost of multiplying observable policy ratios. Since $q_{C}$ is a positive constant on this slice, the all-radius expression in Theorem 1 reduces to the power $T^{- β}$ up to constants. The estimator achieves that rate, and the two-point construction shows that every observable-data estimator faces the same order of risk over the class.

The unit-overlap boundary is the $C = 1$ slice of the same surface. At this endpoint, the normalized radius is $q_{C} = 0$ , and the risk expression in Theorem 1 is on the $T^{- 1}$ scale.

Theorem 3 [prop:unit-overlap-boundary] (Unit overlap boundary).

Let $t_{0} > 0$ and $ζ > 0$ . At unit latent overlap, $C = 1$ , the following assertions hold.

For every trajectory length $T$ , every finite observed and hidden state cardinalities, and every experiment $M \in M_{T} (t_{0}, ζ, 1)$ in the model class of Definition 10, the target stationary law $d_{e}$ and the behavior stationary law $d_{b}$ of the corresponding policy-induced joint-state transition kernels coincide: $d_{e} = d_{b} .$

Moreover, there is a constant $c_{imm} > 0$ , depending only on $t_{0}$ and $ζ$ , such that for every integer $T \geq 1$ , the worst-case squared risk of the immediate observable estimator $θ_{0}$ satisfies $M \in M_{T} (t_{0}, ζ, 1) sup E_{M} [(θ_{0} - θ (K, e))^{2}] \leq \frac{c _{imm}}{T} .$ Finally, for the minimax squared-error risk $R_{T} (t_{0}, ζ, 1)$ of Definition 14, there are constants $c_{lo}, c_{hi} > 0$ with $c_{lo} \leq c_{hi}$ and an integer $T_{⋆} \geq 1$ such that, for every $T \geq T_{⋆}$ , $\frac{c _{lo}}{T} \leq R_{T} (t_{0}, ζ, 1) \leq \frac{c _{hi}}{T} .$ The hidden-state exponent at these positive scales satisfies $\frac{2}{2 + t _{0} ζ} < 1.$

⊢ Lean

The boundary result identifies the equality case as an immediate-weighting regime. The coincidence $d_{e} = d_{b}$ gives the stationary invariance needed for depth zero, and the matching minimax bounds place the endpoint on the $T^{- 1}$ scale.

The remaining regime reads Theorem 1 along a deterministic path approaching the unit-overlap boundary. Write the nonnegative excess as $δ_{T}$ ; then $C_{T} = 1 + δ_{T}$ traces a local path into the nonunit region.

Theorem 4 [thm:shrinking-overlap-frontier] (Shrinking-overlap frontier).

The following conditions hold:

(Fixed smoothness.) The constants satisfy $t_{0} > 0$ and $ζ > 0$ .

(Shrinking radius.) The deterministic radius excess sequence $(δ_{T})_{T \geq 1}$ satisfies $δ_{T} \geq 0$ for every $T$ .

(Local limit.) The sequence satisfies $δ_{T} \to 0$ .
(Rate exponent.) Let $β = 2/ (2 + t_{0} ζ)$ .
(Local rate and depth.) Define $r_{T} (δ) = T^{- 1} + T^{- β} δ_{T}^{2 (1 - β)}$ and $κ_{T} = ⎩ ⎨ ⎧ 0, min {⌊ T /2 ⌋, ⌊ \frac{lo g ( T δ _{T}^{2} )}{2 lo g ( 1/ α ) + lo g L} ⌋}, T δ_{T}^{2} \leq 1, T δ_{T}^{2} > 1.$

Then there exist constants $c_{loc}, C_{loc}$ with $0 < c_{loc} \leq C_{loc}$ , depending only on $t_{0}$ and $ζ$ , such that, for all sufficiently large $T$ , $c_{loc} r_{T} (δ) \leq R_{T} (t_{0}, ζ, 1 + δ_{T}) \leq C_{loc} r_{T} (δ),$ where $R_{T}$ is the minimax squared-error risk in Definition 14. Moreover, the observable partial-history importance-weighted estimator $θ_{κ_{T}}$ of Definition 16 satisfies $M \in M_{T} (t_{0}, ζ, 1 + δ_{T}) sup E_{M} [(θ_{κ_{T}} - θ (M))^{2}] \leq C_{loc} r_{T} (δ)$ for all sufficiently large $T$ . Finally, if there is a constant $D \geq 0$ such that $T δ_{T}^{2} \leq D$ for all sufficiently large $T$ , then there is a constant $c_{imm} > 0$ such that $M \in M_{T} (t_{0}, ζ, 1 + δ_{T}) sup E_{M} [(θ_{0} - θ (M))^{2}] \leq \frac{c _{imm}}{T}$ for all sufficiently large $T$ .

⊢ Lean

The local rate decomposes into the parametric term and a hidden-state term scaled by the shrinking excess. When $T δ_{T}^{2}$ remains bounded, the immediate estimator already attains the parametric order. When the product grows, the depth in the theorem increases logarithmically with the effective radius $T δ_{T}^{2}$ , matching the same risk expression.

A finite insulin-policy adaptation

The minimax characterization in Theorem 1 is stated for an abstract finite partially observed model class. This section records a finite insulin-policy construction that keeps the glucose-threshold structure familiar from mobile-health and diabetes treatment applications while putting every component on a finite alphabet (Maahs et al., 2012; Liao et al., 2021; Luckett et al., 2020). The construction, denoted $I_{grid}$ , uses refreshed diet and activity coordinates to create latent heterogeneity, uniform mixing, and stationary overlap in a single controlled Markov model.

Definition 24 [def:insulin-grid] (Insulin grid

I_{grid}

The finite insulin-grid state is $s = (g, d_{1}, d_{2}, x_{1}, x_{2}, a_{1}) \in {50, 100, 150, 200, 250} \times {0, 78}^{2} \times {0, 31, 850}^{2} \times {0, 1},$ so the state space has $360$ states. The diet coordinates form $H_{t} = (d_{1}, d_{2}),$ and all other coordinates form $X_{t}$ . The behavior and target policies are $b (1 ∣ X_{t}) = 0.3, e (1 ∣ X_{t}) = 1 {g \geq 125} .$ With probability $1/2$ , the next full state is drawn uniformly. With the remaining probability, draw $d_{0} \in {0, 78} with probabilities (0.8, 0.2),$ draw $x_{0} \in {0, 31, 850} with probabilities (0.4, 0.4, 0.2),$ draw $ξ \sim N (0, 25)$ , and set $g^{'} = round_{{50, 100, 150, 200, 250}} clip_{[50, 250]} (10 + 0.9 g + 0.1 d_{1} + 0.1 d_{2} - 0.01 x_{1} - 0.01 x_{2} - 2 A_{t} - 4 a_{1} + ξ) .$ The memory coordinates then shift to $(g^{'}, d_{0}, d_{1}, x_{0}, x_{1}, A_{t}) .$ The reward is $Y_{t} = ⎩ ⎨ ⎧ - 1, - 2/3, - 1/3, 0, g \leq 70, g > 150, 70 < g \leq 80 or 120 < g \leq 150, otherwise .$ This construction defines $I_{grid}$ .

⊢ Lean

Two conventions in the displayed transition fix the numbers the certificate uses, so we state them explicitly. First, $N (0, 25)$ is the centered Gaussian of variance $25$ , that is standard deviation $5$ . Second, the composite clip-and-round operator is the nearest-grid map with cutpoints at the midpoints $75, 125, 175, 225$ of the glucose grid, the two extreme cells absorbing everything beyond them; each cell is the interval closed on the left, so a value landing exactly on a midpoint is assigned to the higher grid point, a tie that has probability zero under the continuous noise. Writing $m$ for the deterministic part of the displayed expression and $Φ$ for the standard normal distribution function, the next-glucose weights are therefore $Pr {g^{'} = 50} = Φ (\frac{75 - m}{5}), Pr {g^{'} = 250} = 1 - Φ (\frac{225 - m}{5}),$ and, for $i \in {1, 2, 3}$ , $Pr {g^{'} = 50 + 50 i} = Φ (\frac{75 + 50 i - m}{5}) - Φ (\frac{75 + 50 ( i - 1 ) - m}{5}) .$

The observed coordinate contains the glucose level, lagged activity, and lagged treatment, while the diet coordinates are latent. The target rule is a deterministic threshold policy on the observed glucose coordinate, and the behavior rule randomizes treatment with fixed probability. The refresh step supplies a direct minorization component, so the example aligns with the contraction and overlap conditions used by Theorem 1 within a finite-state adaptation of the insulin-policy setting.

The next two facts record the mixing and stationary-law ingredients behind the certified contraction and stationary-overlap constants. They apply to any policy rule on the observed grid states, including the behavior and target policies in Definition 24.

Lemma 1 [lem:insulin-policy-contraction] (Uniform one-half contraction of the insulin-grid policy kernels).

Let $T \in N$ and let $p$ assign to each of the $90$ observed states of the refreshed finite insulin adaptation $I_{grid}$ of Definition 24 a probability law on the action set ${0, 1}$ . Write $S_{grid}$ for the $360$ -point joint state space, $x (s)$ for the observed coordinate of a grid state $s$ , $K_{T}$ for the transition kernel of $I_{grid}$ , and $P_{p}^{(T)} (s, s^{'}) = a \in {0, 1} \sum p (a ∣ x (s)) K_{T} {(q, s^{''}) : s^{''} = s^{'} ∣ s, a}, s, s^{'} \in S_{grid},$ for the induced transition matrix. Then, for all probability vectors $ν, ν^{'}$ on $S_{grid}$ , $∥ ν P_{p}^{(T)} - ν^{'} P_{p}^{(T)} ∥_{TV} \leq \frac{1}{2} ∥ ν - ν^{'} ∥_{TV},$ equivalently, in the $ℓ^{1}$ norm $∥ \cdot ∥_{1} = 2∥ \cdot ∥_{TV}$ , $∥ ν P_{p}^{(T)} - ν^{'} P_{p}^{(T)} ∥_{1} \leq \frac{1}{2} ∥ ν - ν^{'} ∥_{1} .$

⊢ Lean

The proof is deferred to Section B.

Lemma 2 [lem:insulin-stationary-law] (The selected insulin-grid stationary law is stationary).

Let $T \in N$ and let $p$ assign to each observed state of the refreshed finite insulin adaptation $I_{grid}$ of Definition 24 a probability law on ${0, 1}$ . Suppose that the induced transition matrix $P_{p}^{(T)}$ on the joint state space $S_{grid}$ satisfies $∥ ν P_{p}^{(T)} - ν^{'} P_{p}^{(T)} ∥_{TV} \leq \frac{1}{2} ∥ ν - ν^{'} ∥_{TV}$ for all probability vectors $ν, ν^{'}$ on $S_{grid}$ . Then the stationary law $d_{p}$ selected for $P_{p}^{(T)}$ is a probability vector and satisfies the stationarity equations $d_{p} (s^{'}) = s \in S_{grid} \sum d_{p} (s) P_{p}^{(T)} (s, s^{'}), s^{'} \in S_{grid} .$

⊢ Lean

The proof is deferred to Section B.

Lemma 1 gives the one-half total-variation contraction supplied by the refresh component. Lemma 2 then identifies the selected stationary distribution as a genuine invariant law for each policy-induced transition matrix, providing the stationary objects needed for the target value and the bias diagnostic.

The reward in Definition 24 is a function of the current grid state. The following regressions make this explicit under target-policy averaging and under one-step target-to-behavior reweighting.

Lemma 3 [lem:insulin-target-reward-regression] (Target-policy reward regression on the insulin grid).

Let $T \in N$ , let $I_{grid}$ be the refreshed finite insulin adaptation of Definition 24 with transition kernel $K_{T}$ and target policy $e$ , and for a grid state $s = (g, d_{1}, d_{2}, x_{1}, x_{2}, a_{1})$ write $x (s) = (g, x_{1}, x_{2}, a_{1})$ for its observed coordinate and $r_{ins} (s) = ⎩ ⎨ ⎧ - 1, - 2/3, - 1/3, 0, g \leq 70, g > 150, 70 < g \leq 80 or 120 < g \leq 150, otherwise,$ for the insulin reward vector. Writing $p_{1}$ for the numeric reward coordinate of a reward/state pair $p$ drawn by $K_{T}$ , we have, for every $s \in S_{grid}$ , $a \in {0, 1} \sum e (a ∣ x (s)) \int p_{1} K_{T} (d p ∣ s, a) = r_{ins} (s) .$

⊢ Lean

The proof is deferred to Section B.

Lemma 4 [lem:insulin-immediate-reward-regression] (Immediate weighted reward regression on the insulin grid).

Let $T \in N$ , let $I_{grid}$ be the refreshed finite insulin adaptation of Definition 24 with transition kernel $K_{T}$ , behavior policy $b$ and target policy $e$ , and for a grid state $s = (g, d_{1}, d_{2}, x_{1}, x_{2}, a_{1})$ write $x (s) = (g, x_{1}, x_{2}, a_{1})$ . Let $ρ (x, a) = {e (a ∣ x) / b (a ∣ x), 0, b (a ∣ x) \neq = 0, otherwise,$ and let $r_{ins}$ be the insulin reward vector $r_{ins} (s) = ⎩ ⎨ ⎧ - 1, - 2/3, - 1/3, 0, g \leq 70, g > 150, 70 < g \leq 80 or 120 < g \leq 150, otherwise,$ Writing $p_{1}$ for the numeric reward coordinate of a reward/state pair $p$ drawn by $K_{T}$ , we have, for every $s \in S_{grid}$ , $a \in {0, 1} \sum b (a ∣ x (s)) ρ (x (s), a) \int p_{1} K_{T} (d p ∣ s, a) = r_{ins} (s) .$

⊢ Lean

The proof is deferred to Section B.

Together, Lemmas 3 and 4 show that the immediate reward regression is the same state reward under target averaging and under one-step reweighting. The remaining discrepancy in current-action weighting therefore comes from the stationary law used to average this regression: behavior-stationary weighting versus the target-stationary value in Definition 12.

The finite diagnostic uses certified interval arithmetic for stationary reward averages. The next statements first give the general enclosure principle and then instantiate it for the behavior and target stationary reward averages on the insulin grid.

Lemma 5 [lem:certified-stationary-reward-enclosure] (Stationary reward enclosure).

Let $S$ be a finite set, and let $P$ be a stochastic matrix on $S$ whose entries satisfy $P (s, s^{'}) \in A (s, s^{'})$ for all $s, s^{'} \in S$ . The intervals are rational intervals $[ℓ, u]$ with $ℓ \leq u$ , combined by exact interval arithmetic: $[a, b] + [c, d] = [a + c, b + d], [a, b] ⊖ [c, d] = [a - d, b - c],$ and $[a, b] \cdot [c, d]$ is the interval whose endpoints are the smallest and largest of the four endpoint products. For an interval $r$ , write $ℓ (r)$ and $u (r)$ for its endpoints and $∣ r ∣_{m a x} = max {∣ ℓ (r) ∣, ∣ u (r) ∣} .$ Suppose the following conditions hold.

(Initial row and interval rows.) There are an exact rational probability vector $μ_{0}$ on $S$ , a number $m \in N$ , and interval rows $T_{0}, \dots, T_{m + 1}$ on $S$ such that $μ_{0} (s) \in T_{0} (s)$ for every $s \in S$ .

(Chunked recurrence.) For every $k \leq m$ and every output state $s^{'}$ , there are pairwise disjoint blocks $G_{1}, \dots, G_{J} \subseteq S$ covering $S$ , each with at most a fixed positive number of states; block intervals $Ξ_{1} (s^{'}), \dots, Ξ_{J} (s^{'})$ ; and partial-sum intervals $Σ_{1}, \dots, Σ_{J + 1}$ such that $s \in G_{j} \sum T_{k} (s) \cdot A (s, s^{'}) \subseteq Ξ_{j} (s^{'})$ for each $j$ , $0 \in Σ_{J + 1}, Ξ_{j} (s^{'}) + Σ_{j + 1} \subseteq Σ_{j}$ for $j = J, \dots, 1$ , and $Σ_{1} = T_{k + 1} (s^{'}) .$

(Reward bounds.) There are a reward vector $r : S \to R$ , rational intervals $r (s) ∋ r (s)$ , and a rational bound $B \geq 0$ such that $∣ r (s) ∣_{m a x} \leq B$ for every $s \in S$ .

(Contraction and stationarity.) There are a rational coefficient $0 \leq ρ < 1$ and a probability vector $π$ such that $∥ ν P - ν^{'} P ∥_{1} \leq ρ ∥ ν - ν^{'} ∥_{1}$ for all probability vectors $ν, ν^{'}$ on $S$ , and $π P = π .$

Define $δ = s \in S \sum T_{m} (s) ⊖ T_{m + 1} (s)_{m a x}, E = s \in S \sum T_{m} (s) \cdot r (s),$ and $J = [ℓ (E) - \frac{B δ}{1 - ρ}, u (E) + \frac{B δ}{1 - ρ}] .$ Then $s \in S \sum π (s) r (s) \in J .$

⊢ Lean

The proof is deferred to Section B.

Lemma 5 converts a finite number of interval matrix calculations into a bound on the stationary reward average. In the insulin grid, it supplies the certificate used to compare the behavior-stationary immediate-weighted average with the target-stationary value.

Lemma 6 [lem:insulin-bias-interval-evaluation] (Insulin bias interval bounds).

For $⋆ \in {b, e}$ , let $T_{⋆, 0}$ and $T_{⋆, 1}$ be the terminal and successor rational interval rows on the $360$ -state grid $S_{grid}$ of Definition 24 recorded by the zero-step chunked iterate certificate, in blocks of at most $40$ states, for the rational interval enclosure of the behavior-policy case $⋆ = b$ or target-policy case $⋆ = e$ . Let $r_{ins} (s) = [r_{ins} (s), r_{ins} (s)]$ , where $r_{ins} (s) \in {- 1, - 2/3, - 1/3, 0}$ is the grid reward from Definition 24. Using the exact rational interval arithmetic of Lemma 5, define $δ_{⋆} = s \in S_{grid} \sum T_{⋆, 0} (s) ⊖ T_{⋆, 1} (s)_{m a x}, E_{⋆} = s \in S_{grid} \sum T_{⋆, 0} (s) \cdot r_{ins} (s),$ $J_{⋆} = [ℓ (E_{⋆}) - \frac{δ _{⋆}}{1 - 1/2}, u (E_{⋆}) + \frac{δ _{⋆}}{1 - 1/2}], B = J_{b} ⊖ J_{e} .$ Equivalently, $B = [ℓ (J_{b}) - u (J_{e}), u (J_{b}) - ℓ (J_{e})] .$ Then the endpoints of this rational interval satisfy $- \frac{6}{1000} < ℓ (B) and u (B) < - \frac{58}{10000} .$

⊢ Lean

The proof is deferred to Section B.

Lemma 6 is the finite arithmetic step behind the diagnostic. It encloses the behavior- and target-stationary reward averages in rational intervals and shows that their interval difference lies strictly below zero.

The first diagnostic isolates the stationary bias of current-action weighting in this finite construction. It compares the one-step reweighted reward under the behavior stationary law with the stationary target value.

Lemma 7 [lem:insulin-bias-certificate] (Insulin-grid bias certificate).

For every natural horizon $T$ , let $Δ_{imm} (T)$ be the stationary immediate-weighting bias for the refreshed finite insulin adaptation $I_{grid}$ of Definition 24, $Δ_{imm} (T) = s \sum d_{b} (s) a \in {0, 1} \sum b (s_{1}, a) ρ (s_{1}, a) \int p_{1} K_{T} (d p ∣ s, a) - θ (K_{T}, e),$ where $d_{b}$ is the stationary law of the behavior-policy transition kernel on the insulin grid, $K_{T}$ is the refreshed insulin-grid transition kernel, $b$ and $e$ are its behavior and target policies, $ρ (s_{1}, a)$ is the one-step target-to-behavior policy ratio, and $θ (K_{T}, e)$ is the target value in Definition 12. Then $- \frac{6}{1000} < Δ_{imm} (T) < - \frac{58}{10000} .$

⊢ Lean

The proof is deferred to Section B.

The interval in Lemma 7 is a finite-state certificate for the immediate-weighting bias diagnostic $Δ_{imm}$ . The theorem below records the model-class properties of the same construction and restates the diagnostic in the notation of the insulin-grid transition law.

Theorem 5 [prop:finite-insulin-demonstration] (Finite insulin witness).

For every horizon $T$ , the refreshed insulin experiment $I_{grid}$ of Definition 24 satisfies:

(State count.) Its observed–latent joint-state alphabet has cardinality $90 \cdot 4 = 360$ .

(Uniform contraction.) It satisfies Assumption 6 with contraction parameter $α = 1/2$ .

(Policy overlap.) It satisfies Assumption 5 with one-step overlap constant $L = 10/3$ .

(Stationary overlap.) It satisfies Assumption 7 with stationary-overlap constant $C = 720$ .

Define the immediate-weighting bias diagnostic by $Δ_{imm} (T) = s \sum d_{b} (s) a \in {0, 1} \sum b (a ∣ s_{X}) \frac{e ( a ∣ s _{X} )}{b ( a ∣ s _{X} )} \int r K (d s^{'}, d r ∣ s, a) - θ (K, e),$ where $K, b, e$ are the transition kernel, behavior policy, and target policy of $I_{grid}$ , $s_{X}$ is the observed coordinate of $s$ , and $d_{b}$ is the stationary law induced by $b$ . This diagnostic is interval-certified as strictly negative: $- 0.006 < Δ_{imm} (T) < - 0.0058.$

⊢ Lean

Theorem 5 certifies five things about the grid, and exactly these five. Its joint observed–latent alphabet has $360$ states. It contracts in total variation at rate $α = 1/2$ , so it meets Assumption 6. Its policy pair has one-step overlap constant $L = 10/3$ , so it meets Assumption 5. Its two stationary joint-state laws satisfy the latent stationary overlap bound of Assumption 7 at radius $C = 720$ . And its immediate-weighting diagnostic $Δ_{imm}$ lies in the interval $(- 0.006, - 0.0058)$ , certified by interval arithmetic.

The last of these is the substantive one, and it is what the construction was built to isolate. Immediate weighting reweights the observed reward by the current-action ratio alone, so it averages the observed-glucose reward under behavior-stationary occupancy; the target value averages the same reward under target-stationary occupancy. The certified interval quantifies the gap between these two occupancies in a model whose latent stationary overlap is finite and whose mixing is fast. Bounded latent occupancy and fast forgetting are therefore compatible with a stationary bias for current-action weighting, and the source of that bias is occupancy rather than slow mixing or a divergent action ratio. This is the mechanism the depth parameter of Definition 16 exists to control.

Discussion and limitations

Theorems 2, 4, and 1 describe a risk surface indexed by the stationary latent-overlap radius. For each fixed $C > 1$ , the minimax risk $R_{T} (t_{0}, ζ, C)$ has rate $T^{- β}$ , while at $C = 1$ it has the parametric $T^{- 1}$ scale. The radius-sensitive statements refine this picture by expressing how the risk changes as $C$ approaches one. In that local regime, the normalized radius $q_{C}$ from Definition 19 is the relevant coordinate: it measures how far the target stationary occupancy may move from the behavior stationary occupancy while retaining the finite-state POMDP restrictions of Definition 10.

The elbow $q_{C} ≍ T^{- 1/2}$ has a direct statistical interpretation. Below this scale, the local overlap distance is small enough that the parametric floor governs the risk, as recorded by Theorem 3 and the small-radius branch of Theorem 4. Above this scale, the hidden-state occupancy discrepancy becomes visible in the observable-data experiment through longer histories, and the larger-radius branch of Theorem 4 gives the corresponding degradation. The radius-calibrated history depth $κ_{T}^{rad} (C)$ in Definition 20 operationalizes this tradeoff: it selects depth zero in the parametric neighborhood and increases the observable history length when the overlap radius makes latent imbalance statistically consequential.

The role of latent stationary overlap is therefore distinct from ordinary one-step action overlap. The one-step condition controls the variance of products of observable policy ratios, while the stationary density-ratio condition controls how target and behavior policies populate hidden states. Theorem 1 is the master rate statement: Theorems 2, 3, and 4 are the fixed-overlap, unit-overlap, and local shrinking-overlap slices. The proof balances the PHIW bias scale $q_{C} α^{k}$ , coming from contraction after conditioning on a length- $k$ observed history, against the variance scale $L^{k} / T$ , coming from products of one-step policy ratios. The signed-depth lower bound chooses a terminal depth $Q$ so that the observable KL is controlled by a constant times $T ε_{C}^{2} α^{2 Q} L^{- Q}$ , while the target value gap is calibrated to the upper-bound bias scale. The radius-calibrated depth in Definitions 20 and 16 attains the displayed frontier when calibrated with $(t_{0}, ζ, C)$ , and a known shrinking law for $q_{C}$ gives the corresponding changing depth.

The finite insulin demonstration in Theorem 5 is a controlled finite-state policy example. The construction supplies a refreshed Markov model with finite contraction, one-step policy overlap, and the sufficient stationary density-ratio bound $C = 720$ , hence $q_{C} = 719/720 \approx 0.9986$ . Its immediate-weighting diagnostic remains separated from zero under the displayed interval arithmetic because current-action weighting averages the observed-glucose reward under behavior-stationary occupancy rather than target-stationary occupancy. The example therefore exhibits, in a stylized treatment-policy model, the occupancy mechanism that the depth parameter of the partial-history estimator is designed to control. Its certified content is the four structural constants and the bias interval; establishing the full model-class predicate for this grid — in particular fixing an initial law and verifying the stationary-start, Markov-kernel, sequential-ignorability, and bounded-reward conditions at $t_{0} = 1/ lo g 2$ and $ζ = lo g (10/3)$ — is a separate verification we do not carry out here, so the frontier theorem is not being applied to this grid.

Limitations.

The signed-depth converse in Definition 18 uses a latent alphabet whose terminal depth grows with $T$ , so the lower-bound construction is calibrated to a sequence of finite experiments. Confidence intervals and policy-learning regret are separate research directions built on top of the value-estimation problem studied here. The finite insulin adaptation in Definition 24 is a controlled finite-state demonstration designed to instantiate the paper’s overlap and contraction constants within an interpretable treatment-policy example. Three limits of that demonstration are worth stating. Its certificate covers the four structural constants and the bias interval, not the full model-class predicate, so no rate guarantee of this paper is asserted for it. It is not a finite-sample comparison of estimators, and it does not show that the depth calibrated from the sufficient bound $C = 720$ is the practically preferable depth on this grid. And a data-driven choice of depth for an unknown radius remains open here as elsewhere in the paper.

Appendices

Proofs and auxiliary constructions

This appendix collects the auxiliary statements used to support the main rate calculations. The first group records the observable filtration and the finite-prefix quantities that enter the signed-depth likelihood comparison.

Definition 25 [def:filter-quantities] (Filter quantities

F_{t - 1}^{obs}, ℓ_{t}, q_{t}^{v}, a_{t}^{(Q)}

Define $F_{t - 1}^{obs} := σ (A_{0}, Y_{0}, \dots, A_{t - 1}, Y_{t - 1}),$ $ℓ_{t} := min {Q, t},$ $q_{t}^{v} := v Pr (J_{t} = Q ∣ F_{t - 1}^{obs}),$ and $a_{t}^{(Q)} := L^{- (Q - ℓ_{t})} j = t - ℓ_{t} \prod t - 1 1 {A_{j} = 1},$ where the empty product is equal to one.

⊢ Lean

The quantities in Definition 25 separate two sources of information in the observed history. The posterior $q_{t}^{v}$ captures the chance that the hidden depth has reached its terminal value, while the factor $a_{t}^{(Q)}$ records the visible action pattern needed for that event under the rare-action construction. This separation is the device that makes the observed likelihood calculation finite-prefix and observable.

The upper-bound argument begins with local second-moment and covariance controls for the PHIW scores. These statements isolate, respectively, the contribution from a single weighted window and the dependence between overlapping windows.

Lemma 8 [lem:phiw-window-second-moment] (Window second moment).

Let $T, n_{X}, n_{H} \in N$ , let $k \in N$ , let $L \in R$ , and let $M$ be a raw POMDP experiment on finite observed and hidden alphabets. Suppose that:

(Sequential ignorability.) $M$ satisfies Assumption 2.

(Policy overlap.) $M$ satisfies Assumption 5 with constant $L$ .

(Overlap level.) $L \geq 1$ .

(Kernel law.) $M$ satisfies Assumption 1.

(Bounded rewards.) $M$ satisfies Assumption 4.

(Window position.) $t \in {0, \dots, T - 1}$ and $k \leq t$ .

For the depth- $k$ score $Z_{t} (k)$ of Definition 16, under the observed trajectory law of $M$ , $E_{obs} [Z_{t} (k)^{2}] \leq L^{k + 1} .$

⊢ Lean

Proof of Lemma 8.

Write an observed trajectory as $o = ((x_{j}, a_{j}, y_{j}))_{j = 0}^{T - 1} \in O_{T},$ and set $W_{t, k} := {j \in {0, \dots, T - 1} : j \leq t, t - j \leq k} .$ For every $j$ , define the observable zero-cell policy ratio $ρ_{j} (o) := ⎩ ⎨ ⎧ 0, e (a_{j} ∣ x_{j}) / b (a_{j} ∣ x_{j}), b (a_{j} ∣ x_{j}) = 0, b (a_{j} ∣ x_{j}) \neq = 0.$ Then Definition 16 gives $Z_{t} (k) (o) = y_{t} Π_{t, k} (o), Π_{t, k} (o) := j \in W_{t, k} \prod ρ_{j} (o) .$ Since $k \leq t$ , the window is the contiguous block $W_{t, k} = {t - k, t - k + 1, \dots, t} .$
The window product has observed-law mean one: $E_{obs} [Π_{t, k}] = 1.$ Put $r = t - k$ . Since $k \leq t$ , this is an integer in ${0, \dots, t}$ , and the window from step 1 is exactly the interval ${r, r + 1, \dots, t}$ . First establish the corresponding identity under the full trajectory law. For a full trajectory $τ$ , write $ρ_{j} (τ) := {0, e (A_{j} (τ) ∣ X_{j} (τ)) / b (A_{j} (τ) ∣ X_{j} (τ)), b (A_{j} (τ) ∣ X_{j} (τ)) = 0, b (A_{j} (τ) ∣ X_{j} (τ)) \neq = 0,$ so the full-law block identity to prove is $E [j = r \prod t ρ_{j} (τ)] = 1.$ The argument uses the theorem hypothesis that the target carrier is a policy vector, so for every observed state $x$ , $e (a ∣ x) \geq 0 (a \in A), a \in A \sum e (a ∣ x) = 1,$ together with the domination inequality supplied by Assumption 5. Under Assumption 2, conditional on the past and current joint state at epoch $m$ , the action has behavior probabilities $b (\cdot ∣ X_{m})$ . If $b (a ∣ x) = 0$ , then $0 \leq e (a ∣ x) \leq L b (a ∣ x) = 0$ , hence $e (a ∣ x) = 0$ ; therefore, for every observed state $x$ , $a \in A \sum b (a ∣ x) {0, e (a ∣ x) / b (a ∣ x), b (a ∣ x) = 0, b (a ∣ x) \neq = 0 = a \in A \sum e (a ∣ x) = 1.$ Thus, for every $m \in {r, \dots, t}$ , conditioning on the history and current joint state up to epoch $m$ gives $E [ρ_{m} (τ) ∣ X_{0 : m}, H_{0 : m}, A_{0 : m - 1}, Y_{0 : m - 1}] = 1.$ Peeling the factors one epoch at a time, with Assumption 1 identifying the next history law after each peeled action, gives $E [j = r \prod m ρ_{j} (τ)] = E [j = r \prod m - 1 ρ_{j} (τ)], m = r, \dots, t,$ where the product over ${r, \dots, r - 1}$ is the empty product. Iterating from $m = t$ down to $m = r$ leaves the empty product, whose expectation is $1$ , and hence $E [j = r \prod t ρ_{j} (τ)] = 1.$ It remains to pass this identity to observed trajectories. Let $O$ be the observation map $O (τ) = ((X_{j} (τ), A_{j} (τ), Y_{j} (τ)))_{j = 0}^{T - 1},$ so that the observed law is the pushforward of the full law by $O$ . By definition of $Π_{t, k}$ and the interval identity $W_{t, k} = {r, \dots, t}$ , for every full trajectory $τ$ , $Π_{t, k} (O (τ)) = j \in W_{t, k} \prod ρ_{j} (O (τ)) = j = r \prod t ρ_{j} (τ) .$ The usual integration formula for a pushforward measure therefore gives $E_{obs} [Π_{t, k}] = \int Π_{t, k} (o) d P_{obs} (o) = \int Π_{t, k} (O (τ)) d P (τ) = \int j = r \prod t ρ_{j} (τ) d P (τ) = 1.$
For every observed trajectory $o$ and every $j$ , $0 \leq ρ_{j} (o) \leq L .$ The lower bound uses the behavior-policy vector property from Assumption 2 and the theorem hypothesis that the target carrier is a policy vector. The upper bound is immediate in a zero cell and otherwise follows from the domination inequality in Assumption 5 by dividing $e (a_{j} ∣ x_{j}) \leq L b (a_{j} ∣ x_{j})$ by the positive number $b (a_{j} ∣ x_{j})$ . Hence $0 \leq Π_{t, k} (o) \leq L^{∣ W_{t, k} ∣} .$ The map $j \mapsto t - j$ injects $W_{t, k}$ into ${0, \dots, k}$ , so $∣ W_{t, k} ∣ \leq k + 1.$ Since $L \geq 1$ , $0 \leq Π_{t, k} (o) \leq L^{k + 1} .$
By Assumption 4, the full trajectory law has $∣ Y_{t} ∣ \leq 1$ almost surely; pushing forward to the observed law gives $∣ y_{t} ∣ \leq 1 P_{obs} -almost surely .$ Together with the preceding product bound, this also gives $∣ Z_{t} (k) ∣ \leq L^{k + 1} P_{obs} -almost surely,$ so the square and the comparison function $L^{k + 1} Π_{t, k}$ are integrable under the observed law.
For $P_{obs}$ -almost every observed trajectory, $Z_{t} (k)^{2} = y_{t}^{2} Π_{t, k}^{2} \leq 1 \cdot Π_{t, k}^{2} \leq L^{k + 1} Π_{t, k},$ where the last inequality uses $0 \leq Π_{t, k} \leq L^{k + 1}$ . Integrating and using the normalization from step 2, $E_{obs} [Z_{t} (k)^{2}] \leq E_{obs} [L^{k + 1} Π_{t, k}] = L^{k + 1} E_{obs} [Π_{t, k}] = L^{k + 1} .$

∎

Lemma 9 [lem:phiw-overlapping-window-covariance] (Overlapping-window covariance).

Let $M$ be a raw POMDP experiment of length $T$ on finite observed and hidden alphabets. Let $k, h \in N$ , let $L \in R$ , and let $t, u \in {0, \dots, T - 1}$ . Assume:

(Model conditions.) $M$ satisfies Assumption 2, Assumption 5 with overlap constant $L$ , Assumption 1, and Assumption 4.

(Overlap constant.) $L \geq 1$ .

(Initial window.) $k \leq t$ .

(Lag identity.) $u = t + h$ .

(Overlapping lag.) $1 \leq h \leq k$ .

Then the depth- $k$ scores $Z_{t} (k)$ of Definition 16, evaluated under the observed trajectory law of $M$ , satisfy $Cov_{obs} (Z_{t} (k), Z_{u} (k)) \leq 2 L^{k + 1 - h} .$

⊢ Lean

Proof of Lemma 9.

Let $μ_{obs} := L_{obs} (M)$ be the observed-trajectory law of $M$ . Since it is the pushforward of the probability law of the full raw experiment under the observation map, $μ_{obs}$ is a probability measure. Also, from $k \leq t$ , $u = t + h$ , and $h \geq 1$ , we have $k \leq u .$
We use the following covariance reduction. If $μ$ is a probability law, $ξ, η \in L^{2} (μ)$ , and a number $B$ satisfies $\int ∣ ξ ∣ d μ \leq 1, \int ∣ η ∣ d μ \leq 1, \int ∣ ξ η ∣ d μ \leq B, 1 \leq B,$ then $∣ Cov_{μ} (ξ, η) ∣ \leq 2 B .$ Indeed, $∣ Cov_{μ} (ξ, η) ∣ \leq \int ∣ ξ η ∣ d μ + (\int ∣ ξ ∣ d μ) (\int ∣ η ∣ d μ),$ by expanding the covariance and applying the triangle inequality. The two $L^{1}$ -bounds give $(\int ∣ ξ ∣ d μ) (\int ∣ η ∣ d μ) \leq 1,$ because both factors are nonnegative. Hence $∣ Cov_{μ} (ξ, η) ∣ \leq B + 1 \leq 2 B .$
Apply this reduction with $ξ (w) := Z_{t} (k) (w), η (w) := Z_{u} (k) (w), B := L^{k + 1 - h},$ for observed trajectories $w$ . The two scores are square-integrable under $μ_{obs}$ . To see this, write the one-step ratio in Definition 16 as $ρ_{j} (w) := {e (A_{j} (w) ∣ X_{j} (w)) / b (A_{j} (w) ∣ X_{j} (w)), 0, b (A_{j} (w) ∣ X_{j} (w)) > 0, b (A_{j} (w) ∣ X_{j} (w)) = 0,$ for every observed trajectory $w$ . The policy-vector parts of Assumptions 2 and 5 give $b (a ∣ x), e (a ∣ x) \geq 0$ . If $b (a ∣ x) > 0$ , Assumption 5 gives $0 \leq \frac{e ( a ∣ x )}{b ( a ∣ x )} \leq L .$ If $b (a ∣ x) = 0$ , then $e (a ∣ x) \leq L b (a ∣ x) = 0$ , hence $e (a ∣ x) = 0$ , and the zero-cell convention sets the ratio equal to zero. Thus, for every observed trajectory $w$ , $0 \leq ρ_{j} (w) \leq L for every j .$ Also $∣ Y_{r} ∣ \leq 1$ almost surely by Assumption 4. Hence, for every time $r$ , $∣ Z_{r} (k) (w) ∣ \leq j \in W_{r} \prod ρ_{j} (w) \leq L^{∣ W_{r} ∣} \leq L^{k + 1}, W_{r} := {j : j \leq r, r - j \leq k},$ outside the null set where the reward bound fails. The window $W_{r}$ is a subset of the integer interval ${r - k, \dots, r}$ , so $∣ W_{r} ∣ \leq k + 1$ . The map $w \mapsto Z_{r} (k) (w)$ is measurable: the observed trajectory space carries real rewards and is not finite, so measurability is not a counting argument, but each coordinate map $w \mapsto (X_{j} (w), A_{j} (w))$ and $w \mapsto Y_{j} (w)$ is measurable, the ratio $ρ_{j}$ is a measurable function of the first of these by the two-case definition above, and $Z_{r} (k)$ is a finite product of these measurable maps. Since $μ_{obs}$ is a probability law, the almost-sure bound above implies $Z_{r} (k) \in L^{2} (μ_{obs})$ , and in particular it applies to $r = t$ and $r = u$ .

Moreover, Assumption 1 and the initial-window conditions $k \leq t$ and $k \leq u$ give $\int ∣ Z_{t} (k) ∣ d μ_{obs} \leq 1, \int ∣ Z_{u} (k) ∣ d μ_{obs} \leq 1.$ For a single time $r$ with $k \leq r$ , this last estimate is obtained as follows. For integers $r_{lo}, r_{hi}$ with $0 \leq r_{lo} \leq r_{hi} < T$ , define $Γ_{r_{lo}, r_{hi}} (w) := j = r_{lo} \prod r_{hi} ρ_{j} (w),$ with the empty product equal to one when the lower endpoint is above the upper endpoint. This chronological block has unit expectation: $\int Γ_{r_{lo}, r_{hi}} (w) d μ_{obs} (w) = 1.$ To prove this, pull the integral back from $μ_{obs}$ to the full trajectory law, so that $X_{j}, A_{j}$ , and $Y_{j}$ are full-trajectory coordinates. For $s \in {r_{lo}, \dots, r_{hi}}$ , let $G_{s} := σ (X_{0}, H_{0}, A_{0}, Y_{0}, \dots, A_{s - 1}, Y_{s - 1}, X_{s}, H_{s})$ and $Γ_{r_{lo}, s}^{-} := j = r_{lo} \prod s - 1 ρ_{j},$ where the product is one for $s = r_{lo}$ . The factor $Γ_{r_{lo}, s}^{-}$ is $G_{s}$ -measurable. By Assumption 2, conditionally on $G_{s}$ the action $A_{s}$ has law $b (\cdot ∣ X_{s})$ . The zero-cell convention and the implication $b (a ∣ x) = 0 \Rightarrow e (a ∣ x) = 0$ computed above give $a \in A \sum b (a ∣ X_{s}) ρ (a, X_{s}) = a \in A \sum e (a ∣ X_{s}) = 1,$ and hence $E [Γ_{r_{lo}, s}^{-} ρ_{s} ∣ G_{s}] = Γ_{r_{lo}, s}^{-} .$ The remaining coordinates after the action at time $s$ are integrated with the probability kernel in Assumption 1; integrating first over the fibre containing $(Y_{s}, X_{s + 1}, H_{s + 1})$ preserves the expectation of any function of $G_{s}$ . Applying the displayed conditional identity successively for $s = r_{hi}, r_{hi} - 1, \dots, r_{lo}$ removes the factors one at a time and leaves the empty product, whose expectation is one. Taking $r_{lo} = r - k$ and $r_{hi} = r$ gives $\int j = r - k \prod r ρ_{j} (w) d μ_{obs} (w) = 1.$ Now $Z_{r} (k) (w) = Y_{r} (w) j = r - k \prod r ρ_{j} (w),$ and $∣ Y_{r} ∣ \leq 1$ almost surely by Assumption 4. Therefore $\int ∣ Z_{r} (k) ∣ d μ_{obs} \leq \int j = r - k \prod r ρ_{j} (w) d μ_{obs} (w) = 1.$
It remains to bound the cross moment. For the two overlapping windows define $W_{t} := {j \in {0, \dots, T - 1} : t - k \leq j \leq t}, W_{u} := {j \in {0, \dots, T - 1} : u - k \leq j \leq u}, W_{\cup} := W_{t} \cup W_{u} .$ Since $t$ and $u$ are elements of ${0, \dots, T - 1}$ , their integer values are valid finite-time endpoints. The assumptions $k \leq t$ , $u = t + h$ , and $h \leq k$ give $t - k \leq u$ and identify the union in the formal finite index set as $W_{t} \cup W_{u} = {j \in {0, \dots, T - 1} : t - k \leq j \leq u} .$ The same endpoint inequalities give the cardinality bound for the overlap, $∣ W_{t} \cap W_{u} ∣ \leq k + 1 - h .$ Using the same zero-cell ratio $ρ_{j}$ as above, the policy-vector and overlap parts of Assumptions 2 and 5, together with $L \geq 1$ , give $0 \leq ρ_{j} (w) \leq L for every j, w .$ Therefore $j \in W_{t} \cap W_{u} \prod ρ_{j} (w) \leq L^{∣ W_{t} \cap W_{u} ∣} \leq L^{k + 1 - h},$ the second inequality by the cardinality bound $∣ W_{t} \cap W_{u} ∣ \leq k + 1 - h$ together with the monotonicity of $n \mapsto L^{n}$ for $L \geq 1$ ; the intersection need not have exactly $k + 1 - h$ indices, so this step is an inequality and not an identity.
Since $∣ Y_{t} ∣ \leq 1$ and $∣ Y_{u} ∣ \leq 1$ almost surely, $∣ Z_{t} (k) Z_{u} (k) ∣ \leq j \in W_{t} \prod ρ_{j} j \in W_{u} \prod ρ_{j} .$ The product over two windows decomposes into the product over the union times the product over the overlap: $j \in W_{t} \prod ρ_{j} j \in W_{u} \prod ρ_{j} = j \in W_{\cup} \prod ρ_{j} j \in W_{t} \cap W_{u} \prod ρ_{j} .$ Combining this with the previous display gives the pointwise almost-sure bound $∣ Z_{t} (k) Z_{u} (k) ∣ \leq L^{k + 1 - h} j \in W_{\cup} \prod ρ_{j} .$ The finite-window identity from the preceding step rewrites the union product in exactly chronological form: $W_{\cup} = {j \in {0, \dots, T - 1} : t - k \leq j \leq u}, j \in W_{\cup} \prod ρ_{j} (w) = j = t - k \prod u ρ_{j} (w) .$ The interval-normalization argument proved above applies with lower endpoint $t - k$ and upper endpoint $u$ . The lower endpoint is nonnegative because $k \leq t$ , the upper endpoint is in ${0, \dots, T - 1}$ because $u$ is a time index, and $t - k \leq u$ follows from $u = t + h$ . Thus $\int j \in W_{\cup} \prod ρ_{j} (w) d μ_{obs} (w) = \int j = t - k \prod u ρ_{j} (w) d μ_{obs} (w) = 1.$ Hence $\int ∣ Z_{t} (k) Z_{u} (k) ∣ d μ_{obs} \leq L^{k + 1 - h} .$
Finally, $1 \leq L^{k + 1 - h},$ because $L \geq 1$ . The covariance reduction from Step 2, applied with $B = L^{k + 1 - h}$ , yields $Cov_{obs} (Z_{t} (k), Z_{u} (k)) \leq 2 L^{k + 1 - h},$ which is the claimed bound.

∎

The analysis of separated windows uses both the observed law and the full trajectory law. The next definitions fix those probability spaces and the observation map connecting them.

Definition 26 [synth_15] (Observed-trajectory law).

For a raw POMDP experiment $M = (X, H, K, b, e, L (O_{T}))$ , let $P_{M}^{obs} := Law_{M}^{obs}$ denote the probability law of the observed trajectory $O_{T} = ((X_{t}, A_{t}, Y_{t}))_{t = 0}^{T - 1}$ induced by the behavior-policy experiment $M$ . Equivalently, under $P_{M}^{obs}$ , $S_{0} \sim d_{b}$ , $A_{t} \sim b (\cdot ∣ X_{t})$ , and $K$ generates $(Y_{t}, S_{t + 1})$ from $(S_{t}, A_{t})$ , with only $(X_{t}, A_{t}, Y_{t})_{t = 0}^{T - 1}$ retained.

Definition 27 [synth_17] (Full trajectory law and observation map).

For a raw POMDP experiment $M = (X, H, K, b, e, L (O_{T}))$ with joint state space $S = X \times H$ , let $P_{M}$ denote the full behavior-policy trajectory law on $(S_{0}, A_{0}, Y_{0}, S_{1}, \dots, A_{T - 1}, Y_{T - 1}, S_{T}) .$ Under $P_{M}$ , the initial state satisfies $S_{0} \sim d_{b}$ , each action is drawn as $A_{t} \sim b (\cdot ∣ X_{t})$ , and the kernel $K$ generates $(Y_{t}, S_{t + 1})$ from $(S_{t}, A_{t})$ , for $t = 0, \dots, T - 1$ .

Let $O$ be the observation map from a full trajectory to its observed trajectory, $O (S_{0}, A_{0}, Y_{0}, \dots, A_{T - 1}, Y_{T - 1}, S_{T}) := ((X_{t}, A_{t}, Y_{t}))_{t = 0}^{T - 1},$ where $S_{t} = (X_{t}, H_{t})$ . The observed behavior-policy law is the pushforward $P_{M}^{obs} := P_{M} \circ O^{- 1} .$

Together, Definitions 26 and 27 let the covariance calculation move between observable scores and latent-state transition operators. The observed law is the sampling law for the estimator, while the full law retains the latent state needed to apply contraction after a gap.

Lemma 10 [lem:phiw-separated-window-peeling] (Separated window peeling).

Let $M$ be a raw POMDP experiment with trajectory length $T$ , and let $L$ , the policy-overlap constant, be real. Let $Z_{s} (k)$ denote the depth- $k$ partial-history weighted reward score from Definition 16. Suppose:

(Model conditions.) $M$ satisfies Assumptions 2, 5, 1, and 4 with overlap constant $L$ .

(Overlap scale.) $1 \leq L$ .

(Mature earlier window.) The history depth $k$ satisfies $k \leq t$ .

(Separated future window.) For an epoch $t$ and a gap $γ \in N$ , the epoch $u = t + 1 + γ + k$ lies in the observed horizon, equivalently $t + 1 + γ + k < T$ .

For $p \in {b, e}$ , let $P_{p}$ be the policy-induced transition operator obtained from the reward-marginalized kernel under policy $p$ , let $g_{e} (s) = a \sum e (a ∣ x (s)) \int y K (d y \times S ∣ s, a),$ and put $Φ_{γ, k} (s) = P_{b}^{γ} P_{e}^{k} g_{e} (s) .$ Then, with $P_{M}^{obs}$ the observed law induced by the observed trajectory map $O$ , $\int Z_{t} (k) (w) Z_{u} (k) (w) d P_{M}^{obs} (w) = \int Z_{t} (k) (O (τ)) Φ_{γ, k} (S_{t + 1} (τ)) d P_{M} (τ) .$

⊢ Lean

Proof of Lemma 10.

Set $ι_{0} = t + 1, ι_{1} = t + 1 + γ, ι_{2} = ι_{1} + k = u .$ The horizon condition gives $ι_{2} < T$ , and $k \leq t$ makes the earlier score mature. For a full trajectory $τ$ , write the joint state at epoch $m$ as $S_{m} (τ) := (X_{m} (τ), H_{m} (τ))$ and write $hist_{m} (τ) = ((S_{j} (τ))_{j < m}, (A_{j} (τ), Y_{j} (τ))_{j < m}, S_{m} (τ))$ for the state-and-epoch history through epoch $m$ . For an observed trajectory $w$ , write its epoch- $j$ coordinates as $w_{j} = (x_{j} (w), a_{j} (w), y_{j} (w)) .$ Define the totalized one-step ratio, for an observed state $x$ and action $a$ , by $ρ (a ∣ x) := {0, e (a ∣ x) / b (a ∣ x), b (a ∣ x) = 0, b (a ∣ x) > 0, ρ_{j} (w) := ρ (a_{j} (w) ∣ x_{j} (w)) .$ For $0 \leq j_{lo} \leq j_{hi} \leq T - 1$ , put $rat_{j_{lo} : j_{hi}}^{<} (τ) := j = j_{lo} \prod j_{hi} - 1 ρ_{j} (O (τ)), rat_{j_{hi} : j_{hi}}^{<} (τ) := 1.$ Thus the superscript $<$ means that the upper endpoint is excluded: the largest ratio in a nonempty block is $ρ_{j_{hi} - 1}$ . In particular, $rat_{ι_{1} : ι_{2}}^{<}$ stops at $ρ_{u - 1}$ , while the terminal factor $ρ_{u}$ is written separately below. The target-policy one-step reward regression is $g_{e} (s) = a \in A \sum e (a ∣ x (s)) \int y K (d y \times S ∣ s, a),$ and the separated future function is $Φ_{γ, k} = P_{b}^{γ} P_{e}^{k} g_{e}$ . In this proof the policy-induced transition operator is the reward-marginalized kernel $(P_{p} F) (s) = a \in A \sum p (a ∣ x (s)) \int F (s^{'}) K (d y \times d s^{'} ∣ s, a), p \in {b, e},$ for every bounded $F : S \to R$ .

Since $P_{M}^{obs}$ is the pushforward of $P_{M}$ under $O$ , $\int Z_{t} (k) (w) Z_{u} (k) (w) d P_{M}^{obs} (w) = \int Z_{t} (k) (O (τ)) Z_{u} (k) (O (τ)) d P_{M} (τ) .$ The measurability needed for this change of variables follows from the observable definition of $Z_{t} (k)$ in Definition 16.
Define lifted carriers on state histories by $Λ_{0} (hist_{ι_{0}} (τ)) := Z_{t} (k) (O (τ)),$ and, for $0 \leq ν < γ + k$ , let $Λ_{ν + 1}$ be the lift of $Λ_{ν}$ that forgets the last appended action, reward, and next state: $Λ_{ν + 1} (hist_{ι_{0} + ν + 1} (τ)) := Λ_{ν} (hist_{ι_{0} + ν} (τ)) .$ The base carrier is well defined because $Z_{t} (k)$ uses only rewards, actions, and observed states through epoch $t$ , while $t < ι_{0}$ ; hence every lifted carrier satisfies $Λ_{ν} (hist_{ι_{0} + ν} (τ)) = Z_{t} (k) (O (τ)) (0 \leq ν \leq γ + k) .$ The depth- $k$ window in $Z_{u} (k)$ is inclusive. Since $u = ι_{2}$ and $u - k = ι_{1}$ , the half-open block contains the factors with $ι_{1} \leq j < ι_{2}$ , and the endpoint contributes the separate terminal factor: $Z_{u} (k) (O (τ)) = Y_{ι_{2}} (τ) rat_{ι_{1} : ι_{2}}^{<} (τ) ρ_{ι_{2}} (O (τ)) .$ Therefore $\int Z_{t} (k) (O (τ)) Z_{u} (k) (O (τ)) d P_{M} (τ) = \int Λ_{γ + k} (hist_{ι_{2}} (τ)) rat_{ι_{1} : ι_{2}}^{<} (τ) ρ_{ι_{2}} (O (τ)) Y_{ι_{2}} (τ) d P_{M} (τ) .$ The totalized ratios obey $0 \leq ρ (a ∣ x) \leq L$ : on a positive behavior cell this is the domination inequality divided by $b (a ∣ x) > 0$ , and on a zero cell it follows from the definition of $ρ$ . Hence finite state spaces, Assumption 4, and the finite product bound $\prod_{j} ρ_{j} \leq L^{# {j}}$ make all displayed integrands integrable. Empty behavior or target blocks are interpreted as products equal to one; in particular, when $k = 0$ the block $rat_{ι_{1} : ι_{2}}^{<}$ is empty and the endpoint factor $ρ_{ι_{2}} (O (τ))$ is still present.
The terminal reward peel is $\int Λ_{γ + k} (hist_{ι_{2}}) rat_{ι_{1} : ι_{2}}^{<} ρ_{ι_{2}} Y_{ι_{2}} d P_{M} = \int Λ_{γ + k} (hist_{ι_{2}}) rat_{ι_{1} : ι_{2}}^{<} g_{e} (S_{ι_{2}}) d P_{M} .$ Indeed, condition first on the history and action at $ι_{2}$ . Assumption 1 supplies the conditional law of $(Y_{ι_{2}}, S_{ι_{2} + 1})$ as $K (\cdot ∣ S_{ι_{2}}, a)$ . Applying this conditional-law identity to the measurable test function $(y, s^{'}) \mapsto y$ , with integrability supplied by Assumption 4, and then marginalizing over the successor state gives $E [Y_{ι_{2}} ∣ hist_{ι_{2}}, A_{ι_{2}} = a] = \int y K (d y \times S ∣ S_{ι_{2}}, a) .$ By Assumption 2, the action is drawn from $b (\cdot ∣ X_{ι_{2}})$ . For every observed state $x$ and action $a$ , the totalized ratio satisfies $b (a ∣ x) ρ (a ∣ x) = e (a ∣ x) .$ Indeed, if $b (a ∣ x) > 0$ , this is the definition of $ρ$ . If $b (a ∣ x) = 0$ , then the domination inequality in Assumption 5 gives $e (a ∣ x) \leq 0$ , while the target policy has nonnegative coordinates, so $e (a ∣ x) = 0$ , matching the zero-cell value. Applying this identity at $x = X_{ι_{2}}$ and summing over $a$ gives $a \in A \sum e (a ∣ X_{ι_{2}}) \int y K (d y \times S ∣ S_{ι_{2}}, a) = g_{e} (S_{ι_{2}}),$ by the displayed definition of $g_{e}$ .
For every $ι_{1} \leq m < ι_{2}$ , put $ν = m - ι_{0}$ , so $γ \leq ν < γ + k$ . For every bounded $F : S \to R$ , the target-ratio peeling step is $\int Λ_{ν + 1} (hist_{m + 1}) rat_{ι_{1} : m + 1}^{<} F (S_{m + 1}) d P_{M} = \int Λ_{ν} (hist_{m}) rat_{ι_{1} : m}^{<} (P_{e} F) (S_{m}) d P_{M} .$ The first target peel has $m = ι_{1}$ and lowers the carrier from $Λ_{γ + 1}$ to $Λ_{γ}$ ; the last has $m = ι_{2} - 1$ and lowers $Λ_{γ + k}$ to $Λ_{γ + k - 1}$ . Iterating the displayed identity backward over these values of $m$ therefore removes exactly the $k$ target-window ratios. Indeed, $Λ_{ν + 1}$ is the lift of $Λ_{ν}$ , and $rat_{ι_{1} : m + 1}^{<} = rat_{ι_{1} : m}^{<} ρ_{m} .$ Conditioning on the history and action at $m$ , Assumption 1 gives the transition integral over $(Y_{m}, S_{m + 1})$ , and the reward coordinate is integrated out: $a \in A \sum b (a ∣ X_{m}) ρ (a ∣ X_{m}) \int F (s^{'}) K (d y \times d s^{'} ∣ S_{m}, a) = a \in A \sum e (a ∣ X_{m}) \int F (s^{'}) K (d y \times d s^{'} ∣ S_{m}, a) .$ The ratio identity displayed in the terminal peel justifies the replacement of $b ρ$ by $e$ , including zero behavior cells, and the last display is exactly $(P_{e} F) (S_{m})$ . Iterating for $m = ι_{2} - 1, \dots, ι_{1}$ , starting from $F = g_{e}$ , yields $\int Λ_{γ + k} (hist_{ι_{2}}) rat_{ι_{1} : ι_{2}}^{<} g_{e} (S_{ι_{2}}) d P_{M} = \int Λ_{γ} (hist_{ι_{1}}) (P_{e}^{k} g_{e}) (S_{ι_{1}}) d P_{M} .$ When $k = 0$ , this iteration is empty and the displayed identity reduces to the same integrand at $ι_{1} = ι_{2}$ .
For every $ι_{0} \leq m < ι_{1}$ , put $ν = m - ι_{0}$ . For every bounded $F : S \to R$ , the behavior-gap peeling step is $\int Λ_{ν + 1} (hist_{m + 1}) F (S_{m + 1}) d P_{M} = \int Λ_{ν} (hist_{m}) (P_{b} F) (S_{m}) d P_{M} .$ Here $hist_{m + 1}$ is the history obtained from $hist_{m}$ by appending $(A_{m}, Y_{m}, S_{m + 1})$ , and $Λ_{ν + 1}$ is the lift of $Λ_{ν}$ , so $Λ_{ν + 1} (hist_{m + 1}) = Λ_{ν} (hist_{m})$ . Conditioning on the history and action at $m$ , Assumption 2 supplies the behavior action law, while Assumption 1 supplies the joint law of $(Y_{m}, S_{m + 1})$ given $(S_{m}, A_{m})$ . Integrating out the reward coordinate gives, for each history through $m$ , $a \in A \sum b (a ∣ X_{m}) \int F (s^{'}) K (d y \times d s^{'} ∣ S_{m}, a) = (P_{b} F) (S_{m}),$ which is the reward-marginalized behavior transition appearing in the displayed identity. Iterating over $m = ι_{1} - 1, \dots, ι_{0}$ with $F = P_{e}^{k} g_{e}$ gives $\int Λ_{γ} (hist_{ι_{1}}) (P_{e}^{k} g_{e}) (S_{ι_{1}}) d P_{M} = \int Λ_{0} (hist_{ι_{0}}) (P_{b}^{γ} P_{e}^{k} g_{e}) (S_{ι_{0}}) d P_{M} .$ When $γ = 0$ , this iteration is empty. Since $Λ_{0} (hist_{ι_{0}} (τ)) = Z_{t} (k) (O (τ))$ , $S_{ι_{0}} = S_{t + 1}$ , and $Φ_{γ, k} = P_{b}^{γ} P_{e}^{k} g_{e}$ , combining the preceding displays proves $\int Z_{t} (k) (w) Z_{u} (k) (w) d P_{M}^{obs} (w) = \int Z_{t} (k) (O (τ)) Φ_{γ, k} (S_{t + 1} (τ)) d P_{M} (τ) .$

∎

Lemma 10 rewrites a separated score product as an earlier observable score multiplied by a future continuation value evaluated at $S_{t + 1}$ . This identity supplies the point where the gap between windows becomes a behavior-transition power, followed by a target-policy window.

Lemma 11 [lem:behavior-support-operator-congruence] (Behavior-support operator congruence).

Let $M \in M_{T} (t_{0}, ζ, C)$ satisfy Definition 10, with behavior policy $b$ , target policy $e$ , and the policy-overlap condition in Assumption 5. For $p \in {b, e}$ , write $P_{p} (s, s^{'}) = a \in {0, 1} \sum p (a ∣ x (s)) K ({S^{'} = s^{'}} ∣ s, a),$ and let $P_{p}^{0} h = h$ and $P_{p}^{m + 1} h = P_{p} (P_{p}^{m} h)$ . Let $d_{b}$ denote the behavior stationary law of $P_{b}$ .

Assume:

(Policy choice.) The policy $p$ is either $b$ or $e$ .

(Terminal agreement.) The functions $f, g : S \to R$ satisfy $f (s) = g (s)$ for every joint state $s$ with $d_{b} (s) > 0$ .

(Initial support.) The joint state $s$ satisfies $d_{b} (s) > 0$ .

Then, for every $m \in N$ , $(P_{p}^{m} f) (s) = (P_{p}^{m} g) (s) .$

⊢ Lean

Proof of Lemma 11.

Fix a policy $p \in {b, e}$ . The argument first records the support-closure fact used at the induction step. For joint states $σ, τ \in S$ and an action $a \in {0, 1}$ , write $κ_{σ, τ} (a) := K ({S^{'} = τ} ∣ σ, a) .$ Then $P_{p} (σ, τ) = a \in {0, 1} \sum p (a ∣ x (σ)) κ_{σ, τ} (a) .$

Suppose $d_{b} (σ) > 0$ and $P_{p} (σ, τ) > 0$ . We claim that $d_{b} (τ) > 0.$ First show that $P_{b} (σ, τ) > 0$ . If $p = b$ , this is immediate. If $p = e$ , suppose instead that $P_{b} (σ, τ) = 0$ . By Definition 10, the behavior policy is a probability vector and $K$ is a Markov kernel, so every summand in $P_{b} (σ, τ) = a \in {0, 1} \sum b (a ∣ x (σ)) κ_{σ, τ} (a)$ is nonnegative. Hence, for each action $a$ , $b (a ∣ x (σ)) κ_{σ, τ} (a) = 0.$ For such an $a$ , if $κ_{σ, τ} (a) = 0$ then $e (a ∣ x (σ)) κ_{σ, τ} (a) = 0.$ If $κ_{σ, τ} (a) > 0$ , the preceding zero product gives $b (a ∣ x (σ)) = 0$ ; by Assumption 5, $e (a ∣ x (σ)) \leq L b (a ∣ x (σ)) = 0,$ and the target policy is nonnegative by Definition 10, so $e (a ∣ x (σ)) = 0$ . Thus again $e (a ∣ x (σ)) κ_{σ, τ} (a) = 0.$ Summing over $a$ gives $P_{e} (σ, τ) = a \in {0, 1} \sum e (a ∣ x (σ)) κ_{σ, τ} (a) = 0,$ contradicting $P_{p} (σ, τ) > 0$ in the case $p = e$ . Therefore $P_{b} (σ, τ) > 0$ .

By Definition 10, $d_{b}$ is stationary for $P_{b}$ , so $d_{b} (τ) = ω \in S \sum d_{b} (ω) P_{b} (ω, τ) .$ All terms in this sum are nonnegative. Therefore $0 < d_{b} (σ) P_{b} (σ, τ) \leq ω \in S \sum d_{b} (ω) P_{b} (ω, τ) = d_{b} (τ),$ which proves the claim.
We now prove the asserted identity by induction on $m$ . For $m = 0$ , $(P_{p}^{0} f) (s) = f (s) = g (s) = (P_{p}^{0} g) (s),$ because $d_{b} (s) > 0$ and $f = g$ on the behavior support.

Assume the statement holds for $m$ : for every joint state $σ$ with $d_{b} (σ) > 0$ , $(P_{p}^{m} f) (σ) = (P_{p}^{m} g) (σ) .$ Let $s$ satisfy $d_{b} (s) > 0$ . Using the recursive definition of the iterated operator, $(P_{p}^{m + 1} f) (s) = τ \in S \sum P_{p} (s, τ) (P_{p}^{m} f) (τ), (P_{p}^{m + 1} g) (s) = τ \in S \sum P_{p} (s, τ) (P_{p}^{m} g) (τ) .$ Fix $τ \in S$ . If $P_{p} (s, τ) = 0$ , then the two $τ$ -summands are both zero. If $P_{p} (s, τ) \neq = 0$ , the row nonnegativity of $P_{p}$ gives $P_{p} (s, τ) > 0$ . The support-closure claim from the previous step then gives $d_{b} (τ) > 0$ , so the induction hypothesis yields $(P_{p}^{m} f) (τ) = (P_{p}^{m} g) (τ) .$ Multiplying by $P_{p} (s, τ)$ , the two $τ$ -summands agree. Since this holds for every $τ$ , the sums agree: $(P_{p}^{m + 1} f) (s) = (P_{p}^{m + 1} g) (s) .$ The induction proves the result for every $m \in N$ .

∎

The support congruence in Lemma 11 keeps transition-operator calculations on the behavior support. Because the overlap condition makes target transitions compatible with behavior support, functions that agree on the behavior-stationary support continue to agree after iterating either policy operator from a supported state.

Lemma 12 [lem:phiw-separated-window-covariance] (Separated window covariance).

Let $T, n_{X}, n_{H}, k, h \in N$ , let $t_{0}, ζ, C \in R$ , and let $M \in M_{T} (t_{0}, ζ, C)$ be a raw POMDP experiment as in Definition 10, with the uniform contraction restriction of Assumption 6. Write the contraction factor as $α = e^{- 1/ t_{0}}$ . Let $t, u \in {0, \dots, T - 1}$ satisfy $k \leq t, u = t + h, k + 1 \leq h .$ Then the depth- $k$ scores $Z_{t} (k)$ of Definition 16, evaluated under the observed trajectory law of $M$ , satisfy $Cov_{obs} (Z_{t} (k), Z_{u} (k)) \leq 2 α^{h - k - 1} .$

⊢ Lean

Proof of Lemma 12.

Put $γ := h - k - 1.$ Then $0 \leq γ$ by $k + 1 \leq h$ , and $u = t + h = t + 1 + γ + k .$ Let $P_{M}$ denote the full trajectory law and $P_{M}^{obs}$ its pushforward under the observation map $O$ . For $p \in {b, e}$ , write $P_{p} (s, s^{'}) = a \in A \sum p (a ∣ x (s)) K ({S^{'} = s^{'}} ∣ s, a), g_{e} (s) = a \in A \sum e (a ∣ x (s)) \int y K (d y \times S ∣ s, a),$ and define $Φ_{γ, k} (s) := P_{b}^{γ} P_{e}^{k} g_{e} (s) .$

Since $L = exp (ζ)$ and $ζ > 0$ in Definition 10, $1 \leq L$ . The maturity and separation hypotheses give $t + 1 + γ + k < T$ , so Lemma 10 applies and yields $\int Z_{t} (k) (w) Z_{u} (k) (w) d P_{M}^{obs} (w) = \int Z_{t} (k) (O (τ)) Φ_{γ, k} (S_{t + 1} (τ)) d P_{M} (τ) .$ For the companion mean identity, let $r_{+} := t + 1$ and $q_{γ} := r_{+} + γ$ , so $u = q_{γ} + k$ . Since $u = q_{γ} + k$ , the depth- $k$ window in $Z_{u} (k)$ is exactly $q_{γ}, q_{γ} + 1, \dots, u$ . We claim $\int Z_{u} (k) (w) d P_{M}^{obs} (w) = \int Φ_{γ, k} (S_{t + 1} (τ)) d P_{M} (τ) .$ To prove this identity, condition first on $S_{q_{γ}}$ . Write $ρ (a, x) := {e (a ∣ x) / b (a ∣ x), 0, b (a ∣ x) > 0, b (a ∣ x) = 0.$ The zero-cell convention and Assumption 5 imply $e (a ∣ x) = 0$ whenever $b (a ∣ x) = 0$ . Hence, for each pre-terminal epoch $m_{win} \in {q_{γ}, \dots, u - 1}$ , $a \in A \sum b (a ∣ x (s)) ρ (a, x (s)) K ({S^{'} = s^{'}} ∣ s, a) = a \in A \sum e (a ∣ x (s)) K ({S^{'} = s^{'}} ∣ s, a) = P_{e} (s, s^{'}),$ while at the terminal epoch $a \in A \sum b (a ∣ x (s)) ρ (a, x (s)) \int y K (d y \times S ∣ s, a) = a \in A \sum e (a ∣ x (s)) \int y K (d y \times S ∣ s, a) = g_{e} (s) .$ Thus the target-weighted terminal window has conditional mean $P_{e}^{k} g_{e} (S_{q_{γ}})$ , with the terminal display alone applying when $k = 0$ . Conditioning backward over the gap epochs $t + 1, \dots, q_{γ} - 1$ , no score ratio appears, and Assumptions 2 and 1 give the behavior propagation $P_{b}^{γ}$ ; when $γ = 0$ , this is the identity operator. Bounded rewards and the finite state and action spaces justify the displayed integrals. Therefore the conditional mean given $S_{t + 1}$ is $P_{b}^{γ} P_{e}^{k} g_{e} (S_{t + 1}) = Φ_{γ, k} (S_{t + 1})$ , proving the claim. Also, by definition of $P_{M}^{obs}$ , $\int Z_{t} (k) (w) d P_{M}^{obs} (w) = \int Z_{t} (k) (O (τ)) d P_{M} (τ) .$ Thus the covariance equals the centered full-law product $Cov_{obs} (Z_{t} (k), Z_{u} (k)) = \int Z_{t} (k) (O (τ)) Φ_{γ, k} (S_{t + 1} (τ)) d P_{M} (τ) - (\int Z_{t} (k) (O (τ)) d P_{M} (τ)) (\int Φ_{γ, k} (S_{t + 1} (τ)) d P_{M} (τ)) .$ The covariance identity is legitimate because $Z_{t} (k), Z_{u} (k) \in L^{2} (P_{M}^{obs}) .$ Indeed, write the window product in $Z_{m} (k)$ as $R_{m}^{(k)} (w) := j \in {0, \dots, T - 1} : j \leq m, m - j \leq k \prod ρ_{j} (w), m \in {t, u} .$ The product contains at most $k + 1$ factors. The zero-cell convention and Assumption 5 make each factor nonnegative and at most $L$ on observed behavior-support cells, while Assumption 4 gives $∣ Y_{m} ∣ \leq 1$ almost surely. Hence $∣ Z_{m} (k) ∣ \leq L^{k + 1}$ almost surely under $P_{M}^{obs}$ , and the two scores are square-integrable.
The first score has unit $L^{1}$ norm: $\int ∣ Z_{t} (k) (O (τ)) ∣ d P_{M} (τ) \leq 1.$ Indeed, with $R_{t}^{(k)} (w) := j = t - k \prod t ρ_{j} (w),$ the zero-cell convention and Assumption 5 give $E_{obs} [R_{t}^{(k)}] = 1$ by peeling the ratios from $j = t$ down to $j = t - k$ : conditionally at each peeled epoch, $a \in A \sum b (a ∣ X_{j}) ρ_{j} (a, X_{j}) = a : b (a ∣ X_{j}) > 0 \sum e (a ∣ X_{j}) = a \in A \sum e (a ∣ X_{j}) = 1.$ By Assumption 4, $∣ Z_{t} (k) ∣ \leq R_{t}^{(k)}$ almost surely, which proves the displayed $L^{1}$ bound after passing between the observed law and the full law by the observation map.
Let $d_{b}$ be the stationary law of $P_{b}$ , and define the support-truncated regression on all of $S$ by $g_{e} (s) := {g_{e} (s), 0, d_{b} (s) > 0, d_{b} (s) = 0.$ For every $s$ with $d_{b} (s) > 0$ , $∣ g_{e} (s) ∣ \leq 1.$ To see this, fix such an $s$ . For an action $a$ with $b (a ∣ x (s)) > 0$ , consider epoch $0$ . The history before epoch $0$ is the unique empty history, so the full history-action singleton with current state $s$ and action $a$ has probability $d_{b} (s) b (a ∣ x (s)) > 0,$ by Assumption 3 and the epoch-zero action factorization in Assumption 2. Under the history-next-pair factorization supplied by Assumption 1, Assumption 4 gives, for almost every history-action pair, a kernel row whose reward coordinate is almost surely in $[- 1, 1]$ . Since this epoch-zero singleton has positive mass, that almost-sure row statement holds at $(s, a)$ , and hence $\int y K (d y \times S ∣ s, a) \leq 1.$ If $e (a ∣ x (s)) > 0$ , then Assumption 5 forces $b (a ∣ x (s)) > 0$ . Therefore $∣ g_{e} (s) ∣ \leq a \in A \sum e (a ∣ x (s)) \int y K (d y \times S ∣ s, a) \leq a \in A \sum e (a ∣ x (s)) = 1.$ For $d_{b} (s) = 0$ , the definition gives $∣ g_{e} (s) ∣ = 0$ . Consequently $s, s^{'} sup ∣ g_{e} (s) - g_{e} (s^{'}) ∣ \leq 2.$
Since $t_{0} > 0$ , $α = exp (- 1/ t_{0})$ satisfies $0 \leq α \leq 1$ . For a function $f : S \to R$ , write $osc (f) := s, s^{'} \in S sup ∣ f (s) - f (s^{'}) ∣.$ For each $p_{⋆} \in {b, e}$ , the rows of $P_{p_{⋆}}$ are probability vectors. Nonnegativity follows from the nonnegativity of the policy probabilities and of the kernel rows. For the row sums, $s^{'} \in S \sum P_{p_{⋆}} (s, s^{'}) = a \in A \sum p_{⋆} (a ∣ x (s)) s^{'} \in S \sum K ({S^{'} = s^{'}} ∣ s, a) = a \in A \sum p_{⋆} (a ∣ x (s)) = 1.$ Here Assumption 1 supplies the kernel row mass, Assumption 2 supplies the behavior probability vector when $p_{⋆} = b$ , and the target-carrier part of Assumption 5 supplies the target probability vector when $p_{⋆} = e$ .

We first convert the total-variation contraction in Assumption 6 into the oscillation contraction used below. Let $ν$ and $ν^{'}$ be probability vectors on the finite state space, and let $f : S \to R$ . Put $f_{m i n} := x \in S min f (x), f_{0} (x) := f (x) - f_{m i n} .$ Then $0 \leq f_{0} (x) \leq osc (f)$ for every $x$ , and, since $ν$ and $ν^{'}$ have equal total mass, $x \in S \sum f (x) (ν (x) - ν^{'} (x)) = x \in S \sum f_{0} (x) (ν (x) - ν^{'} (x)) .$ The layer-cake identity on the finite state space gives $x \in S \sum f_{0} (x) (ν (x) - ν^{'} (x)) = \int_{0}^{osc (f)} (ν {x : f_{0} (x) > λ_{lev}} - ν^{'} {x : f_{0} (x) > λ_{lev}}) d λ_{lev},$ and therefore $x \in S \sum f (x) (ν (x) - ν^{'} (x)) \leq osc (f) ∥ ν - ν^{'} ∥_{TV} .$ Now fix states $s, s^{'}$ . Applying Assumption 6 to the point masses $δ_{s}$ and $δ_{s^{'}}$ gives $∥ δ_{s} P_{p_{⋆}} - δ_{s^{'}} P_{p_{⋆}} ∥_{TV} \leq α ∥ δ_{s} - δ_{s^{'}} ∥_{TV} \leq α .$ Combining this with the preceding dual bound yields the one-step estimate $∣ P_{p_{⋆}} f (s) - P_{p_{⋆}} f (s^{'}) ∣ \leq α osc (f) .$ Taking the supremum over $s, s^{'}$ and iterating gives the finite-state oscillation form: whenever $osc (f) \leq B_{osc}$ for some $B_{osc} \geq 0$ , $osc (P_{p_{⋆}}^{m_{iter}} f) \leq α^{m_{iter}} B_{osc}, m_{iter} \in N .$ Applying this first with $p_{⋆} = e$ , $f = g_{e}$ , and $B_{osc} = 2$ , and then using $α^{k} \leq 1$ , gives $s, s^{'} sup ∣ P_{e}^{k} g_{e} (s) - P_{e}^{k} g_{e} (s^{'}) ∣ \leq 2.$ Applying it next with $p_{⋆} = b$ to $P_{e}^{k} g_{e}$ gives $s, s^{'} sup P_{b}^{γ} P_{e}^{k} g_{e} (s) - P_{b}^{γ} P_{e}^{k} g_{e} (s^{'}) \leq 2 α^{γ} .$
Define $Φ_{γ, k} (s) := P_{b}^{γ} P_{e}^{k} g_{e} (s) .$ Because $g_{e} = g_{e}$ on ${s : d_{b} (s) > 0}$ , Lemma 11 applied with policy $e$ gives, for every behavior-supported state $x$ , $P_{e}^{k} g_{e} (x) = P_{e}^{k} g_{e} (x) whenever d_{b} (x) > 0.$ This pointwise behavior-supported equality is the terminal-agreement hypothesis needed for the second application of Lemma 11, now with policy $b$ . Thus, for every initial state $s$ with $d_{b} (s) > 0$ , $Φ_{γ, k} (s) = P_{b}^{γ} P_{e}^{k} g_{e} (s) = P_{b}^{γ} P_{e}^{k} g_{e} (s) = Φ_{γ, k} (s) .$ We next record the behavior-stationarity identity used at time $t + 1$ . For every bounded $F : S \to R$ and every epoch $m_{time}$ , $\int F (S_{m_{time}} (τ)) d P_{M} (τ) = s \in S \sum d_{b} (s) F (s) .$ This follows by induction on $m_{time}$ . At the initial epoch the identity is Assumption 3. For the induction step, Assumption 2 and Assumption 1 give the behavior peeling identity $\int F (S_{m_{time} + 1} (τ)) d P_{M} (τ) = \int x \in S \sum P_{b} (S_{m_{time}} (τ), x) F (x) d P_{M} (τ) .$ Using the induction hypothesis with the bounded function $s \mapsto \sum_{x} P_{b} (s, x) F (x)$ , the right-hand side becomes $s \in S \sum d_{b} (s) x \in S \sum P_{b} (s, x) F (x) = x \in S \sum (s \in S \sum d_{b} (s) P_{b} (s, x)) F (x) = x \in S \sum d_{b} (x) F (x),$ where the last equality is the stationarity equation for the behavior stationary law $d_{b}$ , part of the model-class data in Definition 10. Taking the zero-support indicator $χ_{null} (s) := 1 {d_{b} (s) = 0}$ at $m_{time} = t + 1$ gives $\int 1 {d_{b} (S_{t + 1} (τ)) = 0} d P_{M} (τ) = 0,$ so $d_{b} (S_{t + 1} (τ)) > 0$ for $P_{M}$ -almost every $τ$ . Hence $Φ_{γ, k} (S_{t + 1} (τ)) = Φ_{γ, k} (S_{t + 1} (τ)) for P_{M} -almost every τ .$
Put $A (τ) := Z_{t} (k) (O (τ)), \overline{Φ} := \int Φ_{γ, k} (S_{t + 1} (τ)) d P_{M} (τ) .$ Using the almost-sure replacement from the previous step, both $\int A (τ) Φ_{γ, k} (S_{t + 1} (τ)) d P_{M} (τ) = \int A (τ) Φ_{γ, k} (S_{t + 1} (τ)) d P_{M} (τ)$ and $\int Φ_{γ, k} (S_{t + 1} (τ)) d P_{M} (τ) = \int Φ_{γ, k} (S_{t + 1} (τ)) d P_{M} (τ) = \overline{Φ}$ hold; the integrability needed for these replacements follows from the finite state space and the score bound above. The centered representation from the first step therefore gives $∣ Cov_{obs} (Z_{t} (k), Z_{u} (k)) ∣ = \int A (τ) (Φ_{γ, k} (S_{t + 1} (τ)) - \overline{Φ}) d P_{M} (τ) .$ The oscillation bound in the fourth step implies, for every $τ$ , $Φ_{γ, k} (S_{t + 1} (τ)) - \overline{Φ} \leq 2 α^{γ},$ because $\overline{Φ}$ is an average of values of $Φ_{γ, k}$ . Therefore, using the unit $L^{1}$ bound from the second step, $∣ Cov_{obs} (Z_{t} (k), Z_{u} (k)) ∣ \leq \int ∣ A (τ) ∣ 2 α^{γ} d P_{M} (τ) \leq 2 α^{γ} = 2 α^{h - k - 1} .$

∎

Together, Lemmas 8, 9, 10, 11, and 12 decompose the sampling fluctuation into a weighted-window term and a geometric dependence tail. The overlap condition controls the inflation inside the window, while contraction controls pairs whose windows are separated by at least one time step.

The next statement sums those covariance bounds over future times. It is the finite-sample accounting step that turns the local bounds into a variance inequality for the average score.

Lemma 13 [lem:phiw-future-covariance-sum] (Aggregate future-covariance bound).

Let $T \geq 1$ , $k \in N$ , $t_{0}, ζ, C \in R$ , and let $M \in M_{T} (t_{0}, ζ, C)$ be as in Definition 10. Write $L = e^{ζ}$ , $α = e^{- 1/ t_{0}}$ , and $T_{k} = {t : k \leq t \leq T - 1}$ . Then, for every $t \in T_{k}$ , the depth- $k$ scores of Definition 16 satisfy, under the observed trajectory law of $M$ , $u \in T_{k} t < u \sum Cov_{obs} (Z_{t} (k), Z_{u} (k)) \leq \frac{2 L ^{k + 1}}{L - 1} + \frac{2}{1 - α} .$

⊢ Lean

Proof of Lemma 13.

Fix $t \in T_{k}$ . By Definition 10, $0 < ζ$ and $0 < t_{0}$ , so $L = e^{ζ} > 1, 0 \leq α = e^{- 1/ t_{0}} < 1.$ The same membership supplies the model restrictions used by the covariance estimates below.

Define the future index set and its two lag classes by $U_{t} := {u \in T_{k} : t < u}, λ_{t, u} := u - t (u \in U_{t}),$ and $N_{t} := {u \in U_{t} : λ_{t, u} \leq k}, G_{t} := {u \in U_{t} : k + 1 \leq λ_{t, u}} .$ For each $u \in U_{t}$ , the lag $λ_{t, u}$ is a positive integer, hence exactly one of $λ_{t, u} \leq k$ and $k + 1 \leq λ_{t, u}$ holds. Thus $U_{t} = N_{t} \dot{\cup} G_{t} .$ Writing $Γ_{t} (u) := Cov_{obs} (Z_{t} (k), Z_{u} (k)), u \in U_{t},$ we get $u \in U_{t} \sum Γ_{t} (u) = u \in N_{t} \sum Γ_{t} (u) + u \in G_{t} \sum Γ_{t} (u) .$
For $u \in N_{t}$ , the lag identity is $u = t + λ_{t, u}$ and $1 \leq λ_{t, u} \leq k$ . Since $k \leq t$ and $L \geq 1$ , Lemma 9 gives $Γ_{t} (u) \leq 2 L^{k + 1 - λ_{t, u}} .$ Therefore $u \in N_{t} \sum Γ_{t} (u) \leq 2 L u \in N_{t} \sum L^{k - λ_{t, u}} .$ The map $j_{near} (u) := k - λ_{t, u}, u \in N_{t},$ is injective and takes its values in ${0, \dots, k - 1}$ . Since $L \geq 0$ , $u \in N_{t} \sum L^{k - λ_{t, u}} \leq 0 \leq j < k \sum L^{j} .$ For $L > 1$ , $0 \leq j < k \sum L^{j} = \frac{L ^{k} - 1}{L - 1} \leq \frac{L ^{k}}{L - 1} .$ Combining the last three displays yields $u \in N_{t} \sum Γ_{t} (u) \leq \frac{2 L ^{k + 1}}{L - 1} .$
For $u \in G_{t}$ , the lag identity is $u = t + λ_{t, u}$ and $k + 1 \leq λ_{t, u}$ . Hence Lemma 12 gives $Γ_{t} (u) \leq 2 α^{λ_{t, u} - k - 1} .$ Thus $u \in G_{t} \sum Γ_{t} (u) \leq 2 u \in G_{t} \sum α^{λ_{t, u} - k - 1} .$ The map $j_{sep} (u) := λ_{t, u} - k - 1, u \in G_{t},$ is injective and takes its values in ${0, \dots, T - 1}$ . Since $α \geq 0$ , $u \in G_{t} \sum α^{λ_{t, u} - k - 1} \leq 0 \leq j < T \sum α^{j} .$ Because $0 \leq α < 1$ , $0 \leq j < T \sum α^{j} \leq j = 0 \sum \infty α^{j} = \frac{1}{1 - α} .$ Consequently, $u \in G_{t} \sum Γ_{t} (u) \leq \frac{2}{1 - α} .$
Adding the two bounds over the disjoint decomposition of $U_{t}$ gives $u \in T_{k} t < u \sum Cov_{obs} (Z_{t} (k), Z_{u} (k)) = u \in U_{t} \sum Γ_{t} (u) \leq \frac{2 L ^{k + 1}}{L - 1} + \frac{2}{1 - α} .$

∎

Lemma 13 supplies the two terms that reappear in the assembled variance bound. The geometric sum over overlapping lags gives the policy-overlap contribution, and the contraction tail gives the dependence contribution.

Lemma 14 [lem:phiw-raw-variance] (Raw variance assembly).

Let $T$ and $k$ be nonnegative integers, let $t_{0}, ζ, C \in R$ , and let $M$ be a raw POMDP experiment. Write $L = e^{ζ}$ for the policy-overlap factor and $α = e^{- 1/ t_{0}}$ for the contraction factor. Suppose that

(Horizon.) $T \geq 1$ .

(Model class.) $M \in M_{T} (t_{0}, ζ, C)$ , with $M_{T} (t_{0}, ζ, C)$ as in Definition 10.

(History depth.) $k \leq ⌊ T /2 ⌋$ .

For the depth- $k$ scores $Z_{t} (k)$ of Definition 16, form the unclipped partial-history average $\frac{1}{T - k} t = k \sum T - 1 Z_{t} (k) .$ Under the observed trajectory law obtained by projecting the full trajectory law of $M$ onto observed histories, its variance satisfies $Var (\frac{1}{T - k} t = k \sum T - 1 Z_{t} (k)) \leq \frac{2}{T} {L^{k + 1} (1 + \frac{4}{L - 1}) + \frac{4}{1 - α}} .$

⊢ Lean

Proof of Lemma 14.

Let $P_{obs}$ denote the observed trajectory law and let $E_{obs}$ denote expectation under this law. Let $P_{full}$ denote the full trajectory law carried by $M$ , and let $E_{full}$ denote expectation under that law. Put $T_{k} := {t \in {0, \dots, T - 1} : k \leq t}, B_{k} := L^{k + 1} (1 + \frac{4}{L - 1}) + \frac{4}{1 - α} .$ For $t \in T_{k}$ , write $Γ_{t}^{(k)} := 0 \leq j \leq T - 1 j \leq t, t - j \leq k \prod ρ_{j}, Z_{t} (k) = Y_{t} Γ_{t}^{(k)} .$ Since $M \in M_{T} (t_{0}, ζ, C)$ , Definition 10 gives $t_{0} > 0$ , $ζ > 0$ , hence $L = e^{ζ} > 1, 0 \leq α = e^{- 1/ t_{0}} < 1.$ Also $k \leq ⌊ T /2 ⌋$ implies $∣ T_{k} ∣ = T - k > 0, \frac{T}{2} \leq T - k .$ In particular $B_{k} \geq 0$ . The raw average is $\frac{1}{T - k} t = k \sum T - 1 Z_{t} (k) = \frac{1}{T - k} t \in T_{k} \sum Z_{t} (k) .$

For each $t \in T_{k}$ , Lemma 8 applies with the sequential-ignorability, policy-overlap, kernel-law, and bounded-reward clauses supplied by Definition 10, together with $L = e^{ζ} \geq 1$ and $k \leq t$ . It gives $E_{obs} [Z_{t} (k)^{2}] \leq L^{k + 1} .$ Using the standard identity $Var_{obs} (Z_{t} (k)) = E_{obs} [Z_{t} (k)^{2}] - (E_{obs} Z_{t} (k))^{2},$ we obtain $Var_{obs} (Z_{t} (k)) \leq L^{k + 1} .$
Fix $t \in T_{k}$ . The aggregate covariance estimate in Lemma 13 applies to the same model class and the same future index set $T_{k}$ . Therefore $u \in T_{k} t < u \sum Cov_{obs} (Z_{t} (k), Z_{u} (k)) \leq \frac{2 L ^{k + 1}}{L - 1} + \frac{2}{1 - α} .$
Now expand the variance of the finite sum: $Var_{obs} (t \in T_{k} \sum Z_{t} (k)) = t \in T_{k} \sum u \in T_{k} \sum Cov_{obs} (Z_{t} (k), Z_{u} (k)) .$ Bounding each covariance by its absolute value and then using covariance symmetry gives $Var_{obs} (t \in T_{k} \sum Z_{t} (k)) \leq t \in T_{k} \sum Var_{obs} (Z_{t} (k)) + 2 t \in T_{k} \sum u \in T_{k} t < u \sum Cov_{obs} (Z_{t} (k), Z_{u} (k)) .$ Using the diagonal and future-sum bounds, $Var_{obs} (t \in T_{k} \sum Z_{t} (k)) \leq ∣ T_{k} ∣ L^{k + 1} + 2∣ T_{k} ∣ (\frac{2 L ^{k + 1}}{L - 1} + \frac{2}{1 - α}) = (T - k) B_{k} .$
Finally, $Var_{obs} (\frac{1}{T - k} t \in T_{k} \sum Z_{t} (k)) = \frac{1}{( T - k ) ^{2}} Var_{obs} (t \in T_{k} \sum Z_{t} (k)) \leq \frac{B _{k}}{T - k} .$ Since $T /2 \leq T - k$ , $T > 0$ , and $B_{k} \geq 0$ , $\frac{B _{k}}{T - k} \leq \frac{2 B _{k}}{T} .$ Substituting the definition of $B_{k}$ gives $Var_{obs} (\frac{1}{T - k} t = k \sum T - 1 Z_{t} (k)) \leq \frac{2}{T} {L^{k + 1} (1 + \frac{4}{L - 1}) + \frac{4}{1 - α}} .$

∎

The two terms in Lemma 14 have distinct origins: multiplying observable policy ratios over a window produces the $L^{k + 1}$ term, while serial dependence contributes the contraction term. The bound is stated for the unclipped average because the displayed variance is the component used before the clipped estimator’s squared-error risk is assembled.

The next statement supplies the fixed-radius upper bound for the partial-history importance-weighted estimator. It gives both the rate bound for the balanced depth and the corresponding variance calculation for the unclipped average.

Lemma 15 [lem:phiw-upper] (Balanced PHIW upper bound).

Fix constants satisfying

(Mixing scale.) $t_{0} > 0$ .

(Policy-overlap scale.) $ζ > 0$ .

(Latent-overlap radius.) $C \geq 1$ .

Let $k_{T}$ be the balanced history depth in Definition 17, and set $β = \frac{2}{2 + t _{0} ζ} .$ Then there exist constants $c^{⋆} > 0$ and $T_{⋆} \in N$ such that, for every $T \geq T_{⋆}$ , the estimator-specific worst-case squared risk from Definition 13 of the observable PHIW estimator $θ_{k_{T}}$ from Definition 16 satisfies $R_{T} (t_{0}, ζ, C; θ_{k_{T}}) \leq c^{⋆} T^{- β} .$ Moreover, for every $T \geq T_{⋆}$ , every finite alphabet sizes $n_{X}, n_{H}$ , and every raw POMDP experiment $M \in M_{T} (t_{0}, ζ, C)$ , the variance under the observed trajectory law induced by $M$ of the unclipped partial-history average $\frac{1}{T - k _{T}} t = k_{T} \sum T - 1 Z_{t} (k_{T})$ satisfies $Var (\frac{1}{T - k _{T}} t = k_{T} \sum T - 1 Z_{t} (k_{T})) \leq \frac{2}{T} [e^{ζ (k_{T} + 1)} (1 + \frac{4}{e ^{ζ} - 1}) + \frac{4}{1 - e ^{- 1/ t_{0}}}] .$

⊢ Lean

Proof of Lemma 15.

Define $α = exp (- 1/ t_{0}), L = exp (ζ), β = \frac{2}{2 + t _{0} ζ}, q_{C} = \frac{C - 1}{C} .$ The assumptions $t_{0} > 0$ and $ζ > 0$ give $0 < α < 1, 1 < L, 0 < β < 1.$ Also $C \geq 1$ implies $0 \leq q_{C} \leq 1, q_{C}^{2} \leq 1.$ Let $d = 2 lo g (1/ α) + lo g L = \frac{2}{t _{0}} + ζ .$ Then $d > 0$ and $β = \frac{2/ t _{0}}{d} .$ Since $lo g T = o (T)$ , there is an integer $T_{bal} \geq 1$ such that, for every $T \geq T_{bal}$ , $0 \leq \frac{lo g T}{d} \leq \frac{T}{2} .$ For such $T$ , Definition 17 gives $k_{T} = ⌊ \frac{lo g T}{d} ⌋,$ and hence $\frac{lo g T}{d} - 1 < k_{T} \leq \frac{lo g T}{d} .$ Therefore $α^{2 k_{T}} = exp (- 2 k_{T} / t_{0}) \leq exp (2/ t_{0}) T^{- β},$ and $\frac{L ^{k_{T} + 1}}{T} = exp {(k_{T} + 1) ζ - lo g T} \leq exp (ζ) T^{- β} .$ With $c_{bal} := exp (2/ t_{0}) + exp (ζ) > 0,$ we have, for every $T \geq T_{bal}$ , $α^{2 k_{T}} + \frac{L ^{k_{T} + 1}}{T} \leq c_{bal} T^{- β} .$
Set $A := 1 + \frac{4}{L - 1}, B := \frac{4}{1 - α}, D := max {4, 2 A},$ and define $c^{⋆} := D c_{bal} + 2 B, T_{⋆} := T_{bal} .$ Since $L > 1$ and $α < 1$ , the constants $A, B, D$ are positive, and so $c^{⋆} > 0$ .
Fix $T \geq T_{⋆}$ . Then $1 \leq T$ , $k_{T} \leq T /2$ , and $k_{T} < T$ . For every $M \in M_{T} (t_{0}, ζ, C)$ , Lemma 34, applied with $k = k_{T}$ , gives $E_{M} [{θ_{k_{T}} - θ (K, e)}^{2}] \leq 4 q_{C}^{2} α^{2 k_{T}} + \frac{2}{T} (L^{k_{T} + 1} A + B) .$ Using $q_{C}^{2} \leq 1$ , $D \geq 4$ , and $D \geq 2 A$ , $4 q_{C}^{2} α^{2 k_{T}} + \frac{2}{T} (L^{k_{T} + 1} A + B) \leq D (α^{2 k_{T}} + \frac{L ^{k_{T} + 1}}{T}) + 2 B T^{- 1} .$ The balance bound from the first step and the inequality $T^{- 1} \leq T^{- β}$ , valid because $T \geq 1$ and $0 < β < 1$ , yield $E_{M} [{θ_{k_{T}} - θ (K, e)}^{2}] \leq D c_{bal} T^{- β} + 2 B T^{- β} = c^{⋆} T^{- β} .$ There are two cases. If the indexed class $M_{T} (t_{0}, ζ, C)$ is nonempty, the pointwise bound over every member of the class, together with the estimator-specific risk definition in Definition 13, gives $R_{T} (t_{0}, ζ, C; θ_{k_{T}}) \leq c^{⋆} T^{- β} .$ If the indexed class is empty, the real-valued worst-case-risk functional used for $R_{T}$ has empty supremum equal to $0$ . Since $c^{⋆} > 0$ , $T \geq 1$ , and $T^{- β} \geq 0$ , the bound also holds in this case.
It remains to record the variance assertion. For every $T \geq T_{⋆}$ , the definition of $T_{⋆}$ gives $1 \leq T$ , and the minimum in Definition 17 gives $k_{T} \leq ⌊ T /2 ⌋$ . Apply Lemma 14 with $k = k_{T}$ , $L = e^{ζ}$ , and $α = e^{- 1/ t_{0}}$ . Its conclusion is exactly $Var_{M} (\frac{1}{T - k _{T}} t = k_{T} \sum T - 1 Z_{t} (k_{T})) \leq \frac{2}{T} [e^{ζ (k_{T} + 1)} (1 + \frac{4}{e ^{ζ} - 1}) + \frac{4}{1 - e ^{- 1/ t_{0}}}],$ uniformly over the finite alphabet sizes and every $M \in M_{T} (t_{0}, ζ, C)$ , as required.

∎

The bound in Lemma 15 identifies the observable estimator used for the upper side of the fixed- $C$ comparison. The variance display isolates the cost of multiplying observable policy ratios over a window of length $k_{T}$ and the serial-dependence contribution controlled by contraction. Balancing those terms gives the exponent $β$ under fixed overlap and fixed latent-stationary domination.

For the lower side, the appendix uses the signed-depth alternatives introduced in Definition 18. The next two statements record the contraction and stationary-law calculations that make the construction a member of the same finite-POMDP class.

Lemma 16 [lem:signed-depth-constant-policy-contraction] (Contraction of the constant-policy signed-depth kernel).

Let $t_{0} > 0$ , $ζ > 0$ , $C \geq 1$ , $Q \geq 1$ , $v \in {- 1, + 1}$ and $r \in [0, 1]$ , and write $α = e^{- 1/ t_{0}}$ . Let $P^{(r)}$ be the transition kernel induced on the joint state space $S_{Q} = {0} \times ({0, \dots, Q} \times {- 1, + 1})$ of the signed-depth alternative of Definition 18 with terminal depth $Q$ and index $v$ by the constant policy $a_{r}$ that plays action one with probability $r$ . Then, for all probability vectors $μ, μ^{'}$ on $S_{Q}$ , $∥ μ P^{(r)} - μ^{'} P^{(r)} ∥_{TV} \leq α ∥ μ - μ^{'} ∥_{TV} .$

⊢ Lean

Proof of Lemma 16.

Let $ε_{C} = min {(C - 1) /2, 1/2}, α = exp (- 1/ t_{0}) .$ For $σ = (0, j, u)$ and $σ^{'} = (0, j^{'}, u^{'})$ in $S_{Q}$ , define the refresh law $η_{C, Q} (σ^{'}) = 1 {j^{'} = 0} \frac{1 + u ^{'} ε _{C}}{2},$ and define the residual kernel $Γ_{C, r, Q} (σ, σ^{'}) = ⎩ ⎨ ⎧ η_{C, Q} (σ^{'}), \frac{1 - r}{2} + r 1 {u^{'} = u}, 0, j = Q, j < Q and j^{'} = j + 1, otherwise.$ Expanding the transition rule in Definition 18 under the constant policy $a_{r} (1) = r$ , $a_{r} (0) = 1 - r$ , gives, for every $σ, σ^{'} \in S_{Q}$ , $P^{(r)} (σ, σ^{'}) = (1 - α) η_{C, Q} (σ^{'}) + α Γ_{C, r, Q} (σ, σ^{'}) .$ Indeed, at terminal depth $j = Q$ both sides equal $η_{C, Q} (σ^{'})$ ; at depths $j < Q$ , the reset part contributes $(1 - α) η_{C, Q} (σ^{'})$ , while the advance-to- $j + 1$ part contributes $α [(1 - r) /2 + r 1 {u^{'} = u}]$ .

The residual kernel is stochastic. Since $C \geq 1$ , one has $ε_{C} \in [0, 1/2]$ , so the two reset weights are nonnegative and $u^{'} \in {- 1, + 1} \sum \frac{1 + u ^{'} ε _{C}}{2} = 1.$ If $j = Q$ , summing $Γ_{C, r, Q} (σ, \cdot)$ over $S_{Q}$ therefore gives $1$ . If $j < Q$ , the only nonzero entries have $j^{'} = j + 1$ , and their two sign weights sum to $(\frac{1 - r}{2} + r) + \frac{1 - r}{2} = 1,$ with nonnegativity following from $r \in [0, 1]$ . Thus $Γ_{C, r, Q} (σ, σ^{'}) \geq 0 and σ^{'} \in S_{Q} \sum Γ_{C, r, Q} (σ, σ^{'}) = 1$ for every $σ \in S_{Q}$ .

For any signed vector $λ$ on $S_{Q}$ , stochasticity of $Γ_{C, r, Q}$ gives the total-variation nonexpansiveness bound $∥ λ Γ_{C, r, Q} ∥_{TV} = \frac{1}{2} σ^{'} \sum σ \sum λ (σ) Γ_{C, r, Q} (σ, σ^{'}) \leq \frac{1}{2} σ^{'} \sum σ \sum ∣ λ (σ) ∣ Γ_{C, r, Q} (σ, σ^{'}) = \frac{1}{2} σ \sum ∣ λ (σ) ∣ σ^{'} \sum Γ_{C, r, Q} (σ, σ^{'}) = ∥ λ ∥_{TV} .$

Now take probability vectors $μ, μ^{'}$ . Because $\sum_{σ} μ (σ) = \sum_{σ} μ^{'} (σ) = 1$ , the common refresh term cancels: $(μ P^{(r)} - μ^{'} P^{(r)}) (σ^{'}) = α σ \sum (μ (σ) - μ^{'} (σ)) Γ_{C, r, Q} (σ, σ^{'}) .$ Since $α = exp (- 1/ t_{0}) > 0$ , the preceding display and nonexpansiveness yield $∥ μ P^{(r)} - μ^{'} P^{(r)} ∥_{TV} = α ∥ (μ - μ^{'}) Γ_{C, r, Q} ∥_{TV} \leq α ∥ μ - μ^{'} ∥_{TV} .$ This is the claimed contraction.

∎

Lemma 17 [lem:signed-depth-stationary-law] (Closed form of the signed-depth stationary law).

Let $t_{0} > 0$ , $ζ > 0$ , $C \geq 1$ , $Q \geq 1$ , $v \in {- 1, + 1}$ and $r \in [0, 1]$ . Write $α = e^{- 1/ t_{0}}$ , $ε_{C} = min {(C - 1) /2, 1/2}$ , and $w_{j} = (1 - α) α^{j} / (1 - α^{Q + 1})$ . Let $P^{(r)}$ be the kernel induced on $S_{Q}$ by the constant policy $a_{r}$ for the signed-depth alternative of Definition 18 with terminal depth $Q$ and index $v$ , and let $d_{r}$ be the stationary law selected for $P^{(r)}$ . Then, for every hidden state $h = (j (h), u (h))$ , $d_{r} (0, h) = w_{j (h)} \frac{1 + u ( h ) ε _{C} r ^{j (h)}}{2} .$

⊢ Lean

Proof of Lemma 17.

Write $\overset{ˉ}{d}_{r} (0, (j, u)) := w_{j} \frac{1 + u ε _{C} r ^{j}}{2}, j \in {0, \dots, Q}, u \in {- 1, + 1} .$ We prove that this probability vector is stationary for $P^{(r)}$ , and then use the contraction statement to identify it with the selected stationary law.

Since $t_{0} > 0$ , the value $α = e^{- 1/ t_{0}}$ lies in $(0, 1)$ . Also $C \geq 1$ gives $0 \leq ε_{C} \leq 1/2$ , and $r \in [0, 1]$ gives $0 \leq r^{j} \leq 1$ . Hence every factor $(1 + u ε_{C} r^{j}) /2$ is nonnegative. For each fixed depth $j$ , $u \in {- 1, + 1} \sum w_{j} \frac{1 + u ε _{C} r ^{j}}{2} = w_{j},$ and the geometric normalization gives $j = 0 \sum Q w_{j} = \frac{1 - α}{1 - α ^{Q + 1}} j = 0 \sum Q α^{j} = 1.$ Thus $\overset{ˉ}{d}_{r}$ is a probability vector on $S_{Q}$ .
The state kernel induced by the constant policy $a_{r}$ has the following hidden-state transition probabilities, obtained from the signed-depth dynamics in Definition 18 by averaging the action-one and action-zero advance weights. If $j < Q$ , then $P^{(r)} ((0, (j, u)), (0, (0, u^{'}))) = (1 - α) \frac{1 + u ^{'} ε _{C}}{2},$ and $P^{(r)} ((0, (j, u)), (0, (j + 1, u^{'}))) = α \frac{1 + u u ^{'} r}{2} .$ If $j = Q$ , then $P^{(r)} ((0, (Q, u)), (0, (0, u^{'}))) = \frac{1 + u ^{'} ε _{C}}{2} .$ The remaining transitions have weight zero.

For a target state of depth $0$ , summing the displayed reset probabilities gives $(\overset{ˉ}{d}_{r} P^{(r)}) (0, (0, u^{'})) = \frac{1 + u ^{'} ε _{C}}{2} {w_{Q} + (1 - α) j = 0 \sum Q - 1 w_{j}} .$ Since $\sum_{j = 0}^{Q} w_{j} = 1$ , $w_{Q} + (1 - α) j = 0 \sum Q - 1 w_{j} = (1 - α) + α w_{Q} = (1 - α) + \frac{( 1 - α ) α ^{Q + 1}}{1 - α ^{Q + 1}} = \frac{1 - α}{1 - α ^{Q + 1}} = w_{0} .$ Therefore $(\overset{ˉ}{d}_{r} P^{(r)}) (0, (0, u^{'})) = w_{0} \frac{1 + u ^{'} ε _{C}}{2} = \overset{ˉ}{d}_{r} (0, (0, u^{'})) .$

For a target state of depth $m \in {1, \dots, Q}$ , only depth $m - 1$ can advance to it, so $(\overset{ˉ}{d}_{r} P^{(r)}) (0, (m, u^{'})) = α w_{m - 1} u \in {- 1, + 1} \sum \frac{1 + u ε _{C} r ^{m - 1}}{2} \frac{1 + u u ^{'} r}{2} .$ The sign sum is $u \in {- 1, + 1} \sum \frac{1 + u ε _{C} r ^{m - 1}}{2} \frac{1 + u u ^{'} r}{2} = \frac{1 + u ^{'} ε _{C} r ^{m}}{2} .$ Using $w_{m} = α w_{m - 1}$ , this yields $(\overset{ˉ}{d}_{r} P^{(r)}) (0, (m, u^{'})) = w_{m} \frac{1 + u ^{'} ε _{C} r ^{m}}{2} = \overset{ˉ}{d}_{r} (0, (m, u^{'})) .$ Thus $\overset{ˉ}{d}_{r} P^{(r)} = \overset{ˉ}{d}_{r}$ .
The contraction supplied by Lemma 16 implies uniqueness of stationary probability vectors. Indeed, if $λ$ and $λ^{'}$ are stationary for $P^{(r)}$ , then $∥ λ - λ^{'} ∥_{TV} = ∥ λ P^{(r)} - λ^{'} P^{(r)} ∥_{TV} \leq α ∥ λ - λ^{'} ∥_{TV} .$ Because $α < 1$ and total variation is nonnegative, the norm is zero, hence $λ = λ^{'}$ on the finite state space. The selected law $d_{r}$ is stationary because the stationary vector $\overset{ˉ}{d}_{r}$ exists, so uniqueness gives $d_{r} = \overset{ˉ}{d}_{r}$ .
Evaluating the identity $d_{r} = \overset{ˉ}{d}_{r}$ at the hidden state $h = (j (h), u (h))$ gives $d_{r} (0, h) = w_{j (h)} \frac{1 + u ( h ) ε _{C} r ^{j (h)}}{2},$ as claimed.

∎

Lemma 16 gives the uniform total-variation contraction of the signed-depth transition under any constant action-one probability. Lemma 17 then gives the stationary law explicitly: the depth mass is a truncated geometric sequence, and the sign imbalance is scaled by the action-one probability to the current depth.

Lemma 18 [lem:signed-depth-membership] (Signed-depth membership).

For every $T \geq 1$ , $Q \geq 1$ , $t_{0} > 0$ , $ζ > 0$ , $C > 1$ , and $v \in {- 1, + 1}$ , let $M_{Q, v}^{T}$ be the signed-depth experiment of Definition 18. Put $ε_{C} = min {(C - 1) /2, 1/2}, L = exp (ζ), w_{j} = \frac{( 1 - α ( t _{0} )) α ( t _{0} ) ^{j}}{1 - α ( t _{0} ) ^{Q + 1}},$ where $α (t_{0})$ is the mixing factor and $j = 0, \dots, Q$ . Then:

(Membership.) $M_{Q, v}^{T} \in M_{T} (t_{0}, ζ, C)$ in the sense of Definition 10.

(Behavior stationary law.) For every hidden state $h$ , with decoded depth $j (h) \in {0, \dots, Q}$ and sign $u (h) \in {- 1, + 1}$ , $d_{b} (0, h) = w_{j (h)} \frac{1 + u ( h ) ε _{C} L ^{- j (h)}}{2} .$

(Constant action-one stationary laws.) For every $r \in [0, 1]$ , if $d_{r}$ denotes the stationary law induced by the constant policy that chooses action one with probability $r$ , then for every hidden state $h$ , $d_{r} (0, h) = w_{j (h)} \frac{1 + u ( h ) ε _{C} r ^{j (h)}}{2} .$

(Target-over-behavior domination.) For every joint state $s$ , $d_{e} (s) \leq (1 + ε_{C}) d_{b} (s), 1 + ε_{C} \leq C .$

The stationary target value of Definition 12 is $θ (K, e) = v c_{0} (t_{0}) ε_{C} w_{Q} .$

⊢ Lean

Proof of Lemma 18.

Write $α = α (t_{0}) = exp (- 1/ t_{0})$ , $L = exp (ζ)$ , and $ε = ε_{C}$ . Since $t_{0} > 0$ and $ζ > 0$ , we have $0 < α < 1$ and $1 < L$ . Also $C > 1$ gives $C \geq 1$ , so $0 \leq ε = min {(C - 1) /2, 1/2} \leq \frac{1}{2} .$ For a hidden state $h$ , abbreviate $j = j (h)$ and $u = u (h) \in {- 1, + 1}$ .

For $r \in [0, 1]$ , let $a_{r}$ be the constant policy with $Pr (A_{t} = 1) = r$ , and let $P^{(r)}$ be the induced transition kernel on the joint state space $S_{Q} = {(0, h) : h \in {0, \dots, Q} \times {- 1, + 1}} .$ By Lemma 16, for all probability laws $μ$ and $μ^{'}$ on $S_{Q}$ , $∥ μ P^{(r)} - μ^{'} P^{(r)} ∥_{TV} \leq α ∥ μ - μ^{'} ∥_{TV} .$ The behavior policy in Definition 18 is the case $r = L^{- 1}$ , while the target policy is the case $r = 1$ . The displayed inequality therefore gives the contraction required in Assumption 6 for both policies.
For $r \in [0, 1]$ , Lemma 17 identifies the selected stationary law of the constant-policy kernel $P^{(r)}$ : for every hidden state $h$ , $d_{r} (0, h) = w_{j (h)} \frac{1 + u ( h ) ε r ^{j (h)}}{2} .$ Taking $r = L^{- 1}$ gives the behavior stationary law because the behavior policy in Definition 18 chooses action one with probability $1/ L$ , and $(L^{- 1})^{j (h)} = L^{- j (h)}$ . Thus $d_{b} (0, h) = w_{j (h)} \frac{1 + u ( h ) ε L ^{- j (h)}}{2} .$ Taking $r = 1$ gives the target stationary law $d_{e} (0, h) = w_{j (h)} \frac{1 + u ( h ) ε}{2} .$
We now verify the model-class predicates in Definition 10. The embedded experiment is generated by the initial law, the behavior policy, and the time-homogeneous reward-transition kernel in Definition 18. Therefore, for each epoch $t = 0, \dots, T - 1$ , $L (Y_{t}, X_{t + 1}, H_{t + 1} ∣ X_{0 : t}, H_{0 : t}, A_{0 : t}, Y_{0 : t - 1}) = K (\cdot ∣ X_{t}, H_{t}, A_{t}),$ which is Assumption 1. The behavior action law is $b (1 ∣ 0) = L^{- 1}, b (0 ∣ 0) = 1 - L^{- 1}, b (0 ∣ 0) + b (1 ∣ 0) = 1,$ and these two masses are nonnegative because $L > 1$ . Since the observed state is the singleton ${0}$ , the chronological law factors at every epoch $t = 0, \dots, T - 1$ as $P {((X_{s}, H_{s})_{s < t}, (A_{s}, Y_{s})_{s < t}, (X_{t}, H_{t}), A_{t}) \in d h d a} = P {((X_{s}, H_{s})_{s < t}, (A_{s}, Y_{s})_{s < t}, (X_{t}, H_{t})) \in d h} b (d a ∣ X_{t} (h)),$ which is Assumption 2. The finite reward symbols are decoded as $- 1$ and $+ 1$ , so $Pr (∣ Y_{t} ∣ \leq 1) = 1,$ giving Assumption 4. The policy-overlap inequality is $e (1 ∣ 0) = 1 = L b (1 ∣ 0), e (0 ∣ 0) = 0 \leq L b (0 ∣ 0),$ so Assumption 5 holds with factor $L$ . The initial law is $Pr {S_{0} = (0, h)} = w_{j (h)} \frac{1 + u ( h ) ε L ^{- j (h)}}{2} = d_{b} (0, h),$ where the last equality is the stationary-law formula from the case $r = L^{- 1}$ . Thus Assumption 3 holds.

For the latent stationary overlap, first identify the two selected stationary laws and the initial law on the same cells. The target and behavior formulas from Step 2 give $d_{e} (0, h) = w_{j} \frac{1 + u ( h ) ε}{2}, d_{b} (0, h) = w_{j} \frac{1 + u ( h ) ε L ^{- j}}{2} .$ The displayed initialization in Definition 18 gives $Pr {S_{0} = (0, h)} = w_{j} \frac{1 + u ( h ) ε L ^{- j}}{2} = d_{b} (0, h),$ where the last equality is the behavior stationary-law formula from Step 2. The depth mass is nonnegative: since $0 < α < 1, 1 - α^{Q + 1} > 0,$ we have $w_{j} = \frac{( 1 - α ) α ^{j}}{1 - α ^{Q + 1}} \geq 0.$ Here $L^{- j} = (L^{j})^{- 1}$ . Since $L > 1$ , we have $0 < L^{- j} \leq 1$ . For $u (h) = - 1$ , $d_{e} (0, h) = w_{j} \frac{1 - ε}{2} \leq w_{j} \frac{1 - ε L ^{- j}}{2} = d_{b} (0, h) \leq (1 + ε) d_{b} (0, h),$ where the first inequality uses $ε L^{- j} \leq ε$ , and the last uses $ε \geq 0$ and nonnegativity of the behavior cell. For $u (h) = + 1$ , $d_{e} (0, h) = w_{j} \frac{1 + ε}{2} \leq w_{j} \frac{1 + ε}{2} (1 + ε L^{- j}) = (1 + ε) d_{b} (0, h),$ because $w_{j} \geq 0$ and $1 + ε \geq 0$ , so the multiplier $w_{j} (1 + ε) /2$ is nonnegative, while $1 + ε L^{- j} \geq 1$ . Multiplying this last inequality by that nonnegative multiplier gives the displayed inequality, and its right-hand side is $(1 + ε) d_{b} (0, h)$ by the formula for $d_{b}$ . Every joint state has the unique observed coordinate $0$ , so these two sign cases give $d_{e} (s) \leq (1 + ε) d_{b} (s) for every s \in S_{Q} .$ Moreover $1 + ε = 1 + min {(C - 1) /2, 1/2} \leq C,$ because either $ε = (C - 1) /2$ , giving $1 + ε = (C + 1) /2 \leq C$ , or $ε = 1/2$ , which is possible only when $C \geq 2$ . Since $d_{b} (s) \geq 0$ , $d_{e} (s) \leq (1 + ε) d_{b} (s) \leq C d_{b} (s),$ so Assumption 7 holds.

It remains to collect the remaining requirements of Definition 10. The horizon requirement $1 \leq T$ is the hypothesis $T \geq 1$ of the present lemma. The observed alphabet of $M_{Q, v}^{T}$ is the singleton ${0}$ , the latent alphabet is the $2 (Q + 1)$ -element set ${0, \dots, Q} \times {- 1, + 1}$ , and the action alphabet is ${0, 1}$ ; all three are finite, and the action alphabet has exactly two elements, as the class requires. The parameter requirements $0 < t_{0}$ and $0 < ζ$ are hypotheses of the lemma, and they fix $L = exp (ζ)$ and $α = exp (- 1/ t_{0})$ as the overlap and contraction constants used throughout. Together with the seven assumptions verified above – the Markov kernel, sequential ignorability, stationary start, bounded rewards, policy overlap, uniform contraction, and latent stationary overlap – this is every defining property of Definition 10, and therefore $M_{Q, v}^{T} \in M_{T} (t_{0}, ζ, C)$ .
It remains to compute the target value. The two-point alternative label enters the reward law only through its numeric sign: the positive label contributes $+ 1$ , the negative label contributes $- 1$ , and in the statement this multiplier is denoted by $v \in {- 1, + 1}$ . Under the target policy the action is always one. From the reward weights in Definition 18, $g_{e} (0, h) = a \sum e (a ∣ 0) E_{K} [Y_{t} ∣ S_{t} = (0, h), A_{t} = a] = v c_{0} (t_{0}) u (h) 1 {j (h) = Q} .$ Using Definition 12 and the already computed target stationary law, $θ (K, e) = h \sum w_{j (h)} \frac{1 + u ( h ) ε}{2} v c_{0} (t_{0}) u (h) 1 {j (h) = Q} .$ Only depth $Q$ contributes, and summing over the two signs gives $θ (K, e) = v c_{0} (t_{0}) w_{Q} [\frac{1 + ε}{2} - \frac{1 - ε}{2}] = v c_{0} (t_{0}) ε w_{Q} .$ This is the claimed value formula.

∎

The stationary laws in Lemma 18 show how the lower construction keeps the target-over-behavior ratio bounded while concentrating the signal at terminal depth. The terminal value is proportional to the stationary depth mass $w_{Q}$ , so increasing $Q$ makes the two alternatives harder to separate from observed data while preserving the fixed latent-overlap radius.

The likelihood calculation rests on finite-prefix posterior bounds, a rare-action second moment, a bootstrap factor that makes the posterior control uniform in the terminal depth, signed-mass identities at terminal depth, exact conditional reward means, and finite-word KL inequalities. The next group develops these ingredients in the order in which the observed-path comparison uses them.

Lemma 19 [lem:posterior-lower-window-power] (Posterior window power bound).

Let $N, Q, k, d \in N$ , let $t_{0}, ζ, C, b \in R$ , let $v \in {- 1, + 1}$ , and let $o = ((a_{0}, y_{0}), \dots, (a_{k - 1}, y_{k - 1}))$ be an observed prefix for the signed-depth family of Definition 18. Write $c_{0} (t_{0}) = \frac{1 - α ( t _{0} )}{4},$ and, for $0 \leq m \leq k$ , write $o_{\leq m}$ for the first $m$ observed pairs. For any observed prefix $o^{'}$ , define the raw terminal-depth posterior by $π_{v} (o^{'}) = \frac{\sum _{h} 1 { J ( h ) = Q } λ _{v} ( h ; o ^{'} )}{\sum _{h} λ _{v} ( h ; o ^{'} )} .$ Assume:

(Nonnegative amplitude.) $0 \leq c_{0} (t_{0})$ .

(Nonnegative base.) $0 \leq 1 - c_{0} (t_{0}) b$ .

(Window length.) $d \leq k$ .

(Strict-prefix posterior bound.) For every $i \in {0, \dots, k - 1}$ , $π_{v} (o_{\leq i}) \leq b .$

Define the final $d$ -prefix lower window by $Λ_{d} (o) = 0 \leq i < k k - d \leq i \prod (1 - c_{0} (t_{0}) π_{v} (o_{\leq i})) .$ Then $(1 - c_{0} (t_{0}) b)^{d} \leq Λ_{d} (o) .$

⊢ Lean

Proof of Lemma 19.

We prove the slightly stronger assertion with the prefix length left variable. For a prefix $o$ of length $n$ , write $Λ_{d}^{(n)} (o) = 0 \leq i < n n - d \leq i \prod (1 - c_{0} (t_{0}) π_{v} (o_{\leq i})),$ with the convention that an empty product is $1$ .

1. If $d = 0$ , then the condition $n \leq i < n$ selects no indices, so $Λ_{0}^{(n)} (o) = 1 = (1 - c_{0} (t_{0}) b)^{0} .$ This proves the base case.

2. Assume the claim has been proved for $d$ . Consider a prefix $o$ of length $n$ with $d + 1 \leq n$ . If $n = 0$ , this inequality is impossible, so there is no case to prove. Thus write $n = k + 1$ . Let $o^{-}$ be the length- $k$ prefix obtained by dropping the last observed pair: $o^{-} (i) = o (i), 0 \leq i < k .$ For every $0 \leq i < k$ , the first $i$ observations of $o^{-}$ are exactly the first $i$ observations of $o$ : $(o^{-})_{\leq i} = o_{\leq i} .$ Hence the strict-prefix posterior hypothesis for $o$ gives $π_{v} ((o^{-})_{\leq i}) = π_{v} (o_{\leq i}) \leq b, 0 \leq i < k .$ Since $d + 1 \leq k + 1$ , we have $d \leq k$ , so the induction hypothesis applied to $o^{-}$ yields $(1 - c_{0} (t_{0}) b)^{d} \leq Λ_{d}^{(k)} (o^{-}) .$

3. The final-window product for length $k + 1$ splits into the length- $k$ final-window product for $o^{-}$ and the last factor: $Λ_{d + 1}^{(k + 1)} (o) = Λ_{d}^{(k)} (o^{-}) (1 - c_{0} (t_{0}) π_{v} (o_{\leq k})) .$ Indeed, the threshold satisfies $(k + 1) - (d + 1) = k - d,$ so the eligible indices before the last one are precisely the eligible indices in the length- $k$ window, and the remaining eligible index is $i = k$ .

4. The posterior hypothesis at the last strict prefix gives $π_{v} (o_{\leq k}) \leq b$ . Because $0 \leq c_{0} (t_{0})$ , multiplying preserves the inequality, $c_{0} (t_{0}) π_{v} (o_{\leq k}) \leq c_{0} (t_{0}) b,$ and subtracting from $1$ reverses the side on which the smaller product appears: $1 - c_{0} (t_{0}) b \leq 1 - c_{0} (t_{0}) π_{v} (o_{\leq k}) .$ Together with the assumed nonnegativity $0 \leq 1 - c_{0} (t_{0}) b$ , we have $0 \leq (1 - c_{0} (t_{0}) b)^{d} .$ Combining this nonnegativity, the induction bound, and the last-factor bound gives $(1 - c_{0} (t_{0}) b)^{d + 1} = (1 - c_{0} (t_{0}) b)^{d} (1 - c_{0} (t_{0}) b) \leq Λ_{d}^{(k)} (o^{-}) (1 - c_{0} (t_{0}) π_{v} (o_{\leq k})) = Λ_{d + 1}^{(k + 1)} (o) .$ This completes the induction, and hence the desired bound for the original prefix of length $k$ .

∎

Lemma 20 [lem:terminal-posterior-mature-window] (Terminal posterior window bound).

Let $N, Q \in N$ , let $t_{0}, ζ, C \in R$ , and let $v \in {- 1, + 1}$ . Put $α = exp {- (1/ t_{0})}$ . In the finite signed-depth model of Definition 18, for an observed prefix $o = ((a_{1}, y_{1}), \dots, (a_{k}, y_{k}))$ , write $π_{v} (o) = \frac{\sum _{h \in {0, \dots, Q} \times {- 1, + 1}} 1 { J ( h ) = Q } λ _{v} ( h ; o )}{M _{v} ( o )} .$ For $d \in N$ , define the lower posterior window $Λ_{d} (o) = 0 \leq i < k k - d \leq i \prod (1 - c_{0} (t_{0}) π_{v} (o_{\leq i})),$ where $o_{\leq i}$ is the prefix of $o$ containing its first $i$ action–reward pairs.

Assume:

(Positive scale.) $t_{0} > 0$ .

(Positive tilt.) $ζ > 0$ .

(Overlap level.) $C > 1$ .

(Mature window.) $Q \leq k$ .

Then the terminal-depth posterior satisfies $π_{v} (o) \leq \frac{α ^{Q}}{Λ _{Q} ( o )} .$

⊢ Lean

Proof of Lemma 20.

Set $o^{-} := o_{\leq k - Q}$ , and introduce $W_{fair} := k - Q \leq i < k \prod b (a_{i + 1}) \frac{1}{2}, M_{pre} := M_{v} (o^{-}),$ $U_{pre} := u \in {- 1, + 1} \sum λ_{v} ((0, u); o^{-}), R_{corr} := k - Q \leq i < k \prod \frac{M _{v} ( o _{\leq i + 1} )}{b ( a _{i + 1} ) M _{v} ( o _{\leq i} )},$ and $P_{low} := Λ_{Q} (o) .$ Here the products are over the final $Q$ action–reward pairs; if $Q = 0$ , they are empty products and equal one.

1. The terminal-depth numerator after the full prefix factors as $h \in {0, \dots, Q} \times {- 1, + 1} \sum 1 {J (h) = Q} λ_{v} (h; o) = α^{Q} W_{fair} U_{pre} .$ Indeed, summing over the two terminal signs gives the unsigned mass at depth $Q$ . Over a mature window of length $Q$ , the only way to arrive at terminal depth $Q$ is to start the window at depth $0$ and advance once at each of the $Q$ updates; this contributes $α^{Q}$ , while the action and sign-symmetric factors over the same window contribute exactly $W_{fair}$ . The remaining mass is the unsigned depth-zero mass $U_{pre}$ at the beginning of the window.

2. The total mass over the observed prefix factors through the same window as $M_{v} (o) = W_{fair} M_{pre} R_{corr} .$ This follows by iterating the one-step mass recursion across the last $Q$ observations: each step extracts the behavior/fair-sign factor and leaves the corresponding reward-correction factor. For $Q = 0$ the window is empty and the identity reduces to $M_{v} (o) = M_{v} (o)$ .

3. The unsigned depth-zero mass at the window start is bounded by the total mass there: $U_{pre} = u \in {- 1, + 1} \sum λ_{v} ((0, u); o^{-}) \leq j = 0 \sum Q u \in {- 1, + 1} \sum λ_{v} ((j, u); o^{-}) = M_{pre} .$ All summands are nonnegative, since each prefix weight is a sum of nonnegative path weights.

4. The lower posterior window is positive and is dominated by the correction window: $0 < P_{low} \leq R_{corr} .$ For each factor in $P_{low}$ , $0 \leq π_{v} (o_{\leq i}) \leq 1, 0 \leq c_{0} (t_{0}) < 1,$ so $1 - c_{0} (t_{0}) π_{v} (o_{\leq i}) > 0$ . The one-step reward-correction factor at the same prefix is at least $1 - c_{0} (t_{0}) π_{v} (o_{\leq i}),$ and taking products over the final window gives the displayed inequality.

5. The remaining factors have the needed signs: $W_{fair} > 0, M_{pre} > 0, R_{corr} > 0, α^{Q} \geq 0.$ The first inequality is the product of positive behavior weights and $1/2$ factors, while the second is positivity of the finite total mass at the window start. The third follows from Step 4 by $0 < P_{low} \leq R_{corr}$ . Finally, $α = exp {- (1/ t_{0})} > 0$ , so its $Q$ -th power is nonnegative.

6. Combining the two window factorizations with the monotonicity bounds gives $U_{pre} P_{low} \leq M_{pre} R_{corr},$ because $0 \leq U_{pre} \leq M_{pre}$ , $0 < P_{low} \leq R_{corr}$ , and $M_{pre} \geq 0$ . Multiplying by the nonnegative factor $α^{Q} W_{fair}$ yields $α^{Q} W_{fair} U_{pre} P_{low} \leq α^{Q} W_{fair} M_{pre} R_{corr} .$ Using the identities from Steps 1 and 2, this is $(h \sum 1 {J (h) = Q} λ_{v} (h; o)) P_{low} \leq α^{Q} M_{v} (o) .$ Since $M_{v} (o) > 0$ and $P_{low} > 0$ , division gives $π_{v} (o) = \frac{\sum _{h} 1 { J ( h ) = Q } λ _{v} ( h ; o )}{M _{v} ( o )} \leq \frac{α ^{Q}}{P _{low}} = \frac{α ^{Q}}{Λ _{Q} ( o )},$ as claimed.

∎

Lemma 21 [lem:terminal-posterior-initial-window] (Initial-window posterior bound).

Let $N, Q \in N$ , let $t_{0}, ζ, C \in R$ , let $v \in {- 1, + 1}$ , let $k \in N$ , and let $o : {0, \dots, k - 1} \to {0, 1} \times {0, 1}$ be an observed action/reward prefix for the signed-depth construction of Definition 18. Write $α = e^{- 1/ t_{0}}$ . For any prefix $o^{'}$ , define the raw terminal-depth posterior by $π_{v} (o^{'}) = \frac{\sum _{h : J (h) = Q} λ _{v} ( h ; o ^{'} )}{\sum _{h} λ _{v} ( h ; o ^{'} )} .$ Define the initial lower posterior window over the available prefixes by $Λ_{k} (o) = i = 0 \prod k - 1 (1 - c_{0} π_{v} (o_{\leq i})),$ where $o_{\leq i}$ is the prefix of $o$ of length $i$ .

Assume:

(Positive scale.) $t_{0} > 0$ .

(Positive overlap exponent.) $ζ > 0$ .

(Nontrivial overlap radius.) $C > 1$ .

(Initial window.) $k < Q$ .

Then the terminal-depth posterior after observing $o$ satisfies $π_{v} (o) \leq \frac{α ^{Q}}{Λ _{k} ( o )} .$

⊢ Lean

Proof of Lemma 21.

Write $o_{i} = (a_{i}, y_{i})$ for $0 \leq i < k$ , and let $o_{\leq 0}$ denote the empty prefix. For any prefix $ω$ , set $Mass_{v} (ω) := h \sum λ_{v} (h; ω), Term_{v} (ω) := h : J (h) = Q \sum λ_{v} (h; ω) .$ Let $b_{ζ} (a)$ be the behavior-policy probability of action $a$ from Definition 18, and define $Fair_{k} (o) := i = 0 \prod k - 1 b_{ζ} (a_{i}) \frac{1}{2} .$ For a prefix $ω$ and a next observation $(a, y)$ , define the reward-correction factor $Γ_{v} (ω, (a, y)) := \frac{2 \sum _{h} λ _{v} ( h ; ω ) \frac{1 + ( 2 y - 1 ) v c _{0} U ( h ) 1 { J ( h ) = Q }}{2}}{Mass _{v} ( ω )},$ whenever $Mass_{v} (ω) > 0$ , with any fixed value when the denominator is zero. Along the observed prefixes below the denominator is positive. Finally set $Corr_{k} (o) := i = 0 \prod k - 1 Γ_{v} (o_{\leq i}, o_{i}), Low_{k} (o) := Λ_{k} (o) .$

Since $k < Q$ , a hidden path contributing to $Term_{v} (o)$ starts, after summing over the sign coordinate, from depth $Q - k$ and then makes $k$ successive depth-increment moves. Thus the terminal numerator factors as $Term_{v} (o) = α^{k} w_{Q - k} Fair_{k} (o),$ where $w_{j} = \frac{( 1 - α ) α ^{j}}{1 - α ^{Q + 1}}, 0 \leq j \leq Q,$ is the signed-depth initial depth mass from Definition 18. Since $k < Q$ , $α^{k} w_{Q - k} = α^{k} \frac{( 1 - α ) α ^{Q - k}}{1 - α ^{Q + 1}} = \frac{( 1 - α ) α ^{Q}}{1 - α ^{Q + 1}} = w_{Q},$ and therefore $Term_{v} (o) = w_{Q} Fair_{k} (o) .$
Summing the one-step kernel over the next hidden state leaves total hidden-transition mass one. Hence each extension satisfies $Mass_{v} (o_{\leq i + 1}) = b_{ζ} (a_{i}) \frac{1}{2} Γ_{v} (o_{\leq i}, o_{i}) Mass_{v} (o_{\leq i}) .$ Multiplying this identity for $i = 0, \dots, k - 1$ , with the empty-product convention when $k = 0$ , gives $Mass_{v} (o) = Fair_{k} (o) Mass_{v} (o_{\leq 0}) Corr_{k} (o) .$
The empty-prefix mass is one. Indeed, the initial weights sum to $j = 0 \sum Q w_{j} u \in {- 1, + 1} \sum \frac{1 + u ε _{C} L ^{- j}}{2} = j = 0 \sum Q w_{j} = 1.$ Thus $Mass_{v} (o_{\leq 0}) = 1.$
The lower window is positive and is bounded by the correction window: $0 < Low_{k} (o), Low_{k} (o) \leq Corr_{k} (o) .$ To see this, $t_{0} > 0$ gives $0 < α < 1$ , hence $0 \leq c_{0} = \frac{1 - α}{4} < 1.$ For each observed prefix $o_{\leq i}$ , the filter weights are nonnegative and their total mass is positive, so $0 \leq π_{v} (o_{\leq i}) \leq 1.$ Consequently every factor $1 - c_{0} π_{v} (o_{\leq i})$ is positive. Moreover, for each $i$ , $Γ_{v} (o_{\leq i}, o_{i}) \geq (1 - π_{v} (o_{\leq i})) \cdot 1 + π_{v} (o_{\leq i}) (1 - c_{0}) = 1 - c_{0} π_{v} (o_{\leq i}),$ because the doubled reward factor equals $1$ off terminal depth and is at least $1 - c_{0}$ on terminal depth. Multiplying these pointwise inequalities gives the displayed product bound.
The fair-window factor is strictly positive: $0 < Fair_{k} (o) .$ Each behavior probability $b_{ζ} (a_{i})$ is positive under $ζ > 0$ , and each reward-symbol base factor $1/2$ is positive; the empty product is $1$ .
The terminal depth mass is bounded by $α^{Q}$ : $w_{Q} \leq α^{Q} .$ Indeed, $w_{Q} = \frac{( 1 - α ) α ^{Q}}{1 - α ^{Q + 1}},$ and $0 < α < 1$ implies $1 - α^{Q + 1} > 0$ and $α^{Q + 1} \leq α$ . After multiplying by the positive denominator, the desired inequality is $(1 - α) α^{Q} \leq (1 - α^{Q + 1}) α^{Q},$ which is exactly $α^{Q + 1} \leq α$ times the nonnegative factor $α^{Q}$ .
Combining the previous bounds, $w_{Q} Low_{k} (o) \leq α^{Q} Corr_{k} (o) .$ Multiplying by the nonnegative factor $Fair_{k} (o)$ , and using the identities above, yields $Term_{v} (o) Low_{k} (o) \leq α^{Q} Mass_{v} (o) .$ Since $Mass_{v} (o) > 0$ and $Low_{k} (o) > 0$ , division gives $π_{v} (o) = \frac{Term _{v} ( o )}{Mass _{v} ( o )} \leq \frac{α ^{Q}}{Low _{k} ( o )} = \frac{α ^{Q}}{Λ _{k} ( o )} .$

∎

Lemma 22 [lem:terminal-posterior-crude-bound] (Crude terminal posterior bound).

Let $N, Q, k \in N$ , let $t_{0}, ζ, C \in R$ , and let $v \in {- 1, + 1}$ . Suppose that

(Positive temperature.) $t_{0} > 0$ .

(Positive log-overlap.) $ζ > 0$ .

(Overlap level.) $C > 1$ .

For the finite signed-depth model of Definition 18, write $α = exp (- 1/ t_{0}), c_{0} = \frac{1 - α}{4} .$ For every observed prefix $o \in ({0, 1} \times {0, 1})^{k}$ , define the raw terminal-depth posterior by $π_{v} (o) = \frac{\sum _{h} 1 { J ( h ) = Q } λ _{v} ( h ; o )}{M _{v} ( o )}, M_{v} (o) = h \sum λ_{v} (h; o),$ where the sums range over the signed-depth hidden states at depth horizon $Q$ . Then $π_{v} (o) \leq (\frac{α}{1 - c _{0}})^{Q} .$

⊢ Lean

Proof of Lemma 22.

Write $α = exp (- 1/ t_{0})$ and $c_{0} = (1 - α) /4$ .

1. Since $t_{0} > 0$ , one has $0 < α < 1$ . Hence $0 \leq c_{0} = \frac{1 - α}{4} < 1, 0 < 1 - c_{0}, 0 \leq 1 - c_{0} \cdot 1.$ For every observed prefix $o^{'}$ , the raw terminal-depth posterior satisfies $0 \leq π_{v} (o^{'}) \leq 1.$ Indeed, the numerator is a sum of nonnegative filter masses over terminal-depth states, the denominator $M_{v} (o^{'})$ is positive, and the terminal-depth mass is bounded by the total mass.

2. First suppose $Q \leq k$ . By Lemma 20, $π_{v} (o) \leq \frac{α ^{Q}}{Λ _{Q} ( o )} .$ Apply Lemma 19 with $b = 1$ and $d = Q$ . The hypotheses are exactly those checked above: $0 \leq c_{0}$ , $0 \leq 1 - c_{0} \cdot 1$ , $Q \leq k$ , and $π_{v} (o_{\leq i}) \leq 1$ for every strict prefix. Thus $(1 - c_{0})^{Q} \leq Λ_{Q} (o) .$ Since $α^{Q} \geq 0$ and $1 - c_{0} > 0$ , division by the larger positive denominator gives $π_{v} (o) \leq \frac{α ^{Q}}{( 1 - c _{0} ) ^{Q}} = (\frac{α}{1 - c _{0}})^{Q} .$

3. It remains to consider $k < Q$ . By Lemma 21, $π_{v} (o) \leq \frac{α ^{Q}}{Λ _{k} ( o )} .$ Apply Lemma 19 with $b = 1$ and $d = k$ . Since $k \leq k$ , and the prefix posterior bound $π_{v} (o_{\leq i}) \leq 1$ holds for every strict prefix, this gives $(1 - c_{0})^{k} \leq Λ_{k} (o) .$ Therefore $π_{v} (o) \leq \frac{α ^{Q}}{( 1 - c _{0} ) ^{k}} .$ Because $0 < 1 - c_{0} \leq 1$ and $k < Q$ , monotonicity of powers on $[0, 1]$ yields $(1 - c_{0})^{Q} \leq (1 - c_{0})^{k} .$ Dividing the nonnegative numerator $α^{Q}$ by these positive denominators gives $\frac{α ^{Q}}{( 1 - c _{0} ) ^{k}} \leq \frac{α ^{Q}}{( 1 - c _{0} ) ^{Q}} = (\frac{α}{1 - c _{0}})^{Q} .$ Combining the displayed inequalities proves the claimed bound in the initial-window case as well.

∎

Lemma 23 [lem:refined-terminal-posterior] (Refined terminal posterior bound).

Let $N, Q, k$ be nonnegative integers. Suppose that

(Positive scale.) $t_{0} > 0$ .

(Positive perturbation.) $ζ > 0$ .

(Overlap level.) $C > 1$ .

Write $α = exp (- 1/ t_{0})$ for the contraction parameter and $c_{0} = (1 - α) /4$ for the signed-depth reward-amplitude constant. For an alternative $v \in {- 1, + 1}$ in the signed-depth family of Definition 18 and any observed prefix $o = (a_{1}, y_{1}, \dots, a_{k}, y_{k})$ , let $π_{N, Q, v} (o)$ be the raw terminal-depth posterior: the run-local unnormalized filter mass at latent depth $Q$ , divided by the total run-local unnormalized filter mass of the prefix. Then $π_{N, Q, v} (o) \leq \frac{α ^{Q}}{{ 1 - c _{0} ( \frac{α}{1 - c _{0}} ) ^{Q} } ^{Q}} .$

⊢ Lean

Proof of Lemma 23.

Let $Γ := \frac{α}{1 - c _{0}} .$ Since $t_{0} > 0$ , the contraction parameter satisfies $0 < α < 1$ . Hence $0 \leq c_{0} = \frac{1 - α}{4} \leq \frac{1}{4}, 1 - c_{0} > 0,$ and therefore $Γ \geq 0$ . Moreover, $Γ < 1 ⟺ α < 1 - \frac{1 - α}{4} ⟺ α < 1.$ Thus $Γ^{Q} \leq 1$ . Consequently $B_{Q} := 1 - c_{0} Γ^{Q}$ is strictly positive; indeed $c_{0} Γ^{Q} \leq c_{0} \leq 1/4$ .
For $0 \leq d \leq k$ , define the lower posterior window $Λ_{d} (o) := s = k - d + 1 \prod k (1 - c_{0} π_{N, Q, v} (o_{\leq s - 1})),$ where $o_{\leq m}$ denotes the first $m$ observed action–reward pairs, with $o_{\leq 0}$ the empty prefix; for $d = 0$ this is the empty product and equals $1$ . If $Q \leq k$ , Lemma 20 applies under the present hypotheses and gives $π_{N, Q, v} (o) \leq \frac{α ^{Q}}{Λ _{Q} ( o )} .$ If $k < Q$ , Lemma 21 applies under the present hypotheses and gives $π_{N, Q, v} (o) \leq \frac{α ^{Q}}{Λ _{k} ( o )} .$ These two estimates are exactly the two window cases needed below.
The crude terminal-posterior estimate in Lemma 22 applies to every observed prefix $o^{'}$ , of any length, under the same assumptions $t_{0} > 0$ , $ζ > 0$ , and $C > 1$ . With the present notation it states $π_{N, Q, v} (o^{'}) \leq (\frac{α}{1 - c _{0}})^{Q} = Γ^{Q} .$ This supplies the prefix-uniform bootstrap bound used in the lower-window estimate.
Apply Lemma 19 with $b := Γ^{Q} .$ Its nonnegative-amplitude hypothesis is $0 \leq c_{0}$ , established in the first step, and its nonnegative-base hypothesis is $0 \leq 1 - c_{0} Γ^{Q} = B_{Q},$ also established there. Its strict-prefix posterior hypothesis holds because the preceding step gives, for every prefix appearing in the window, $π_{N, Q, v} (o_{\leq s - 1}) \leq Γ^{Q} .$ The indexing in Lemma 19 ranges over the final $d$ strict prefixes of $o$ , precisely the factors in $Λ_{d} (o)$ . Hence, for every $d \leq k$ , $B_{Q}^{d} = (1 - c_{0} Γ^{Q})^{d} \leq Λ_{d} (o) .$
If $Q \leq k$ , the terminal-window bound with $d = Q$ and the preceding display give $π_{N, Q, v} (o) \leq \frac{α ^{Q}}{Λ _{Q} ( o )} \leq \frac{α ^{Q}}{B _{Q}^{Q}} .$ If $k < Q$ , the terminal-window bound with $d = k$ gives $π_{N, Q, v} (o) \leq \frac{α ^{Q}}{Λ _{k} ( o )} \leq \frac{α ^{Q}}{B _{Q}^{k}} .$ Since $0 < B_{Q} \leq 1$ and $k < Q$ , we have $B_{Q}^{Q} \leq B_{Q}^{k}$ , hence $\frac{α ^{Q}}{B _{Q}^{k}} \leq \frac{α ^{Q}}{B _{Q}^{Q}} .$ In both cases, $π_{N, Q, v} (o) \leq \frac{α ^{Q}}{( 1 - c _{0} ( \frac{α}{1 - c _{0}} ) ^{Q} ) ^{Q}},$ which is the claimed bound.

∎

The posterior bounds in Lemmas 19, 20, 21, 22, and 23 separate the survival of the terminal depth from the normalization accumulated along the observed prefix. The mature-window and initial-window cases cover prefixes on the two sides of the terminal depth $Q$ , the crude bound supplies a uniform starting point, and the refined statement packages these ingredients into the terminal-posterior control used below.

Lemma 24 [lem:rare-action-second-moment-exact] (Rare-action second moment).

Let $Q, k \in N$ , let $t_{0}, ζ, C \in R$ , and let $v \in {- 1, + 1}$ . Assume:

(Time scale.) $t_{0} > 0$ .

(Overlap exponent.) $ζ > 0$ .

(Radius constant.) $C > 1$ .

Write $L = exp (ζ)$ , let $ℓ = min {Q, k}$ , and consider the finite signed-depth model of Definition 18 at horizon $k + 1$ . For an action/reward-symbol prefix $o = ((a_{1}, y_{1}), \dots, (a_{k}, y_{k})),$ let $π_{k}^{v} (o)$ be its prefix mass under this model, and define the rare-action-history factor from Definition 25 by $a_{k} (o) = L^{- (Q - ℓ)} i : k - ℓ < i \leq k \prod 1 {a_{i} = 1},$ with the empty product equal to one. Then $o \sum π_{k}^{v} (o) a_{k} (o)^{2} = L^{- 2 (Q - ℓ)} (\frac{1}{L})^{∣ {i : k - ℓ < i \leq k} ∣},$ where the sum ranges over all action/reward-symbol prefixes of length $k$ .

⊢ Lean

Proof of Lemma 24.

Let $S := {i \in {1, \dots, k} : k - ℓ < i \leq k} .$ Reindex each observed prefix $o$ by its action sequence $a = (a_{1}, \dots, a_{k})$ and reward-symbol sequence $y = (y_{1}, \dots, y_{k})$ . This bijection gives $o \sum π_{k}^{v} (o) a_{k} (o)^{2} = a \sum y \sum π_{k}^{v} (a, y) (L^{- (Q - ℓ)} i \in S \prod 1 {a_{i} = 1})^{2} .$

For each action sequence $a$ , the product $I_{S} (a) := i \in S \prod 1 {a_{i} = 1}$ is either $0$ or $1$ . Hence $I_{S} (a)^{2} = I_{S} (a)$ , including the empty-product case $S = \emptyset$ , where $I_{S} (a) = 1$ . Therefore $o \sum π_{k}^{v} (o) a_{k} (o)^{2} = L^{- 2 (Q - ℓ)} a \sum I_{S} (a) y \sum π_{k}^{v} (a, y) .$

After summing over the reward symbols with the actions fixed, the observed prefix mass factors as the behavior action probability product: $y \sum π_{k}^{v} (a, y) = i = 1 \prod k b (a_{i}), b (1) = \frac{1}{L}, b (0) = 1 - \frac{1}{L} .$ This is the reward-summed prefix identity for the signed-depth family of Definition 18. Thus $o \sum π_{k}^{v} (o) a_{k} (o)^{2} = L^{- 2 (Q - ℓ)} a \sum (i = 1 \prod k b (a_{i})) (i \in S \prod 1 {a_{i} = 1}) .$

It remains to evaluate the action sum. Since the behavior action law is product-form, $a \sum (i = 1 \prod k b (a_{i})) (i \in S \prod 1 {a_{i} = 1}) = i = 1 \prod k a_{i} \in {0, 1} \sum b (a_{i}) {1 {a_{i} = 1}, 1, i \in S, i \in / S .$ For $i \in S$ , the inner sum is $b (1) = 1/ L$ ; for $i \in / S$ , it is $b (0) + b (1) = 1$ . Hence $a \sum (i = 1 \prod k b (a_{i})) (i \in S \prod 1 {a_{i} = 1}) = (\frac{1}{L})^{∣ S ∣} .$

Combining the preceding displays and recalling $S = {i : k - ℓ < i \leq k}$ gives $o \sum π_{k}^{v} (o) a_{k} (o)^{2} = L^{- 2 (Q - ℓ)} (\frac{1}{L})^{∣ {i : k - ℓ < i \leq k} ∣},$ as claimed.

∎

Lemma 24 gives the exact second moment of the visible rare-action factor. The displayed expression isolates the contribution of the forced terminal-depth action pattern, and this is the $L^{- Q}$ term that enters the observed KL calculation after summing conditional mean differences.

Lemma 25 [lem:finite-signed-depth-kl] (Finite observed KL bound).

Let $T, Q \in N$ and $t_{0}, ζ, C, K_{0} \in R$ . Write $α = e^{- 1/ t_{0}}$ , $L = e^{ζ}$ , $c_{0} = (1 - α) /4$ , and $ε_{C} = min {(C - 1) /2, 1/2}$ . Suppose that

(Positive scale.) $t_{0} > 0$ , $ζ > 0$ , $C > 1$ , and $K_{0} > 0$ .

(Uniform terminal posterior.) For every horizon $N$ , every alternative index $v \in {- 1, + 1}$ , every time $t < N$ , and every observed prefix $o$ of actions and two-symbol rewards before $t$ , the finite conditional terminal-depth probability $q_{t}^{v} (o)$ from Definition 25 satisfies $q_{t}^{v} (o) \leq K_{0} α^{Q} .$

Then the finite observed-word laws of the signed-depth alternatives of Definition 18, taken before the reward symbols are decoded as real rewards, satisfy $KL (P_{Q, + 1}^{fin, T} P_{Q, - 1}^{fin, T}) \leq \frac{4 c _{0}^{2} K _{0}^{2}}{1 - c _{0}^{2}} T ε_{C}^{2} α^{2 Q} L^{- Q} .$

⊢ Lean

Proof of Lemma 25.

Let $D := \frac{4 c _{0}^{2} K _{0}^{2}}{1 - c _{0}^{2}} .$ Since $t_{0} > 0$ , $α = exp (- 1/ t_{0})$ lies in $(0, 1)$ , and hence $0 < c_{0} = \frac{1 - α}{4} < 1, 0 \leq 1 - c_{0}^{2}, D \geq 0.$ Also $ε_{C}^{2} \geq 0$ , $α^{2 Q} \geq 0$ , and $L^{- Q} \geq 0$ . We prove the claimed bound by induction on the horizon $T$ .

Step 1. The zero-horizon case. For $T = 0$ , the observed word has a single possible value. The finite-word chain rule gives $KL (P_{Q, + 1}^{fin, 0} P_{Q, - 1}^{fin, 0}) = 0,$ so the desired inequality is immediate.

Step 2. Full support and projectivity. Fix $k \geq 0$ and assume the bound at horizon $k$ . Every observed word $w = ((x_{1}, a_{1}, y_{1}), \dots, (x_{k + 1}, a_{k + 1}, y_{k + 1}))$ has positive mass under each signed-depth alternative. Indeed, the observed coordinate has only one value, and one may choose a hidden trajectory with the displayed observed actions and reward symbols. The initial weights are positive because $0 < α < 1$ and $0 < ε_{C} \leq 1/2$ ; the behavior weights are positive because $L = exp (ζ) > 1$ ; and the selected hidden transitions have positive reset or continuation weight, with the case $Q = 0$ handled directly by the reset transition. Thus $P_{Q, v}^{fin, k + 1} (w) > 0, v \in {- 1, + 1} .$ Moreover, summing the chronological law over the last observed action-reward symbol gives the prefix law: $pref_{k} P_{Q, v}^{fin, k + 1} = P_{Q, v}^{fin, k} .$ Consequently the last-coordinate chain rule applies and yields $KL (P_{Q, + 1}^{fin, k + 1} P_{Q, - 1}^{fin, k + 1}) = KL (P_{Q, + 1}^{fin, k} P_{Q, - 1}^{fin, k}) + w \sum P_{Q, + 1}^{fin, k} (w) KL (P_{+, w}^{next} P_{-, w}^{next}),$ where $P_{v, w}^{next}$ is the conditional law of the next full observed symbol $(x, a, r) \in {0} \times {0, 1} \times {0, 1}$ after prefix $w$ .

Step 3. The one-step increment. For a prefix $w$ of length $k$ , write $o (w) = ((a_{1}, y_{1}), \dots, (a_{k}, y_{k}))$ for its action/reward-symbol projection and define $a_{k} (o) := L^{- (Q - m i n {Q, k})} i : k - m i n {Q, k} < i \leq k \prod 1 {a_{i} = 1} .$ For $v \in {- 1, + 1}$ , let $P_{v, w}^{ar}$ be the image of $P_{v, w}^{next}$ under the bijection ${0} \times {0, 1} \times {0, 1} ⟶ {0, 1} \times {0, 1}, (0, a, r) ⟼ (a, r) .$ The KL divergence is unchanged by this relabelling, so $KL (P_{+, w}^{next} P_{-, w}^{next}) = KL (P_{+, w}^{ar} P_{-, w}^{ar}) .$ For $a \in {0, 1}$ and reward symbol $r \in {0, 1}$ , write $\overset{y}{ˉ} (r) = - 1$ if $r = 0$ and $\overset{y}{ˉ} (r) = 1$ if $r = 1$ . Let $m_{v} (w) := E_{v} [Y_{k + 1} ∣ F_{k}^{obs} = o (w)]$ with the conditional ratio evaluated on the finite prefix mass; full support from Step 2 makes this ordinary conditional expectation on the present prefix. Summing the chronological law over all hidden states at time $k + 1$ gives, for every $(a, r)$ , $P_{v, w}^{ar} (a, r) = \frac{Pr _{v} { F _{k}^{obs} = o ( w ) , A _{k + 1} = a , Y _{k + 1} = y ˉ ( r )}}{Pr _{v} { F _{k}^{obs} = o ( w )}} = b (a) v Pr {Y_{k + 1} = \overset{y}{ˉ} (r) ∣ F_{k}^{obs} = o (w)} = b (a) \frac{1 + y ˉ ( r ) m _{v} ( w )}{2},$ where $b (1) = L^{- 1}$ and $b (0) = 1 - L^{- 1}$ . The second equality uses the behavior policy in Definition 18, which chooses the next action with mass $b (a)$ independently of the alternative after the observed prefix. The last equality is the two-symbol reward law determined by its mean: for a variable taking values ${- 1, + 1}$ , mass $(1 + m) /2$ is assigned to $+ 1$ and mass $(1 - m) /2$ to $- 1$ . The positivity assumptions give $∣ m_{v} (w) ∣ < 1$ , so all displayed masses are positive.

By Lemma 30, these means are $m_{+} (w) = c_{0} ε_{C} a_{k} (o (w)) q_{k + 1}^{+ 1} (o (w)), m_{-} (w) = - c_{0} ε_{C} a_{k} (o (w)) q_{k + 1}^{- 1} (o (w)) .$ Using the preceding mass formula and writing out the finite KL sum, $KL (P_{+, w}^{ar} P_{-, w}^{ar}) = a \in {0, 1} \sum r \in {0, 1} \sum b (a) \frac{1 + y ˉ ( r ) m _{+} ( w )}{2} lo g \frac{b ( a ) ( 1 + y ˉ ( r ) m _{+} ( w )) /2}{b ( a ) ( 1 + y ˉ ( r ) m _{-} ( w )) /2} = r \in {0, 1} \sum \frac{1 + y ˉ ( r ) m _{+} ( w )}{2} lo g \frac{1 + y ˉ ( r ) m _{+} ( w )}{1 + y ˉ ( r ) m _{-} ( w )} .$ Here $\sum_{a} b (a) = 1$ , and the positive common factor $b (a)$ cancels inside the logarithm. The standard two-point comparison, valid for $∣ m_{+} (w) ∣ < 1$ and $∣ m_{-} (w) ∣ < 1$ , yields $KL (P_{+, w}^{next} P_{-, w}^{next}) \leq \frac{( m _{+} ( w ) - m _{-} ( w ) ) ^{2}}{1 - m _{-} ( w ) ^{2}} .$

Step 4. The conditional chi-square calibration. Set $d (w) := c_{0} ε_{C} a_{k} (o (w)) .$ The signed-depth factors are nonnegative: $c_{0} > 0$ , $ε_{C} \geq 0$ , and $a_{k} (o (w)) \geq 0$ . The action factor also obeys $0 \leq a_{k} (o (w)) \leq 1, ε_{C} a_{k} (o (w)) \leq 1.$ Hence $0 \leq d (w) and d (w)^{2} = c_{0}^{2} (ε_{C} a_{k} (o (w)))^{2} \leq c_{0}^{2} .$ The terminal posteriors are probabilities, so $0 \leq q_{k + 1}^{v} (o (w)) \leq 1, v \in {- 1, + 1} .$ Together with the assumed posterior bound, $q_{k + 1}^{v} (o (w)) \leq K_{0} α^{Q}, v \in {- 1, + 1},$ this gives $q_{k + 1}^{+ 1} (o (w)) + q_{k + 1}^{- 1} (o (w)) \leq 2 K_{0} α^{Q} .$ Substituting the conditional-mean identities from Step 3, $(m_{+} (w) - m_{-} (w))^{2} = d (w)^{2} (q_{k + 1}^{+ 1} (o (w)) + q_{k + 1}^{- 1} (o (w)))^{2} = c_{0}^{2} ε_{C}^{2} a_{k} (o (w))^{2} (q_{k + 1}^{+ 1} (o (w)) + q_{k + 1}^{- 1} (o (w)))^{2} \leq 4 c_{0}^{2} K_{0}^{2} ε_{C}^{2} a_{k} (o (w))^{2} α^{2 Q} .$ The denominator is compared using both $d (w)^{2} \leq c_{0}^{2}$ and $q_{k + 1}^{- 1} (o (w))^{2} \leq 1$ : $1 - m_{-} (w)^{2} = 1 - d (w)^{2} q_{k + 1}^{- 1} (o (w))^{2} \geq 1 - c_{0}^{2} .$ Since $0 < c_{0} < 1$ , the lower bound $1 - c_{0}^{2}$ is positive. Applying the preceding numerator and denominator comparisons to the one-step KL bound from Step 3 gives $KL (P_{+, w}^{next} P_{-, w}^{next}) \leq \frac{4 c _{0}^{2} K _{0}^{2} ε _{C}^{2} a _{k} ( o ( w ) ) ^{2} α ^{2 Q}}{1 - c _{0}^{2}} = D ε_{C}^{2} α^{2 Q} a_{k} (o (w))^{2} .$

Step 5. Averaging over prefixes. For an action/reward-symbol prefix $o$ of length $k$ , define its finite prefix mass using the horizon $k + 1$ model by $π_{k, k + 1}^{+ 1} (o) := Q, + 1 Pr fin, k + 1 ((A_{1}, Y_{1}), \dots, (A_{k}, Y_{k}) = o) .$ For every observed word $w$ of length $k$ , the observed state coordinate is fixed, and the mass of $w$ under the horizon- $k$ observed law is $P_{Q, + 1}^{fin, k} (w) = π_{k, k + 1}^{+ 1} (o (w)) .$ Thus the average increment is bounded by $w \sum P_{Q, + 1}^{fin, k} (w) KL (P_{+, w}^{next} P_{-, w}^{next}) \leq D ε_{C}^{2} α^{2 Q} w \sum π_{k, k + 1}^{+ 1} (o (w)) a_{k} (o (w))^{2} .$ The map $w \mapsto o (w)$ is a bijection from length- $k$ observed words to length- $k$ action/reward-symbol prefixes, because the observed state coordinate has the unique value. Reindexing the finite sum gives $w \sum π_{k, k + 1}^{+ 1} (o (w)) a_{k} (o (w))^{2} = o \sum π_{k, k + 1}^{+ 1} (o) a_{k} (o)^{2} .$ By Lemma 24, with $ℓ = min {Q, k}$ , $o \sum π_{k, k + 1}^{+ 1} (o) a_{k} (o)^{2} = L^{- 2 (Q - ℓ)} (\frac{1}{L})^{∣ {i : k - ℓ < i \leq k} ∣} .$ The index set ${i : k - ℓ < i \leq k}$ contains exactly the final $ℓ$ positions of the length- $k$ prefix; if $ℓ = 0$ it is empty, and otherwise the shift $j \mapsto k - ℓ + j$ , $1 \leq j \leq ℓ$ , is a bijection. Therefore $o \sum π_{k, k + 1}^{+ 1} (o) a_{k} (o)^{2} = L^{- 2 (Q - ℓ)} L^{- ℓ} = L^{- 2 Q + ℓ} \leq L^{- Q},$ because $ℓ \leq Q$ and $L \geq 1$ . Hence the increment is bounded by $D ε_{C}^{2} α^{2 Q} L^{- Q} .$

Step 6. Closing the induction. Using the induction hypothesis and the increment bound, $KL (P_{Q, + 1}^{fin, k + 1} P_{Q, - 1}^{fin, k + 1}) \leq D k ε_{C}^{2} α^{2 Q} L^{- Q} + D ε_{C}^{2} α^{2 Q} L^{- Q} = D (k + 1) ε_{C}^{2} α^{2 Q} L^{- Q} .$ Substituting the definition of $D$ gives the asserted bound for $T = k + 1$ , completing the induction.

∎

Lemma 26 [lem:signed-depth-bootstrap-factor] (Uniform bootstrap factor).

Let $t_{0} > 0$ and write $α = e^{- 1/ t_{0}}$ , $c_{0} = (1 - α) /4$ , $η_{0} = α / (1 - c_{0})$ . Then there exists $K_{0} > 0$ such that $\frac{1}{( 1 - c _{0} ( α / ( 1 - c _{0} ) ) ^{Q} ) ^{Q}} \leq K_{0} (Q \in N) .$

⊢ Lean

Proof of Lemma 26.

Let $α = e^{- 1/ t_{0}}, c_{0} = \frac{1 - α}{4}, η_{0} = \frac{α}{1 - c _{0}} .$ We prove the stated uniform bound by constructing the constant explicitly.

Since $t_{0} > 0$ , one has $0 < α < 1$ . Hence $0 \leq c_{0} = \frac{1 - α}{4} \leq \frac{1}{4}, 1 - c_{0} = \frac{3 + α}{4} > 0,$ and therefore $0 \leq η_{0} = \frac{α}{1 - c _{0}} < 1,$ because $α < 1 - c_{0}$ is equivalent to $α < (3 + α) /4$ , or $α < 1$ .
The sequence $Q η_{0}^{Q}$ is bounded above. To see this in the same normalization, set $\overset{η}{ˉ} = \frac{1 + η _{0}}{2} .$ Then $0 \leq \overset{η}{ˉ} < 1$ and $∣ η_{0} ∣ < \overset{η}{ˉ}$ . Writing $Q η_{0}^{Q} = (Q (η_{0} / \overset{η}{ˉ})^{Q}) \overset{η}{ˉ}^{Q},$ the first factor tends to $0$ by the ratio test, since $\frac{( Q + 1 ) ( η _{0} / η ˉ ) ^{Q + 1}}{Q ( η _{0} / η ˉ ) ^{Q}} = (1 + \frac{1}{Q}) \frac{η _{0}}{η ˉ} ⟶ \frac{η _{0}}{η ˉ} < 1,$ while $\overset{η}{ˉ}^{Q} \to 0$ . Thus $Q η_{0}^{Q} \to 0$ , and in particular there is a finite $M_{⋆} \geq 0$ such that $Q η_{0}^{Q} \leq M_{⋆} (Q \in N) .$
Define $K_{0} = exp (2 c_{0} M_{⋆}) .$ Then $K_{0} > 0$ . Fix $Q \in N$ and put $x_{Q} = c_{0} η_{0}^{Q} .$ From $0 \leq c_{0} \leq 1/4$ and $0 \leq η_{0} < 1$ , we have $0 \leq x_{Q} \leq \frac{1}{4}, 1 - x_{Q} > 0.$ For every $x \in [0, 1/4]$ , $(1 - x)^{- 1} \leq 1 + 2 x \leq e^{2 x} .$ Indeed, the first inequality follows after multiplication by $1 - x > 0$ , since $(1 + 2 x) (1 - x) = 1 + x - 2 x^{2} \geq 1,$ and the second is the standard bound $1 + y \leq e^{y}$ with $y = 2 x$ . Applying this with $x = x_{Q}$ and raising to the $Q$ -th power gives $((1 - x_{Q})^{- 1})^{Q} \leq (e^{2 x_{Q}})^{Q} = e^{2 Q x_{Q}} .$
Finally, $2 Q x_{Q} = 2 c_{0} Q η_{0}^{Q} \leq 2 c_{0} M_{⋆},$ so monotonicity of the exponential yields $e^{2 Q x_{Q}} \leq e^{2 c_{0} M_{⋆}} = K_{0} .$ Combining the preceding displays, $\frac{1}{( 1 - c _{0} η _{0}^{Q} ) ^{Q}} = ((1 - x_{Q})^{- 1})^{Q} \leq K_{0} .$ Since $Q \in N$ was arbitrary, this $K_{0}$ proves the claim.

∎

Lemma 27 [lem:signed-depth-terminal-sign-invariant] (Terminal signed mass invariant).

Let $N, Q, k \in N$ , let $t_{0}, ζ, C \in R$ , let $v \in {- 1, + 1}$ , and consider the finite signed-depth model of Definition 18. For an observed prefix $o = ((a_{i}, y_{i}))_{i = 1}^{k}$ , write $λ_{v} (h; o)$ for the unnormalized filter mass of terminal hidden state $h$ , write $J (h) \in {0, \dots, Q}$ and $U (h) \in {- 1, + 1}$ for its depth and sign coordinates, and set $ε_{C} = min {\frac{C - 1}{2}, \frac{1}{2}} .$ Assume:

(Positive initial scale.) $t_{0} > 0$ .

(Positive overlap exponent.) $ζ > 0$ .

(Nontrivial overlap radius.) $C > 1$ .

With $L = e^{ζ}$ , define the depth- $Q$ action-history factor $a^{(Q)} (o) = L^{- m a x {Q - k, 0}} 1 \leq i \leq k i > k - m i n {Q, k} \prod 1 {a_{i} = 1} .$ Then $h : J (h) = Q \sum U (h) λ_{v} (h; o) = ε_{C} a^{(Q)} (o) h : J (h) = Q \sum λ_{v} (h; o) .$

⊢ Lean

Proof of Lemma 27.

Put $α = e^{- 1/ t_{0}}$ and $L = e^{ζ}$ . For a prefix $ω = ((\tilde{a}_{i}, \tilde{y}_{i}))_{i = 1}^{m}$ and a depth $j \in {0, \dots, Q}$ , define $Un_{j} (ω) := h : J (h) = j \sum λ_{v} (h; ω), Sg_{j} (ω) := h : J (h) = j \sum U (h) λ_{v} (h; ω),$ and $A_{j} (ω) := L^{- (j - m i n {j, m})} i = m - m i n {j, m} + 1 \prod m 1 {\tilde{a}_{i} = 1},$ with the empty product equal to one. Thus $A_{Q} (o) = a^{(Q)} (o)$ .

We first prove the depthwise invariant $Sg_{j} (ω) = ε_{C} A_{j} (ω) Un_{j} (ω) for every m, ω, and j \in {0, \dots, Q} .$ For $m = 0$ , the initial mass at depth $j$ and sign $u \in {- 1, + 1}$ is $w_{j} \frac{1 + u ε _{C} L ^{- j}}{2} .$ Summing over $u = + 1$ and $u = - 1$ gives $Un_{j} (ω) = w_{j}, Sg_{j} (ω) = w_{j} ε_{C} L^{- j} = ε_{C} A_{j} (ω) Un_{j} (ω),$ since $A_{j} (ω) = L^{- j}$ when the prefix is empty.
Suppose the invariant holds for every prefix of length $m$ , and let $ω^{+} = ((\tilde{a}_{i}, \tilde{y}_{i}))_{i = 1}^{m + 1}$ . Write $ω$ for its first $m$ observations and $a_{+} = \tilde{a}_{m + 1}$ . At reset depth $0$ , direct summation of the two reset signs in Definition 18 gives, for each previous hidden state $h$ , $u^{'} \in {- 1, + 1} \sum u^{'} p {J^{'} = 0, U^{'} = u^{'} ∣ h, a_{+}} = ε_{C} u^{'} \in {- 1, + 1} \sum p {J^{'} = 0, U^{'} = u^{'} ∣ h, a_{+}} .$ After multiplying by the previous unnormalized filter mass, the behavior-policy weight, and the reward-emission weight, and then summing over $h$ , this yields $Sg_{0} (ω^{+}) = ε_{C} Un_{0} (ω^{+}) .$ Because $A_{0} (ω^{+}) = 1$ , the invariant holds at depth $0$ .
For a positive target depth, write it as $j + 1$ with $j < Q$ . The last observation of $ω^{+}$ is $(a_{+}, \tilde{y}_{m + 1})$ , and the preceding prefix $ω$ is the restriction of $ω^{+}$ to its first $m$ coordinates. Let $Γ_{a_{+}} ((δ, u), (δ^{'}, u^{'}))$ be the hidden-transition weight from $(δ, u)$ to $(δ^{'}, u^{'})$ under action $a_{+}$ , for $δ, δ^{'} \in {0, \dots, Q}$ and $u, u^{'} \in {- 1, + 1}$ . The raw filter recursion gives $Un_{j + 1} (ω^{+}) = b (a_{+}) δ = 0 \sum Q u \in {- 1, + 1} \sum λ_{v} ((δ, u); ω) \frac{1 + y ~ _{m + 1} v c _{0} u 1 { δ = Q }}{2} u^{'} \in {- 1, + 1} \sum Γ_{a_{+}} ((δ, u), (j + 1, u^{'})),$ where $b (a_{+})$ is the behavior-policy probability of the observed action and $c_{0} = (1 - α) /4$ . By the transition weights in Definition 18, reaching the positive depth $j + 1$ is possible exactly by advancing from depth $j$ , and $u^{'} \in {- 1, + 1} \sum Γ_{a_{+}} ((δ, u), (j + 1, u^{'})) = α 1 {δ = j} .$ Because $j < Q$ , the reward factor on the surviving terms is $1/2$ . Hence $Un_{j + 1} (ω^{+}) = b (a_{+}) α \frac{1}{2} Un_{j} (ω) .$
With the same notation, the signed advancing aggregate is obtained by inserting the new sign in the inner sum: $Sg_{j + 1} (ω^{+}) = b (a_{+}) δ = 0 \sum Q u \in {- 1, + 1} \sum λ_{v} ((δ, u); ω) \frac{1 + y ~ _{m + 1} v c _{0} u 1 { δ = Q }}{2} u^{'} \in {- 1, + 1} \sum u^{'} Γ_{a_{+}} ((δ, u), (j + 1, u^{'})) .$ Again only $δ = j$ survives. For that depth, the transition weights in Definition 18 give $u^{'} \in {- 1, + 1} \sum u^{'} Γ_{a_{+}} ((δ, u), (j + 1, u^{'})) = α 1 {δ = j} 1 {a_{+} = 1} u .$ Indeed, action $1$ preserves the old sign on an advancing transition, while action $0$ assigns equal total advancing weight to the two possible new signs. Using once more that the reward factor is $1/2$ at depth $j < Q$ , we obtain $Sg_{j + 1} (ω^{+}) = b (a_{+}) α \frac{1}{2} 1 {a_{+} = 1} Sg_{j} (ω) .$
The action-history factor has the corresponding one-step recursion. From the definition of $A_{j}$ , $A_{j + 1} (ω^{+}) = L^{- ((j + 1) - m i n {j + 1, m + 1})} i = m + 2 - m i n {j + 1, m + 1} \prod m + 1 1 {\tilde{a}_{i} = 1} .$ The identities $min {j + 1, m + 1} = min {j, m} + 1, (j + 1) - min {j + 1, m + 1} = j - min {j, m},$ and $m + 2 - min {j + 1, m + 1} = m + 1 - min {j, m}$ show that the displayed product is the product defining $A_{j} (ω)$ , followed by the last factor $1 {a_{+} = 1}$ . Thus $A_{j + 1} (ω^{+}) = A_{j} (ω) 1 {a_{+} = 1},$ with the same empty-product convention as before.
Applying the induction hypothesis at depth $j$ to the preceding prefix $ω$ gives $Sg_{j + 1} (ω^{+}) = b (a_{+}) α \frac{1}{2} 1 {a_{+} = 1} ε_{C} A_{j} (ω) Un_{j} (ω) .$ The unsigned recursion and the action-factor recursion give $ε_{C} A_{j + 1} (ω^{+}) Un_{j + 1} (ω^{+}) = ε_{C} A_{j} (ω) 1 {a_{+} = 1} b (a_{+}) α \frac{1}{2} Un_{j} (ω),$ which is the same quantity. Equivalently, splitting on the last action, when $a_{+} = 1$ the common factor is $b (a_{+}) α /2$ times the induction hypothesis, and when $a_{+} = 0$ both sides are zero. This completes the induction.
Finally take $j = Q$ and $ω = o$ . Reindexing the hidden-state sum by its depth and sign coordinates gives $h : J (h) = Q \sum λ_{v} (h; o) = Un_{Q} (o), h : J (h) = Q \sum U (h) λ_{v} (h; o) = Sg_{Q} (o) .$ The depthwise invariant at $j = Q$ , together with $A_{Q} (o) = a^{(Q)} (o)$ , therefore gives $h : J (h) = Q \sum U (h) λ_{v} (h; o) = ε_{C} a^{(Q)} (o) h : J (h) = Q \sum λ_{v} (h; o),$ as claimed.

∎

Lemma 28 [lem:signed-depth-next-reward-numerator] (Next reward numerator).

Let $U, k, N, Q \in N$ , let $v \in {- 1, + 1}$ , and consider the finite signed-depth model of Definition 18 with horizon $N$ , terminal depth $Q$ , initial weights $p_{v}$ , behavior policy $b$ , and reward-symbol/hidden-state kernel $K_{v}$ . Suppose that

(Positive time scale.) $t_{0} > 0$ .

(Positive behavior perturbation.) $ζ > 0$ .

(Overlap radius.) $C > 1$ .

Write $c_{0} (t_{0}) = \frac{1 - α}{4},$ where $α$ is the signed-depth mixing parameter, and write $ϕ (0) = - 1$ , $ϕ (1) = 1$ . For any observed action/reward-symbol prefix $o \in ({0, 1} \times {0, 1})^{k}$ , let $λ_{v} (h; o)$ denote the unnormalized signed-depth filter mass, after the prefix $o$ , assigned to observed state $0$ and packed hidden state $h \in {0, \dots, 2 (Q + 1) - 1}$ . Decode the packed hidden state by $J (h) = ⌊ h /2 ⌋, U (h) = {+ 1, - 1, h \equiv 1 (mod 2), h \equiv 0 (mod 2) .$ Then $τ \sum 1 {((a_{i}, r_{i}))_{i = 1}^{k} (τ) = o} ω_{v} (τ) ϕ (r_{k + 1} (τ)) = v c_{0} (t_{0}) h = 0 \sum 2 (Q + 1) - 1 1 {J (h) = Q} U (h) λ_{v} (h; o),$ where the outer sum is over all finite signed-depth trajectories $τ$ of horizon $U + k + 1$ , and $ω_{v} (τ)$ is their chronological path weight under $p_{v}$ , $b$ , and $K_{v}$ .

⊢ Lean

Proof of Lemma 28.

1. Let $Σ_{k}$ be the set of joint-state prefixes $σ = (σ_{0}, \dots, σ_{k}), σ_{i} = (0, h_{i}), h_{i} \in {0, \dots, 2 (Q + 1) - 1} .$ For such a prefix, write $W_{o} (σ)$ for the chronological weight of realizing the observed prefix $o$ and ending with hidden state $h_{k}$ . Summing first over all continuations beyond the observed prefix gives $τ \sum 1 {((a_{i}, r_{i}))_{i = 1}^{k} (τ) = o} ω_{v} (τ) ϕ (r_{k + 1} (τ)) = σ \in Σ_{k} \sum W_{o} (σ) a \in {0, 1} \sum b (a) y \in {0, 1} \sum ϕ (y) s^{'} \sum K_{v} ((0, h_{k}), a) (y, s^{'}) .$ Indeed, partition trajectories by their first $k$ observed action–reward symbols and by the joint-state prefix through time $k$ . The chronological product factors into $W_{o} (σ)$ , the next behavior probability, and the next kernel probability; summing over the future hidden state $s^{'}$ and then over $a, y$ gives the displayed identity.

2. For every packed hidden state $h$ , the signed-depth reward kernel in Definition 18 satisfies $a \in {0, 1} \sum b (a) y \in {0, 1} \sum ϕ (y) s^{'} \sum K_{v} ((0, h), a) (y, s^{'}) = v c_{0} (t_{0}) U (h) 1 {J (h) = Q} .$ To see this, use the product form of the kernel: the reward-symbol weight depends on $h$ and $v$ , while the hidden-transition weights sum to one for each current state and action. Hence the left-hand side reduces to $(a \in {0, 1} \sum b (a)) y \in {0, 1} \sum ϕ (y) \frac{1 + ϕ ( y ) v c _{0} ( t _{0} ) U ( h ) 1 { J ( h ) = Q }}{2} .$ The behavior probabilities sum to one, and since $ϕ (0) = - 1$ , $ϕ (1) = 1$ , $y \in {0, 1} \sum ϕ (y) \frac{1}{2} = 0, y \in {0, 1} \sum ϕ (y)^{2} \frac{v c _{0} ( t _{0} ) U ( h ) 1 { J ( h ) = Q }}{2} = v c_{0} (t_{0}) U (h) 1 {J (h) = Q} .$ This proves the displayed one-step payoff identity.

3. Substituting the one-step identity into the prefix decomposition gives $τ \sum 1 {((a_{i}, r_{i}))_{i = 1}^{k} (τ) = o} ω_{v} (τ) ϕ (r_{k + 1} (τ)) = v c_{0} (t_{0}) σ \in Σ_{k} \sum W_{o} (σ) U (h_{k}) 1 {J (h_{k}) = Q} .$ The observed state alphabet has one element, so each terminal joint state is uniquely of the form $(0, h_{k})$ ; this is the only bookkeeping hidden in the notation above.

4. By definition of the unnormalized filter mass, $λ_{v} (h; o) = σ \in Σ_{k} : h_{k} = h \sum W_{o} (σ) .$ Therefore $h = 0 \sum 2 (Q + 1) - 1 1 {J (h) = Q} U (h) λ_{v} (h; o) = σ \in Σ_{k} \sum W_{o} (σ) U (h_{k}) 1 {J (h_{k}) = Q} .$ This is just regrouping the preceding prefix sum by its terminal packed hidden state.

5. Combining the last two displays yields $τ \sum 1 {((a_{i}, r_{i}))_{i = 1}^{k} (τ) = o} ω_{v} (τ) ϕ (r_{k + 1} (τ)) = v c_{0} (t_{0}) h = 0 \sum 2 (Q + 1) - 1 1 {J (h) = Q} U (h) λ_{v} (h; o),$ as claimed.

∎

Lemma 29 [lem:signed-depth-terminal-prefix-mass] (Terminal prefix mass identity).

Let $U, k, Q \in N$ with $U \geq 1$ , let $t_{0}, ζ, C \in R$ , let $v \in {- 1, + 1}$ , and consider the finite signed-depth model of Definition 18 at horizon $U + k$ . For an action/reward-symbol prefix $o$ of length $k$ , define the joint prefix-and-terminal-depth mass at the cut after the first $k$ epochs by $m_{v}^{Q} (o) = v Pr {(A_{1}, R_{1}), \dots, (A_{k}, R_{k}) = o, J_{k + 1} = Q} .$ For each hidden coordinate $h \in {0, \dots, 2 (Q + 1) - 1}$ , let $J (h) = ⌊ h /2 ⌋$ , and let $λ_{v} (h; o)$ be the unnormalized filter mass, as in Definition 25, assigned after observing $o$ to the cut state with hidden coordinate $h$ . Then $m_{v}^{Q} (o) = h = 0 \sum 2 (Q + 1) - 1 1 {J (h) = Q} λ_{v} (h; o) .$

⊢ Lean

Proof of Lemma 29.

1. Let $Γ_{k + 1}$ denote the set of joint-state prefixes $γ = (S_{1}, \dots, S_{k + 1}), S_{i} \in {0} \times {0, \dots, 2 (Q + 1) - 1} .$ For such a prefix, write $W_{v} (γ; o)$ for the real chronological weight of the prefix whose observed action/reward word is $o$ . Summing the full trajectory law over all continuations after the cut gives $m_{v}^{Q} (o) = γ \in Γ_{k + 1} \sum 1 {J (S_{k + 1}^{hid}) = Q} W_{v} (γ; o) .$ This is exactly the fixed-prefix expansion of the terminal-depth event at the cut after $k$ observed epochs; the hypothesis $U \geq 1$ ensures that this cut is a valid time in the horizon $U + k$ .

2. For a joint state $s \in {0} \times {0, \dots, 2 (Q + 1) - 1}$ , define $Λ_{v} (s; o) := γ \in Γ_{k + 1} \sum 1 {S_{k + 1} = s} W_{v} (γ; o) .$ Then $s \sum 1 {J (s^{hid}) = Q} Λ_{v} (s; o) = γ \in Γ_{k + 1} \sum 1 {J (S_{k + 1}^{hid}) = Q} W_{v} (γ; o) .$ Indeed, interchange the two finite sums. For each fixed prefix $γ$ , the inner sum over $s$ has exactly one nonzero contribution, namely $s = S_{k + 1}$ , and therefore returns the displayed indicator and weight.

3. Combining the preceding two displays yields $m_{v}^{Q} (o) = s \in {0} \times {0, \dots, 2 (Q + 1) - 1} \sum 1 {J (s^{hid}) = Q} Λ_{v} (s; o) .$ The observed-state coordinate has the singleton value $0$ , so this sum is $m_{v}^{Q} (o) = h = 0 \sum 2 (Q + 1) - 1 1 {J (h) = Q} Λ_{v} ((0, h); o) .$ For the signed-depth filter at the cut after the word $o$ , the unnormalized mass assigned to hidden coordinate $h$ is the terminal-state prefix mass at the joint state $(0, h)$ : $λ_{v} (h; o) := γ \in Γ_{k + 1} \sum 1 {S_{k + 1} = (0, h)} W_{v} (γ; o) .$ Thus $λ_{v} (h; o) = Λ_{v} ((0, h); o)$ for every $h$ , and the claimed identity follows.

∎

Lemma 30 [lem:signed-depth-conditional-reward-mean] (Conditional reward mean identity).

Let $T, Q \in N$ , let $t_{0}, ζ, C \in R$ , and let $v$ be a Boolean alternative, written through $σ (v) = {1, - 1, v = true, v = false .$ Assume

(Positive baseline.) $t_{0} > 0$ .

(Positive policy scale.) $ζ > 0$ .

(Overlap radius.) $C > 1$ .

For every epoch $t < T$ and every observed prefix $o \in {false, true}^{t} \times {0, 1}^{t}$ , the conditional next-reward mean satisfies $E_{v} [Y_{t} ∣ F_{t - 1}^{obs} = o] = σ (v) c_{0} (t_{0}) ε_{C} (C) a_{t}^{(Q)} (o) q_{t}^{v} (o),$ where the left-hand side is the totalized ratio defining the finite-prefix conditional reward mean, $c_{0} (t_{0}) = (1 - α) /4$ , $ε_{C} (C) = min {(C - 1) /2, 1/2}$ , and $a_{t}^{(Q)} (o)$ and $q_{t}^{v} (o)$ are the rare-action-history factor and terminal posterior of Definition 25.

⊢ Lean

Proof of Lemma 30.

Write the epoch as a finite ordinal $n_{pre} < T$ , and set $n_{tail} := T - (n_{pre} + 1) .$ Then $T = n_{tail} + (n_{pre} + 1)$ . Thus it is enough to prove the identity at the canonical cut after an observed prefix of length $n_{pre}$ , with at least the next epoch present.
Fix such a canonical horizon and write the observed prefix as $o = ((a_{i}, r_{i}))_{i = 1}^{k}$ , where $a_{i} \in {0, 1}$ , $r_{i} \in {0, 1}$ , and the real reward map is $ϕ (0) = - 1, ϕ (1) = 1.$ Let $T_{n_{tail}, k}^{+}$ be the finite trajectory space with horizon $n_{tail} + (k + 1)$ . Its elements record the initial hidden state, the behavior actions, reward symbols, and subsequent hidden states through the remaining epochs of the finite signed-depth weights displayed below. For every $Q \in N$ , the hidden coordinates are $(J, U) \in {0, \dots, Q} \times {- 1, + 1}$ . When $Q \geq 1$ , these weights are the signed-depth construction of Definition 18; when $Q = 0$ , the same formulas are read on the one-point depth set ${0}$ , so only the terminal-depth reset line can occur. For $τ \in T_{n_{tail}, k}^{+}$ , define $Obs_{pre} (τ) := ((a_{1} (τ), r_{1} (τ)), \dots, (a_{k} (τ), r_{k} (τ))), r_{next} (τ) := r_{k + 1} (τ),$ and let $ω_{v} (τ)$ be the product of the signed-depth initial hidden-state weight, the behavior action probabilities, and the reward-symbol/hidden-transition weights along $τ$ . Thus, if $h_{0}, \dots, h_{n_{tail} + k + 1}$ are the hidden states in $τ$ , $ω_{v} (τ) = μ_{v} (h_{0}) i = 1 \prod n_{tail} + k + 1 π_{ζ} (a_{i} (τ)) m_{v} (r_{i} (τ), h_{i} ∣ h_{i - 1}, a_{i} (τ)),$ where $μ_{v}$ is the initial hidden-state law with weights $μ_{v} (j, u) = \frac{( 1 - α ) α ^{j}}{1 - α ^{Q + 1}} \frac{1 + u ε _{C} ( C ) L ^{- j}}{2}, 0 \leq j \leq Q, u \in {- 1, + 1},$ $π_{ζ} (1) = L^{- 1}$ , $π_{ζ} (0) = 1 - L^{- 1}$ , and $m_{v} (r, h^{'} ∣ h, a)$ is the following one-step reward-symbol and next-hidden-state weight. If $h = (j, u)$ , $h^{'} = (j^{'}, u^{'})$ , and $ϕ (0) = - 1$ , $ϕ (1) = 1$ , then $m_{v} (r, h^{'} ∣ h, a) = \frac{1 + ϕ ( r ) σ ( v ) c _{0} ( t _{0} ) u 1 { j = Q }}{2} κ (h^{'} ∣ h, a),$ where $κ (0, u^{'} ∣ Q, u, a) = (1 + u^{'} ε_{C} (C)) /2$ ; for $j < Q$ , $κ (0, u^{'} ∣ j, u, a) = (1 - α) \frac{1 + u ^{'} ε _{C} ( C )}{2},$ and $κ (j + 1, u^{'} ∣ j, u, 1) = α 1 {u^{'} = u}, κ (j + 1, u^{'} ∣ j, u, 0) = \frac{α}{2},$ with all other hidden transitions assigned weight zero. These formulas are valid for all $Q \in N$ ; if $Q = 0$ , only the reset line $j = Q$ occurs. Summing the two reward weights gives one, and the displayed hidden-transition weights sum to one for each hidden state and action. Hence $r \in {0, 1} \sum h^{'} \sum m_{v} (r, h^{'} ∣ h, a) = 1 for every hidden state h and action a .$ The numerator of the totalized finite-prefix conditional reward mean is $N_{v} (o) := τ \in T_{n_{tail}, k}^{+} \sum 1 {Obs_{pre} (τ) = o} ϕ (r_{next} (τ)) ω_{v} (τ) .$ For a hidden prefix $η = (h_{0}, \dots, h_{k})$ , define its observed-prefix weight by $Ω_{v} (η; o) := μ_{v} (h_{0}) i = 1 \prod k π_{ζ} (a_{i}) m_{v} (r_{i}, h_{i} ∣ h_{i - 1}, a_{i}),$ with value $μ_{v} (h_{0})$ when $k = 0$ . The unnormalized filter mass at the cut is $λ_{v} (h; o) := η : h_{k} = h \sum Ω_{v} (η; o), Λ_{v} (o) := h \sum λ_{v} (h; o), Λ_{v, Q} (o) := h : J (h) = Q \sum λ_{v} (h; o),$ where $J (h) \in {0, \dots, Q}$ and $U (h) \in {- 1, + 1}$ are the depth and sign coordinates of the hidden state $h$ .

We next decompose the finite trajectories at the cut. For a hidden state $h$ and integer $m \geq 0$ , set $Ξ_{m} (h) := (a_{1}, r_{1}, h_{1}), \dots, (a_{m}, r_{m}, h_{m}) \sum j = 1 \prod m π_{ζ} (a_{j}) m_{v} (r_{j}, h_{j} ∣ h_{j - 1}, a_{j}), h_{0} = h,$ with $Ξ_{0} (h) = 1$ . Since $\sum_{a} π_{ζ} (a) = 1$ and the displayed normalization of $m_{v}$ holds at each state-action pair, induction gives $Ξ_{m} (h) = 1 for every m \geq 0 and every h .$ Splitting $N_{v} (o)$ according to the hidden prefix $η$ , the next action $a$ , the next reward symbol $r$ , and the next hidden state $h^{'}$ , then summing the remaining suffix by $Ξ_{n_{tail}} (h^{'}) = 1$ , yields $N_{v} (o) = η \sum Ω_{v} (η; o) a \in {0, 1} \sum π_{ζ} (a) r \in {0, 1} \sum ϕ (r) h^{'} \sum m_{v} (r, h^{'} ∣ h_{k}, a) .$ This is a partition of the same finite set of trajectories: the data $(η, a, r, h^{'})$ give the first $k + 1$ epochs, and the suffix sum runs over all chronological continuations from $h^{'}$ .

For fixed $h$ and $a$ , the displayed one-step formula factors $m_{v} (r, h^{'} ∣ h, a)$ into the reward weight and a hidden-transition probability. Hence $h^{'} \sum m_{v} (r, h^{'} ∣ h, a) = \frac{1 + ϕ ( r ) σ ( v ) c _{0} ( t _{0} ) U ( h ) 1 { J ( h ) = Q }}{2} .$ Using $\sum_{a} π_{ζ} (a) = 1$ , $\sum_{r} ϕ (r) /2 = 0$ , and $\sum_{r} ϕ (r)^{2} /2 = 1$ , the inner one-step payoff is $a \sum π_{ζ} (a) r \sum ϕ (r) h^{'} \sum m_{v} (r, h^{'} ∣ h, a) = a \sum π_{ζ} (a) r \sum ϕ (r) \frac{1 + ϕ ( r ) σ ( v ) c _{0} ( t _{0} ) U ( h ) 1 { J ( h ) = Q }}{2} = σ (v) c_{0} (t_{0}) U (h) 1 {J (h) = Q} .$ Therefore $N_{v} (o) = σ (v) c_{0} (t_{0}) η \sum Ω_{v} (η; o) U (h_{k}) 1 {J (h_{k}) = Q} = σ (v) c_{0} (t_{0}) h : J (h) = Q \sum U (h) λ_{v} (h; o) .$ This is the numerator identity supplied by Lemma 28; the displayed derivation records the finite-trajectory regrouping used at the canonical cut. The denominator and the terminal-depth quotient are identified below.
Lemma 27 applies under $t_{0} > 0$ , $ζ > 0$ , and $C > 1$ , and gives the prefix-level identity $h : J (h) = Q \sum U (h) λ_{v} (h; o) = ε_{C} (C) a_{pre}^{(Q)} (o) h : J (h) = Q \sum λ_{v} (h; o),$ where, for the length- $k$ prefix $o = ((a_{i}, r_{i}))_{i = 1}^{k}$ , $a_{pre}^{(Q)} (o) = L^{- m a x {Q - k, 0}} 1 \leq i \leq k i > k - m i n {Q, k} \prod 1 {a_{i} = 1} .$ At the canonical cut, the time-indexed factor of Definition 25 is the same quantity: $a_{t}^{(Q)} (o) = L^{- (Q - m i n {Q, k})} 1 \leq i \leq k i > k - m i n {Q, k} \prod 1 {a_{i} = 1} = a_{pre}^{(Q)} (o),$ since $Q - min {Q, k} = max {Q - k, 0}$ . Thus $h : J (h) = Q \sum U (h) λ_{v} (h; o) = ε_{C} (C) a_{t}^{(Q)} (o) Λ_{v, Q} (o) .$ The empty-prefix case is included by the empty-product convention in Definition 25.
It remains to identify the denominator and terminal-depth numerator. For a trajectory $τ \in T_{n_{tail}, k}^{+}$ , let $h_{cut} (τ) := h_{k} (τ)$ be the hidden state after the first $k$ observed epochs. In the one-based indexing used for the signed-depth path statement, this same cut state is $H_{k + 1}$ ; hence the event $J (h_{cut} (τ)) = Q$ is exactly the event written there as $J_{k + 1} = Q$ . Define the finite prefix mass and the terminal-depth prefix mass by $Π_{v} (o) := τ \in T_{n_{tail}, k}^{+} \sum 1 {Obs_{pre} (τ) = o} ω_{v} (τ),$ and $Π_{v, Q} (o) := τ \in T_{n_{tail}, k}^{+} \sum 1 {Obs_{pre} (τ) = o} 1 {J (h_{cut} (τ)) = Q} ω_{v} (τ) .$ The same cut decomposition as above, now without weighting by the next reward, gives $Π_{v} (o) = η \sum Ω_{v} (η; o) (a, r, h^{'}) \sum π_{ζ} (a) m_{v} (r, h^{'} ∣ h_{k}, a) Ξ_{n_{tail}} (h^{'}) = η \sum Ω_{v} (η; o) = h \sum λ_{v} (h; o) = Λ_{v} (o) .$ Indeed, the first sum partitions trajectories by their length- $k$ hidden prefix, next action, next reward symbol, and next hidden state; the continuation has total mass $Ξ_{n_{tail}} (h^{'}) = 1$ , and the next one-step mass sums to one. Carrying the terminal-depth indicator at the cut through the same partition gives $Π_{v, Q} (o) = η \sum Ω_{v} (η; o) 1 {J (h_{k}) = Q} (a, r, h^{'}) \sum π_{ζ} (a) m_{v} (r, h^{'} ∣ h_{k}, a) Ξ_{n_{tail}} (h^{'}) = η \sum Ω_{v} (η; o) 1 {J (h_{k}) = Q} = h : J (h) = Q \sum λ_{v} (h; o) = Λ_{v, Q} (o) .$ Thus $Π_{v} (o)$ is the mass of the observed-prefix event and $Π_{v, Q} (o)$ is the mass of the same event together with terminal depth at the cut. The latter identity is exactly the terminal-prefix mass identification in Lemma 29 for the canonical horizon.

The finite-prefix conditional reward mean is the totalized quotient $E_{v} [Y_{t} ∣ F_{t - 1}^{obs} = o] = \frac{N _{v} ( o )}{Π _{v} ( o )},$ where real division is totalized by $x /0 = 0$ . The terminal posterior in Definition 25 is read at this finite prefix through the same totalized convention: $q_{t}^{v} (o) = \frac{Π _{v, Q} ( o )}{Π _{v} ( o )} .$ If $Π_{v} (o) = 0$ , then $Π_{v, Q} (o) = 0$ because the terminal-depth prefix event is contained in the prefix event; for $Π_{v} (o) > 0$ , these quotients are the usual conditional averages. Combining the numerator identity, Lemma 27, and $Π_{v, Q} (o) = Λ_{v, Q} (o)$ , we obtain $E_{v} [Y_{t} ∣ F_{t - 1}^{obs} = o] = \frac{σ ( v ) c _{0} ( t _{0} ) ε _{C} ( C ) a _{t}^{(Q)} ( o ) Π _{v, Q} ( o )}{Π _{v} ( o )} = σ (v) c_{0} (t_{0}) ε_{C} (C) a_{t}^{(Q)} (o) q_{t}^{v} (o) .$ This is the claimed identity for the original $T$ and $t < T$ .

∎

Lemma 26 turns the finite-prefix normalization in Lemma 23 into a constant depending on the mixing scale. Lemmas 27, 28, and 29 connect the signed terminal mass, the next reward numerator, and the prefix mass at terminal depth. These identities convert the latent terminal sign into the observable conditional reward mean in Lemma 30, while Lemma 25 converts the resulting terminal-posterior control into a finite observed-word divergence bound. The product $α^{2 Q} L^{- Q}$ reflects the joint requirement that terminal depth survives and that the visible action history carries the rare sequence needed to reveal it.

The likelihood calculation next fixes the observed signed-depth law and combines the exact observed reward mean with a bound on the divergence between the two signs.

Definition 28 [synth_18] (Observed signed-depth law).

For $T, Q \in N$ , $t_{0}, ζ, C \in R$ , and two-point alternative index $v$ , define $P_{Q, v}^{obs, T}$ to be the observed-path law obtained by applying the observed-coordinate projection to the trajectory law of the embedded finite signed-depth model with horizon $T$ , parameters $(t_{0}, ζ, C, Q)$ , and alternative index $v$ .

⊢ Lean

The law in Definition 28 is the distribution used in the two-point information comparison. It records the observable image of the embedded signed-depth experiment, so the divergence calculation is stated on the same data observed by an estimator.

Lemma 31 [lem:observed-path-kl] (Observed-path KL bound).

Fix $t_{0} > 0$ , and write $α = exp (- 1/ t_{0})$ . There exists a constant $K_{0} > 0$ , depending on $t_{0}$ , such that whenever the following conditions hold:

(Overlap exponent.) $ζ > 0$ , and $L = exp (ζ)$ .

(Reset amplitude.) $C > 1$ , and $ε_{C} = min {(C - 1) /2, 1/2}$ .

(Sequential ignorability.) For every horizon $T$ , terminal depth $Q \geq 1$ , and alternative $v \in {- 1, + 1}$ , the signed-depth experiment in Definition 18 satisfies Assumption 2.

(Stationary start.) For every horizon $T$ , terminal depth $Q \geq 1$ , and alternative $v \in {- 1, + 1}$ , the signed-depth experiment in Definition 18 satisfies Assumption 3.

there exists a constant $B_{0} > 0$ , depending on $t_{0}, ζ, C$ , such that for every $T$ and every $Q \geq 1$ , with $P_{Q, v}^{obs, T}$ denoting the observed-path law of the signed-depth alternative $v$ , $KL (P_{Q, + 1}^{obs, T} P_{Q, - 1}^{obs, T}) \leq B_{0} T α^{2 Q} L^{- Q} .$ Moreover, for each $v \in {- 1, + 1}$ , each $t < T$ , and each observed prefix $o \in F_{t - 1}^{obs}$ , the conditional reward mean satisfies the exact finite-prefix identity $E_{v} [Y_{t} ∣ F_{t - 1}^{obs} = o] = v c_{0} ε_{C} a_{t}^{(Q)} (o) q_{t}^{v} (o),$ where $c_{0} = (1 - α) /4$ , and $a_{t}^{(Q)}$ and $q_{t}^{v}$ are the rare-action-history factor and terminal-depth posterior from Definition 25. The posterior satisfies $q_{t}^{v} (o) \leq K_{0} α^{Q} .$ The same observed-path laws also satisfy $KL (P_{Q, + 1}^{obs, T} P_{Q, - 1}^{obs, T}) \leq \frac{4 c _{0}^{2} K _{0}^{2}}{1 - c _{0}^{2}} T ε_{C}^{2} α^{2 Q} L^{- Q} .$

⊢ Lean

Proof of Lemma 31.

Fix $t_{0} > 0$ , put $α = exp (- 1/ t_{0})$ , and set $c_{0} = (1 - α) /4$ .

1. By Lemma 26, there is a constant $K_{0} > 0$ , depending only on $t_{0}$ , such that for every $Q \in N$ , $\frac{1}{{ 1 - c _{0} ( α / ( 1 - c _{0} ) ) ^{Q} } ^{Q}} \leq K_{0} .$ Fix $ζ > 0$ , $C > 1$ , and the two stated model-class hypotheses. Let $B_{0} := \frac{4 c _{0}^{2} K _{0}^{2}}{1 - c _{0}^{2}} .$ Since $0 < α < 1$ , we have $0 < c_{0} < 1$ , hence $1 - c_{0}^{2} > 0$ , and therefore $B_{0} > 0$ .

2. Fix $T \in N$ and $Q \geq 1$ . The refined posterior estimate in Lemma 23, combined with the choice of $K_{0}$ in step 1, gives the uniform terminal-depth posterior bound $q_{t}^{v} (o) \leq K_{0} α^{Q}$ for every auxiliary horizon $N$ , every $v \in {- 1, + 1}$ , every $t < N$ , and every observed prefix $o \in F_{t - 1}^{obs}$ . In particular, for the present horizon $T$ , $q_{t}^{v} (o) \leq K_{0} α^{Q}, v \in {- 1, + 1}, t < T, o \in F_{t - 1}^{obs} .$

3. Applying Lemma 25 with the posterior bound from step 2 yields the finite observed-word inequality $KL (P_{Q, + 1}^{fin, T} P_{Q, - 1}^{fin, T}) \leq \frac{4 c _{0}^{2} K _{0}^{2}}{1 - c _{0}^{2}} T ε_{C}^{2} α^{2 Q} L^{- Q} .$ The observed-path law is obtained from the finite observed-word law by the deterministic reward decoding map in Definition 18. The data-processing inequality for this embedding gives $KL (P_{Q, + 1}^{obs, T} P_{Q, - 1}^{obs, T}) \leq \frac{4 c _{0}^{2} K _{0}^{2}}{1 - c _{0}^{2}} T ε_{C}^{2} α^{2 Q} L^{- Q} .$ Equivalently, by the definition of $B_{0}$ , $KL (P_{Q, + 1}^{obs, T} P_{Q, - 1}^{obs, T}) \leq B_{0} T ε_{C}^{2} α^{2 Q} L^{- Q} .$

4. Since $C > 1$ , the reset amplitude satisfies $0 \leq ε_{C} = min {(C - 1) /2, 1/2} \leq 1.$ Also $L = exp (ζ) > 0$ , so $L^{- Q} \geq 0$ , and $α^{2 Q} \geq 0$ . Hence $B_{0} T ε_{C}^{2} α^{2 Q} L^{- Q} \leq B_{0} T α^{2 Q} L^{- Q} .$ Monotonicity of $x \mapsto ofReal (x)$ on this real inequality gives $KL (P_{Q, + 1}^{obs, T} P_{Q, - 1}^{obs, T}) \leq B_{0} T α^{2 Q} L^{- Q} .$

5. Finally, Lemma 30 supplies, for each $v \in {- 1, + 1}$ , each $t < T$ , and each observed prefix $o \in F_{t - 1}^{obs}$ , the exact identity $E_{v} [Y_{t} ∣ F_{t - 1}^{obs} = o] = v c_{0} ε_{C} a_{t}^{(Q)} (o) q_{t}^{v} (o) .$ Together with the posterior bound in step 2 and the sharper KL estimate from step 3, this gives all three asserted conclusions: $q_{t}^{v} (o) \leq K_{0} α^{Q}$ and $KL (P_{Q, + 1}^{obs, T} P_{Q, - 1}^{obs, T}) \leq \frac{4 c _{0}^{2} K _{0}^{2}}{1 - c _{0}^{2}} T ε_{C}^{2} α^{2 Q} L^{- Q} .$

∎

The identity in Lemma 31 gives the exact observed conditional mean in terms of a rare visible action run and a terminal-depth posterior. The KL bound then combines the survival factor $α^{2 Q}$ with the policy-overlap factor $L^{- Q}$ , matching the fixed-overlap exponent used in the minimax comparison. This is the point at which latent stationarity and observed histories enter the lower-bound calculation together.

The upper-bound argument also uses the stationary mean of each PHIW score. With the observed law already fixed in Definition 26, the next statement bridges the observable score to the target transition operator that controls the truncation term.

Lemma 32 [lem:phiw-score-stationary-mean] (Stationary PHIW score mean).

Let $T, n_{X}, n_{H}, k \in N$ , let $t_{0}, ζ, C \in R$ , and let $M$ be a raw POMDP experiment. Suppose:

(Model class.) $M \in M_{T} (t_{0}, ζ, C)$ , as in Definition 10.

(Mature time.) $t \in {0, \dots, T - 1}$ and $k \leq t$ .

Then the depth- $k$ score $Z_{t} (k)$ of Definition 16, evaluated under the observed trajectory law of $M$ , satisfies $\int Z_{t} (k) d P_{M}^{obs} = s \sum d_{b} (s) (P_{e}^{k} g_{e}) (s) .$ Here $d_{b}$ is the stationary law selected for the behavior-policy transition matrix $P_{b} (s, s^{'}) = a \in {0, 1} \sum b (a ∣ x_{s}) K (s, a) {(r, s^{''}) : s^{''} = s^{'}},$ and $P_{e}^{k} g_{e}$ is the $k$ -fold iterate of the target-policy transition operator applied to the target-policy reward regression $g_{e} (s) = a \in {0, 1} \sum e (a ∣ x_{s}) \int r K (s, a) (d r, d s^{'}) .$

⊢ Lean

Proof of Lemma 32.

Write $S_{j}$ for the joint state at zero-based epoch $j$ , on the same time scale as $Z_{t} (k)$ . For a policy $p \in {b, e}$ and a function $φ$ on joint states, set $(P_{p} φ) (s) := s^{'} \sum P_{p} (s, s^{'}) φ (s^{'}), P_{p} (s, s^{'}) = a \in {0, 1} \sum p (a ∣ x_{s}) K (s, a) {(y, s^{''}) : s^{''} = s^{'}} .$ Also write the observable ratio, with its zero-cell convention, as $ρ_{j} = ρ (X_{j}, A_{j}) := {e (A_{j} ∣ X_{j}) / b (A_{j} ∣ X_{j}), 0, b (A_{j} ∣ X_{j}) > 0, b (A_{j} ∣ X_{j}) = 0.$ By Definition 10, the model supplies Assumptions 1, 2, 3, 4, and 5, with $L = exp (ζ)$ . Since $0 < ζ$ , $1 \leq L$ .

Let $ι := t - k .$ The hypothesis $k \leq t$ makes $ι$ a valid epoch, and $ι + k = t < T$ . We first compute the mean of the score from the state at the beginning of its ratio window: $\int Z_{t} (k) d P_{M}^{obs} = \int (P_{e}^{k} g_{e}) (S_{ι}) d P_{M} .$ Indeed, the score is a function of the observed trajectory, so its observed-law integral is the corresponding full-trajectory integral, $\int Z_{t} (k) d P_{M}^{obs} = \int Y_{t} j = ι \prod t ρ_{j} d P_{M} .$ For any bounded $φ$ and any epoch $j < T - 1$ , Assumptions 2, 1, and 5 give $E_{M} [ρ_{j} φ (S_{j + 1}) ∣ S_{j} = s] = a \in {0, 1} \sum e (a ∣ x_{s}) \int φ (s^{'}) K (s, a) (d y, d s^{'}) = (P_{e} φ) (s) .$ The same conditional calculation at the terminal reward gives $E_{M} [ρ_{t} Y_{t} ∣ S_{t} = s] = a \in {0, 1} \sum e (a ∣ x_{s}) \int y K (s, a) (d y, d s^{'}) = g_{e} (s) .$ The zero-cell convention is compatible with the displayed equalities because $b (a ∣ x_{s}) = 0$ and $e (a ∣ x_{s}) \leq L b (a ∣ x_{s})$ imply $e (a ∣ x_{s}) = 0$ . Iterating the preceding identities backward from $t$ to $ι$ , with boundedness supplied by Assumption 4 and finite state spaces from Definition 10, yields the displayed score identity.

It remains to identify the law of $S_{ι}$ . For every function $φ$ on joint states, $\int φ (S_{ι}) d P_{M} = s \sum d_{b} (s) φ (s) .$ At the initial epoch this is Assumption 3. If the identity holds at epoch $j$ , then Assumptions 2 and 1 give $\int φ (S_{j + 1}) d P_{M} = \int (P_{b} φ) (S_{j}) d P_{M} = s \sum d_{b} (s) s^{'} \sum P_{b} (s, s^{'}) φ (s^{'}) .$ The selected behavior stationary law is stationary for $P_{b}$ , so for every $s^{'}$ , $s \sum d_{b} (s) P_{b} (s, s^{'}) = d_{b} (s^{'}),$ and therefore $\int φ (S_{j + 1}) d P_{M} = s^{'} \sum d_{b} (s^{'}) φ (s^{'}) .$ Induction gives the claim at $ι$ .

Applying the last identity with $φ = P_{e}^{k} g_{e}$ and substituting into the score identity gives $\int Z_{t} (k) d P_{M}^{obs} = s \sum d_{b} (s) (P_{e}^{k} g_{e}) (s),$ which is the asserted equality.

∎

Lemma 32 states the exact stationary target of each mature score. Under stationarity, a length- $k$ observable reweighting transports the behavior stationary distribution through $k$ target-policy transitions, so the remaining discrepancy is the contraction-controlled distance between this finite-lag quantity and the stationary target value.

The bounded-reward normalization also pins down the range of the target value. This range fact is used when risks are compared over estimators whose outputs are clipped to the same unit interval.

Lemma 33 [lem:target-value-unit-interval] (Target value is bounded).

Let $T \geq 1$ , let $t_{0}, ζ, C \in R$ , and let $M = (X, H, K, b, e, L (O_{T})) \in M_{T} (t_{0}, ζ, C)$ be a model in Definition 10. Assume the model satisfies the stationary-start, sequential-ignorability, POMDP-kernel, bounded-reward, latent-stationary-overlap, and policy-overlap conditions of Assumptions 3, 2, 1, 4, 7, and 5. Then the stationary target-policy mean reward $θ (K, e)$ of Definition 12 lies in the unit interval: $θ (K, e) \in [- 1, 1] .$

⊢ Lean

Proof of Lemma 33.

Let $P_{b}$ and $P_{e}$ be the joint-state transition kernels induced by the behavior and target policies, and let $d_{b}$ and $d_{e}$ be their selected stationary laws. For a joint state $s = (x, h)$ and action $a \in A$ , define the one-step reward mean $\overset{r}{ˉ} (s, a) := \int r K (d r, d x^{'}, d h^{'} ∣ s, a) .$ We also use the local notation $g_{e} (s) := a \in A \sum e (a ∣ x) \overset{r}{ˉ} (s, a),$ which is the target-policy reward regression appearing inside $θ (K, e)$ . With this notation, Definition 12 gives $θ (K, e) = s \sum d_{e} (s) g_{e} (s) .$

Since $T \geq 1$ , consider epoch $0$ . Let $η_{0}$ denote the unique empty pre-history value at epoch $0$ , and define the finite state-history and state-action-history spaces $H_{0}^{st} := {(η_{0}, s) : s \in S}, H_{0}^{act} := H_{0}^{st} \times A .$ For $υ = (η_{0}, s) \in H_{0}^{st}$ , write $cur (υ) = s$ . Let $Γ_{0}^{st} := (η_{0}, S_{0}), Γ_{0}^{act} := (Γ_{0}^{st}, A_{0}), μ_{0} := L (Γ_{0}^{act}) .$ For every $γ = (υ, a) \in H_{0}^{act}$ and every measurable $B \subseteq R \times S$ , define the row selected by the whole state-action history view as $row_{0} (γ; B) := K (B ∣ cur (υ), a) .$ By the probability-kernel part of Assumption 1, $row_{0} (γ; \cdot)$ is a probability measure for every $γ$ . Define the total function $J_{0} (υ, a) := \int r row_{0} ((υ, a); d r, d x^{'}, d h^{'}) = \int r K (d r, d x^{'}, d h^{'} ∣ cur (υ), a),$ for all $(υ, a) \in H_{0}^{act}$ ; bounds for $J_{0}$ will be asserted only on the full-measure set obtained below. Thus, when $υ = (η_{0}, s)$ , one has $J_{0} (υ, a) = \overset{r}{ˉ} (s, a)$ .

By Assumption 4, $∣ Y_{0} ∣ \leq 1$ holds almost surely under the trajectory law, so the event $R_{0} := {(γ, (r, s^{'})) \in H_{0}^{act} \times (R \times S) : ∣ r ∣ \leq 1}$ has full mass under the law of $(Γ_{0}^{act}, (Y_{0}, S_{1}))$ . After both sides are pushed forward to the epoch- $0$ conditioning view, Assumption 1 gives the composition-product identity $L {(Γ_{0}^{act}, (Y_{0}, S_{1})) \in d γ d (r, s^{'})} = μ_{0} (d γ) row_{0} (γ; d r, d s^{'}) .$ The section argument is now the standard disintegration fact for a composition product: if a set has full mass under the displayed law, then for $μ_{0}$ -almost every $γ$ its section has full mass under the row $row_{0} (γ; \cdot)$ . Applying this to $R_{0}$ gives $row_{0} (γ; {(r, x^{'}, h^{'}) : ∣ r ∣ \leq 1}) = 1 for μ_{0} -almost every γ .$ For every such $γ = (υ, a)$ , the row is a probability measure and the coordinate function $(r, x^{'}, h^{'}) \mapsto r$ is bounded by $1$ in absolute value almost surely under that row. The usual norm bound for the integral of an almost-surely bounded real function under a probability measure therefore gives $∣ J_{0} (υ, a) ∣ = \int r row_{0} ((υ, a); d r, d x^{'}, d h^{'}) \leq \int ∣ r ∣ row_{0} ((υ, a); d r, d x^{'}, d h^{'}) \leq 1.$ Equivalently, using the definition of $row_{0}$ , $∣ J_{0} (υ, a) ∣ = \int r K (d r, d x^{'}, d h^{'} ∣ cur (υ), a) \leq 1 for μ_{0} -almost every (υ, a) .$
Fix $s = (x, h)$ and $a \in A$ with $d_{b} (s) > 0$ and $b (a ∣ x) > 0$ . Put $υ_{s} := (η_{0}, s) \in H_{0}^{st}, γ_{s, a} := (υ_{s}, a) \in H_{0}^{act} .$ The singleton mass of $γ_{s, a}$ is obtained from three separate ingredients, none of which gives it alone. The first is Assumption 2, which says only that the law $μ_{0}$ of $Γ_{0}^{act}$ factors as the law of $Γ_{0}^{st}$ followed by the behavior kernel $b (\cdot ∣ X_{0})$ , so that $μ_{0} {γ_{s, a}} = Pr {Γ_{0}^{st} = υ_{s}} b (a ∣ x) .$ The second ingredient is the epoch- $0$ history projection: the pre-history coordinate at epoch $0$ is forced to be $η_{0}$ , so the event ${Γ_{0}^{st} = υ_{s}}$ is exactly ${S_{0} = s}$ . The third is Assumption 3, which identifies the law of $S_{0}$ and gives $Pr {Γ_{0}^{st} = υ_{s}} = Pr (S_{0} = s) = d_{b} (s) .$ Combining the last two displays with the two strict-positivity hypotheses yields $μ_{0} {γ_{s, a}} = d_{b} (s) b (a ∣ x) > 0.$ If a full- $μ_{0}$ -measure set omitted this singleton, its complement would have positive $μ_{0}$ -mass. Hence the almost-sure bound from the previous step holds at $γ_{s, a}$ , and therefore $∣ \overset{r}{ˉ} (s, a) ∣ \leq 1 whenever d_{b} (s) > 0 and b (a ∣ x) > 0.$
The selected behavior and target stationary laws are probability vectors: by Definition 6 each $d_{p}$ lies in $Δ (S)$ , the simplex of nonnegative probability vectors on the joint-state alphabet, fixed by the corresponding policy-induced kernel. Unpacking this simplex membership gives $d_{b} (s) \geq 0, d_{e} (s) \geq 0 for all s, s \sum d_{e} (s) = 1.$ Now suppose $d_{e} (s) > 0$ and $e (a ∣ x) > 0$ . By Assumption 7, $d_{e} (s) \leq C d_{b} (s) .$ If $d_{b} (s)$ were not positive, its nonnegativity would force $d_{b} (s) = 0$ , and substituting that into the displayed overlap inequality would give $0 < d_{e} (s) \leq C \cdot 0 = 0$ , a contradiction. This uses the displayed overlap inequality, the nonnegativity $d_{b} (s) \geq 0$ , and the local hypothesis $d_{e} (s) > 0$ ; the radius condition $C \geq 1$ plays no further role in this step. Thus $d_{b} (s) > 0$ .

The action-coordinate argument uses the local hypothesis $e (a ∣ x) > 0$ , the nonnegativity of the behavior carrier, and the policy-overlap inequality. The inequality comes from Assumption 5, which for the policy-overlap constant $L$ gives $e (a ∣ x) \leq L b (a ∣ x);$ the nonnegativity $b (a ∣ x) \geq 0$ comes from Assumption 2, which is where $b$ is declared a probability vector. If $b (a ∣ x)$ were not positive, that nonnegativity would force $b (a ∣ x) = 0$ , whence $0 < e (a ∣ x) \leq L \cdot 0 = 0$ , a contradiction. Thus $b (a ∣ x) > 0$ . The row bound from the previous step applies, and hence $∣ \overset{r}{ˉ} (s, a) ∣ \leq 1 whenever d_{e} (s) > 0 and e (a ∣ x) > 0.$
For any $s = (x, h)$ with $d_{e} (s) > 0$ , the target reward regression satisfies $∣ g_{e} (s) ∣ = a \in A \sum e (a ∣ x) \overset{r}{ˉ} (s, a) \leq a \in A \sum ∣ e (a ∣ x) \overset{r}{ˉ} (s, a) ∣ = a \in A \sum e (a ∣ x) ∣ \overset{r}{ˉ} (s, a) ∣ \leq a \in A \sum e (a ∣ x) = 1.$ The equality in the third line uses $e (a ∣ x) \geq 0$ ; the last inequality uses the preceding step when $e (a ∣ x) > 0$ , while the corresponding summand is zero when $e (a ∣ x) = 0$ ; and the final equality is the target-policy probability-vector property in Assumption 5. Two distinct facts meet in that last equality, and they come from different places. Which set the sum runs over is a carrier fact: Definition 3 fixes the action coordinate as the Boolean two-point alphabet identified with $A = {0, 1}$ , and it imposes nothing else — in particular no nonnegativity and no normalization. That the two values at $x$ are nonnegative and add to one is the separate probability-vector restriction on $e$ , which is imposed in Assumption 5. So the normalization is the sum of exactly two action probabilities at $x$ , and no larger action space is involved.
Finally, $∣ θ (K, e) ∣ = s \sum d_{e} (s) g_{e} (s) \leq s \sum ∣ d_{e} (s) g_{e} (s) ∣ = s \sum d_{e} (s) ∣ g_{e} (s) ∣ \leq s \sum d_{e} (s) = 1.$ Here $d_{e} (s) \geq 0$ gives the third line; in the fourth line the summand is zero when $d_{e} (s) = 0$ , and otherwise the regression bound from the previous step applies. Thus $- 1 \leq θ (K, e) \leq 1,$ which is exactly membership of $θ (K, e)$ in the closed real interval $[- 1, 1]$ .

∎

Lemma 33 records the bounded target range implied by the model-class reward normalization and the target stationary distribution. This range is the common scale for the clipped PHIW estimator and for the bounded observable estimators used in the minimax comparison.

The radius-sensitive upper calculation refines the estimator bound by keeping the latent-overlap distance $q_{C}$ explicit.

Lemma 34 [lem:radius-sensitive-phiw] (Radius-sensitive PHIW risk).

For every trajectory length $T$ , observable alphabet size $n_{X}$ , latent alphabet size $n_{H}$ , and history depth $k$ , let $α = exp {- (1/ t_{0})}, L = exp {ζ}, q_{C} = \frac{C - 1}{C} .$ Suppose that

(Mixing scale.) $0 < t_{0}$ .

(Overlap scale.) $0 < ζ$ .

(Radius parameter.) $1 \leq C$ , with $q_{C}$ as in Definition 19.

(Trajectory length.) $1 \leq T$ .

(Model class.) The experiment $M$ belongs to $M_{T} (t_{0}, ζ, C)$ in the sense of Definition 10.

(History depth.) $k \leq T /2$ .

Then the clipped partial-history importance-weighted estimator $θ_{k}$ from Definition 16, evaluated under the observed trajectory law induced by $M$ , satisfies $E_{M} [{θ_{k} - θ (K, e)}^{2}] \leq 4 q_{C}^{2} α^{2 k} + \frac{2}{T} {L^{k + 1} (1 + \frac{4}{L - 1}) + \frac{4}{1 - α}},$ where $θ (K, e)$ is the stationary target-policy mean reward from Definition 12.

⊢ Lean

Proof of Lemma 34.

Write $P_{M}^{obs}$ for the observed trajectory law induced by $M$ , and let $q_{C} := \frac{C - 1}{C}$ be the overlap-distance radius from Definition 19. Define the unclipped partial-history average $θ_{k} := \frac{1}{T - k} t = k \sum T - 1 Z_{t} (k), θ_{k} = clip_{[- 1, 1]} (θ_{k}),$ with $Z_{t} (k)$ as in Definition 16. Since $1 \leq T$ and $k \leq ⌊ T /2 ⌋$ , one has $k < T$ , so $T - k \geq 1$ . The conditions defining $M \in M_{T} (t_{0}, ζ, C)$ in Definition 10 give $L = e^{ζ} > 1$ and $0 < α = e^{- 1/ t_{0}} < 1$ . The finite observation space makes each score $Z_{t} (k)$ measurable. Moreover $L = e^{ζ} \geq 1$ , and the sequential-ignorability, policy-overlap, and bounded-reward restrictions in Definition 10 imply the window envelope $∣ Z_{t} (k) ∣ \leq L^{k + 1} P_{M}^{obs} -a.s.,$ for every $t$ . Thus each score is square-integrable under $P_{M}^{obs}$ , and so is $θ_{k}$ . Moreover, Lemma 33 applies to the present horizon and model class membership, giving $θ (K, e) \in [- 1, 1] .$
For every $u \in R$ and every $ϑ \in [- 1, 1]$ , $(clip_{[- 1, 1]} (u) - ϑ)^{2} \leq (u - ϑ)^{2} .$ Applying this pointwise with $u = θ_{k}$ and $ϑ = θ (K, e)$ , and then using the usual bias–variance identity for the square-integrable random variable $θ_{k}$ , yields $E_{M} [{θ_{k} - θ (K, e)}^{2}] \leq Var_{M} (θ_{k}) + (E_{M} [θ_{k}] - θ (K, e))^{2} .$
We next bound the bias term. Let $d_{b}$ and $d_{e}$ be the stationary laws of the behavior- and target-policy transition kernels, and let $g_{e}$ be the target-policy reward regression appearing in Definition 12. Define $g_{e}^{\circ} (s) := {g_{e} (s), 0, d_{b} (s) > 0, d_{b} (s) = 0, Ψ_{k} (s) := (P_{e}^{k} g_{e}^{\circ}) (s) .$ On states with $d_{b} (s) > 0$ , the horizon condition and the model-class restrictions give $∣ g_{e} (s) ∣ \leq 1$ ; on states with $d_{b} (s) = 0$ , the definition gives $g_{e}^{\circ} (s) = 0$ . Hence $∣ g_{e}^{\circ} (s) ∣ \leq 1$ for every $s$ , and therefore $osc (g_{e}^{\circ}) := s, s^{'} sup ∣ g_{e}^{\circ} (s) - g_{e}^{\circ} (s^{'}) ∣ \leq 2.$ The kernel and policy restrictions in Definition 10 make $P_{e}$ a Markov transition kernel, and Assumption 6 supplies $∥ λ P_{e} - λ^{'} P_{e} ∥_{TV} \leq α ∥ λ - λ^{'} ∥_{TV}$ for all probability laws $λ, λ^{'}$ on $S$ . Consequently, for any bounded function $r : S \to R$ and any states $s, s^{'}$ , the dual total-variation inequality gives $(P_{e} r) (s) - (P_{e} r) (s^{'}) \leq osc (r) ∥ δ_{s} P_{e} - δ_{s^{'}} P_{e} ∥_{TV} \leq α osc (r) .$ Iterating this one-step oscillation contraction yields $osc (P_{e}^{k} r) \leq α^{k} osc (r) .$ Applying the last display to $r = g_{e}^{\circ}$ gives $osc (Ψ_{k}) = osc (P_{e}^{k} g_{e}^{\circ}) \leq 2 α^{k} .$ By Lemma 32, every score in the average defining $θ_{k}$ has mean $E_{M} [Z_{t} (k)] = s \in S \sum d_{b} (s) (P_{e}^{k} g_{e}) (s), t = k, \dots, T - 1.$ Therefore $E_{M} [θ_{k}] = s \in S \sum d_{b} (s) (P_{e}^{k} g_{e}) (s) .$ For each state $s$ with $d_{b} (s) > 0$ , the functions $g_{e}$ and $g_{e}^{\circ}$ agree on the behavior support, so Lemma 11, applied with the target policy and initial state $s$ , gives $(P_{e}^{k} g_{e}) (s) = Ψ_{k} (s) .$ For states with $d_{b} (s) = 0$ , the common prefactor $d_{b} (s)$ makes both weighted summands vanish, regardless of the two function values. Hence $s \sum d_{b} (s) (P_{e}^{k} g_{e}) (s) = s \sum d_{b} (s) Ψ_{k} (s) .$ By Assumption 7, included among the restrictions in Definition 10, $d_{e} (s) \leq C d_{b} (s)$ . Thus $d_{e} (s) > 0$ implies $d_{b} (s) > 0$ , so $θ (K, e) = s \sum d_{e} (s) g_{e} (s) = s \sum d_{e} (s) g_{e}^{\circ} (s) .$ By the meaning of the target-policy stationary law used in Definition 12, $d_{e}$ is invariant for $P_{e}$ . Thus, for every function $r_{test} : S \to R$ and every integer $m \geq 0$ , $s \sum d_{e} (s) r_{test} (s) = s \sum d_{e} (s) (P_{e}^{m} r_{test}) (s) .$ Applying this identity with $r_{test} = g_{e}^{\circ}$ and $m = k$ yields $s \sum d_{e} (s) g_{e}^{\circ} (s) = s \sum d_{e} (s) Ψ_{k} (s) .$ Combining the preceding displays, $E_{M} [θ_{k}] - θ (K, e) = s \sum {d_{b} (s) - d_{e} (s)} Ψ_{k} (s) .$ For any function with oscillation at most $2 α^{k}$ , $s \sum {d_{b} (s) - d_{e} (s)} Ψ_{k} (s) \leq 2 α^{k} ∥ d_{b} - d_{e} ∥_{TV} .$ It remains to prove the stationary-law distance subclaim $∥ d_{b} - d_{e} ∥_{TV} \leq q_{C} .$ This subclaim uses the pointwise comparison $d_{e} (s) \leq C d_{b} (s)$ from Assumption 7. Throughout this step, $∥ μ - η ∥_{TV} := \frac{1}{2} s \in S \sum ∣ μ (s) - η (s) ∣ = B \subseteq S sup ∣ μ (B) - η (B) ∣$ for probability laws on the finite joint-state space. With this convention, any two probability laws have total-variation distance at most $1$ , since $\frac{1}{2} s \sum ∣ μ (s) - η (s) ∣ \leq \frac{1}{2} s \sum {μ (s) + η (s)} = 1.$ If $C = 1$ , then $d_{e} (s) \leq d_{b} (s)$ for every $s$ , and the two probability laws have equal total mass, so $d_{b} = d_{e}$ . Hence $∥ d_{b} - d_{e} ∥_{TV} = 0 = q_{C} .$ If $C > 1$ , define $η_{C} (s) := \frac{C d _{b} ( s ) - d _{e} ( s )}{C - 1}, s \in S .$ The latent stationary overlap condition gives $C d_{b} (s) - d_{e} (s) \geq 0$ for every $s$ , and $s \sum η_{C} (s) = \frac{C \sum _{s} d _{b} ( s ) - \sum _{s} d _{e} ( s )}{C - 1} = 1,$ so $η_{C}$ is a probability law. The identity $d_{b} = \frac{1}{C} d_{e} + q_{C} η_{C}, q_{C} = \frac{C - 1}{C},$ then gives $d_{b} - d_{e} = q_{C} (η_{C} - d_{e}) .$ Therefore $∥ d_{b} - d_{e} ∥_{TV} = q_{C} ∥ η_{C} - d_{e} ∥_{TV} \leq q_{C} .$ Thus in all cases, $E_{M} [θ_{k}] - θ (K, e) \leq 2 q_{C} α^{k} .$
The variance estimate supplied by Lemma 14, with $L = e^{ζ}$ and $α = e^{- 1/ t_{0}}$ , gives $Var_{M} (θ_{k}) \leq \frac{2}{T} {L^{k + 1} (1 + \frac{4}{L - 1}) + \frac{4}{1 - α}} .$
Since $q_{C} \geq 0$ and $α \geq 0$ , $(E_{M} [θ_{k}] - θ (K, e))^{2} \leq (2 q_{C} α^{k})^{2} = 4 q_{C}^{2} α^{2 k} .$ Substituting this and the variance bound into the clipped bias–variance inequality proves $E_{M} [{θ_{k} - θ (K, e)}^{2}] \leq 4 q_{C}^{2} α^{2 k} + \frac{2}{T} {L^{k + 1} (1 + \frac{4}{L - 1}) + \frac{4}{1 - α}} .$

∎

In Lemma 34, the first term is the squared truncation contribution after $k$ observable lags, scaled by the radius $q_{C}$ . The second term is the sampling contribution from estimating with length- $k$ products of observable policy ratios. Choosing the radius-calibrated depth in Definition 20 balances these terms in the shrinking-overlap results.

The matching radius-explicit KL statement records the lower-bound calculation with the same radius normalization.

Lemma 35 [lem:radius-explicit-observed-kl] (Observed KL radius bound).

Let $t_{0} > 0$ . Define $α (t_{0}) = exp (- 1/ t_{0}), c_{0} (t_{0}) = \frac{1 - α ( t _{0} )}{4} .$ Then there exists a constant $K_{0} = K_{0} (t_{0}) > 0$ such that, for every $ζ > 0$ , $C > 1$ , $T \in N$ , and $Q \in N$ with $Q \geq 1$ , if $ε_{C} = min {\frac{C - 1}{2}, \frac{1}{2}}, L (ζ) = exp (ζ),$ and if $q_{C}$ is the radius in Definition 19, then $\frac{q _{C}}{2} \leq ε_{C} \leq q_{C}$ and the observed-path laws $P_{Q, + 1}^{obs, T}$ and $P_{Q, - 1}^{obs, T}$ of the signed-depth alternatives satisfy $KL (P_{Q, + 1}^{obs, T} P_{Q, - 1}^{obs, T}) \leq \frac{4 c _{0} ( t _{0} ) ^{2} K _{0}^{2}}{1 - c _{0} ( t _{0} ) ^{2}} T ε_{C}^{2} α (t_{0})^{2 Q} L (ζ)^{- Q} .$

⊢ Lean

Proof of Lemma 35.

Fix $t_{0} > 0$ .

1. Apply Lemma 31 with this value of $t_{0}$ . It gives a constant $K_{0} = K_{0} (t_{0}) > 0$ , and it remains only to supply the sequential-ignorability and stationary-start hypotheses for the signed-depth experiments over the range requested there. For every auxiliary horizon $T^{'} \in N$ , terminal depth $Q^{'} \geq 1$ , and sign alternative $v \in {- 1, + 1}$ , the finite construction underlying Definition 18 has the observed-state behavior kernel $v Pr (A_{t} = a ∣ X_{1 : t}, H_{1 : t}, A_{1 : t - 1}, Y_{1 : t - 1}) = b (a ∣ X_{t}),$ and the embedding into the observable-data experiment preserves this factorization, so Assumption 2 holds. The same construction starts the joint state from its behavior stationary law, $S_{1} \sim d_{b},$ so Assumption 3 holds for the same $(T^{'}, Q^{'}, v)$ , including the zero-horizon case. Hence Lemma 31 supplies, for every $ζ > 0$ , $C > 1$ , $T \in N$ , and $Q \geq 1$ , the sharpened observed-path bound $KL (P_{Q, + 1}^{obs, T} P_{Q, - 1}^{obs, T}) \leq \frac{4 c _{0} ( t _{0} ) ^{2} K _{0}^{2}}{1 - c _{0} ( t _{0} ) ^{2}} T ε_{C}^{2} α (t_{0})^{2 Q} L (ζ)^{- Q} .$ This KL estimate is one conjunct of the conclusion; the radius comparison is verified separately.

2. It remains to prove $\frac{q _{C}}{2} \leq ε_{C} \leq q_{C}, q_{C} = \frac{C - 1}{C}, ε_{C} = min {\frac{C - 1}{2}, \frac{1}{2}} .$ Since $C > 1$ , we have $C > 0$ and $C - 1 \geq 0$ . First suppose $C \leq 2$ . Then $ε_{C} = \frac{C - 1}{2} .$ For the lower bound, $q_{C} /2 \leq ε_{C}$ is $\frac{C - 1}{2 C} \leq \frac{C - 1}{2},$ which follows from $C > 0$ , $C \geq 1$ , and $C - 1 \geq 0$ . For the upper bound, $ε_{C} \leq q_{C}$ is $\frac{C - 1}{2} \leq \frac{C - 1}{C} .$ Multiplying by $2 C > 0$ , this becomes $C (C - 1) \leq 2 (C - 1),$ and this follows from $C - 1 \geq 0$ and $C \leq 2$ .

3. Now suppose $C > 2$ . Then $ε_{C} = \frac{1}{2} .$ For the lower bound, $q_{C} /2 \leq ε_{C}$ is $\frac{C - 1}{2 C} \leq \frac{1}{2},$ equivalently $C - 1 \leq C$ , which is immediate. For the upper bound, $ε_{C} \leq q_{C}$ is $\frac{1}{2} \leq \frac{C - 1}{C} .$ Multiplying by $2 C > 0$ , this becomes $C \leq 2 (C - 1)$ , equivalently $2 \leq C$ , which follows from $C > 2$ .

Combining the two cases gives the asserted radius comparison, and the KL bound is the one obtained in Step 1 with $L (ζ) = exp (ζ)$ and $α (t_{0}) = exp (- 1/ t_{0})$ .

∎

The inequalities in Lemma 35 connect the construction amplitude $ε_{C}$ to the normalized radius from Definition 19. Consequently, the lower calculation can be stated on the same radius scale as the estimator bound in Lemma 34.

The final group records the parametric two-experiment comparison used to anchor the risk at the usual $T^{- 1}$ scale across all radii in the model class. The first two statements compute the separation in target value and the observed KL budget for the singleton-state pair; the following bounded-risk and information inequalities convert those calculations into risk floors.

Lemma 36 [lem:parametric-pair-target-value] (Parametric pair target value).

Let $T \in N$ , let $v \in {0, 1}$ , and define the sign $σ (v) = {+ 1, - 1, v = 1, v = 0.$ Put $η_{T} = \frac{1}{4 T},$ with value $0$ when $T = 0$ . Suppose the scalar parameters satisfy:

(Initial scale.) $t_{0} > 0$ .

(Overlap scale.) $ζ > 0$ .

(Radius scale.) $C \geq 1$ .

Let $M_{T}^{v}$ be the singleton-state finite experiment with one observed state and one hidden state, fair behavior and target policies $b (a ∣ x_{⋆}) = e (a ∣ x_{⋆}) = 1/2$ , reward support ${- 1, + 1}$ , and reward-transition kernel that assigns, from every state-action pair, probability $\frac{1 + σ ( v ) y η _{T}}{2}$ to reward $y \in {- 1, + 1}$ and next state $(x_{⋆}, h_{⋆})$ . Then the stationary target-policy mean reward of Definition 12, evaluated in this experiment, satisfies $θ (K_{M_{T}^{v}}, e_{M_{T}^{v}}) = σ (v) η_{T} .$

⊢ Lean

Proof of Lemma 36.

Let $F = M_{T}^{v}$ , viewed first as the finite reward model before applying the embedding into the raw experiment. Write $d_{T}^{v} (s) := d_{e} (s)$ for the target-policy stationary law on the singleton joint-state space $S = {(x_{⋆}, h_{⋆})}$ associated with this finite model. The construction of $M_{T}^{v}$ , under $t_{0} > 0$ , $ζ > 0$ , and $C \geq 1$ , belongs to $M_{T} (t_{0}, ζ, C)$ ; in particular the target stationary law is a probability vector. Thus $d_{T}^{v} (s) \geq 0 for all s \in S, s \in S \sum d_{T}^{v} (s) = 1.$ The equality between the finite encoding and the raw experiment preserves this stationary law, so the same $d_{T}^{v}$ is the law used in $θ (K_{M_{T}^{v}}, e_{M_{T}^{v}})$ as defined in Definition 12.

For every joint state $s \in S$ and action $a \in {0, 1}$ , the conditional reward mean in the finite kernel is $y \in {- 1, + 1} \sum y Pr (Y = y ∣ S = s, A = a) = (- 1) \frac{1 - σ ( v ) η _{T}}{2} + (+ 1) \frac{1 + σ ( v ) η _{T}}{2} = σ (v) η_{T} .$ The displayed calculation is exactly the reward-kernel mean, with the encoding $0 \mapsto - 1$ and $1 \mapsto + 1$ .

By Definition 12 and the finite-to-raw value identity for this encoding, $θ (K_{M_{T}^{v}}, e_{M_{T}^{v}}) = s \in S \sum d_{T}^{v} (s) a \in {0, 1} \sum e_{M_{T}^{v}} (a ∣ x (s)) y \in {- 1, + 1} \sum y Pr (Y = y ∣ S = s, A = a) .$ Substituting the preceding reward-mean identity gives $θ (K_{M_{T}^{v}}, e_{M_{T}^{v}}) = s \in S \sum d_{T}^{v} (s) a \in {0, 1} \sum e_{M_{T}^{v}} (a ∣ x (s)) σ (v) η_{T} .$ The target policy in $M_{T}^{v}$ is fair, hence for every $s$ , $a \in {0, 1} \sum e_{M_{T}^{v}} (a ∣ x (s)) = 1.$ Therefore $θ (K_{M_{T}^{v}}, e_{M_{T}^{v}}) = s \in S \sum d_{T}^{v} (s) σ (v) η_{T} = σ (v) η_{T} s \in S \sum d_{T}^{v} (s) = σ (v) η_{T},$ using the total mass identity for $d_{T}^{v}$ .

∎

Lemma 37 [lem:parametric-pair-observed-kl] (Observed KL budget).

Let $T \in N$ satisfy $T \geq 1$ , and set $η_{T} = 1/ (4 T)$ . Let $M_{T}^{-}$ and $M_{T}^{+}$ be the two finite singleton-state experiments with one observed state $x_{⋆}$ , one hidden state, initial mass at the single joint state, fair behavior and target policies $b = e$ on ${0, 1}$ , and reward code $r \in {0, 1}$ decoded as $2 r - 1$ . Under $M_{T}^{σ}$ , for $σ \in {- 1, + 1}$ , the reward-transition kernel returns to the single joint state and assigns reward $y \in {- 1, + 1}$ probability $\frac{1 + σ y η _{T}}{2} .$ Writing $P_{M_{T}^{σ}}^{obs}$ for the image of the decoded trajectory law under the observed-path projection, the required KL budget holds in the direction $KL (P_{M_{T}^{-}}^{obs} P_{M_{T}^{+}}^{obs}) \leq \frac{1}{4} .$

⊢ Lean

Proof of Lemma 37.

Write $F_{-}$ and $F_{+}$ for the two finite reward-code models with Boolean indices $false$ and $true$ , so that the associated signs are $- 1$ and $+ 1$ . Let $η_{T} = \frac{1}{4 T} .$ Since $T \geq 1$ , we have $0 < T$ and $0 \leq η_{T} \leq 1/4$ .

The two finite models use the same reward decoder $r \mapsto 2 r - 1$ . Therefore applying the decoder and then projecting to the observed path is a common measurable map, and KL contracts under this map: $KL (P_{M_{T}^{-}}^{obs} P_{M_{T}^{+}}^{obs}) \leq KL ((F_{-})^{obs} (F_{+})^{obs}),$ where $(F_{σ})^{obs}$ denotes the finite observed-path law before decoding the reward code into ${- 1, + 1}$ .
For one epoch define probability laws $μ_{-}$ and $μ_{+}$ on ${x_{⋆}} \times {0, 1} \times {0, 1}$ by $μ_{σ} (x_{⋆}, a, r) = \frac{1}{2} \frac{1 + σ ( 2 r - 1 ) η _{T}}{2}, σ \in {- 1, + 1}, a, r \in {0, 1} .$ The first factor is the fair action law, and the second factor is the reward-code law. Because $0 \leq η_{T} \leq 1/4$ , every displayed cell mass is positive.
Multiplying the singleton initial mass, the fair action probabilities, and the displayed time-homogeneous reward-transition probabilities gives the product form $(F_{σ})^{obs} = t = 1 ⨂ T μ_{σ}, σ \in {- 1, + 1} .$ The singleton observed state contributes unit mass at each time, so the only nontrivial coordinates in each factor are the fair action and the reward code.
The one-epoch divergence is bounded by $KL (μ_{-} ∥ μ_{+}) \leq 4 η_{T}^{2} .$ Let $A$ be the fair law on ${0, 1}$ , let $Ber_{{0, 1}} (u)$ be the law on ${0, 1}$ assigning mass $u$ to reward code $1$ , and set $u_{-} := \frac{1 - η _{T}}{2}, u_{+} := \frac{1 + η _{T}}{2} .$ Since $0 \leq η_{T} \leq 1/4$ , both $u_{-}$ and $u_{+}$ lie in $[3/8, 5/8]$ . The law $μ_{σ}$ is obtained from $Ber_{{0, 1}} (u_{σ}) \otimes A$ by the bijective coordinate map $(r, a) \mapsto (x_{⋆}, a, r)$ . Hence invariance under this reordering and the common action coordinate give $KL (μ_{-} ∥ μ_{+}) = KL (Ber_{{0, 1}} (u_{-}) \otimes A Ber_{{0, 1}} (u_{+}) \otimes A) = KL (Ber_{{0, 1}} (u_{-}) Ber_{{0, 1}} (u_{+})) .$ The Boolean Bernoulli laws are images of the two-point real laws $(1 - u) δ_{0} + u δ_{1}$ under the measurable map that records whether the point is $1$ , so data processing gives the upper bound by the corresponding real two-point divergence. Evaluating that finite divergence yields $KL (Ber_{{0, 1}} (u_{-}) Ber_{{0, 1}} (u_{+})) \leq u_{-} lo g \frac{u _{-}}{u _{+}} + (1 - u_{-}) lo g \frac{1 - u _{-}}{1 - u _{+}} = η_{T} lo g \frac{1 + η _{T}}{1 - η _{T}} .$ For $0 \leq s \leq 1/4$ , the function $4 s - lo g ((1 + s) / (1 - s))$ has value $0$ at $s = 0$ and derivative $4 - \frac{2}{1 - s ^{2}} \geq 4 - \frac{2}{1 - ( 1/4 ) ^{2}} > 0.$ Thus $lo g ((1 + s) / (1 - s)) \leq 4 s$ on this interval, and substituting $s = η_{T}$ gives the displayed one-epoch bound.
Since every one-epoch cell under $μ_{+}$ has positive mass, $μ_{-} ≪ μ_{+}$ ; on this finite space the log-likelihood ratio is integrable. Tensorization of KL for the $T$ -fold product laws gives two finite-valued conclusions: the product divergence is finite, and after applying the usual real value map to the extended nonnegative divergence, $KL (t = 1 ⨂ T μ_{-} t = 1 ⨂ T μ_{+})_{R} \leq T KL (μ_{-} ∥ μ_{+})_{R} .$ The one-epoch bound from the previous step is finite, so the same real-value comparison gives $KL (μ_{-} ∥ μ_{+})_{R} \leq 4 η_{T}^{2} .$ The product representation of the finite observed-path law therefore yields $KL ((F_{-})^{obs} (F_{+})^{obs})_{R} \leq T \cdot 4 η_{T}^{2} .$ Because the displayed product divergence is finite, comparison with any finite numerical bound is equivalent to comparison after taking this real value.
Finally, $T \cdot 4 η_{T}^{2} = T \cdot 4 (\frac{1}{4 T})^{2} = \frac{1}{4},$ using $0 < T$ and $(T)^{2} = T$ . The finite-valued comparison from the preceding step gives $KL ((F_{-})^{obs} (F_{+})^{obs}) \leq \frac{1}{4} .$

The contraction inequality from the first step and the finite observed-path bound from the last step give $KL (P_{M_{T}^{-}}^{obs} P_{M_{T}^{+}}^{obs}) \leq \frac{1}{4},$ as claimed.

∎

Before applying the two-point inequality, the next statement records a uniform boundedness fact for clipped observable-data estimators. It uses the target-value range from Lemma 33 and the estimator range built into Definition 14.

Lemma 38 [lem:observed-risk-bounded-by-four] (Observed risk bound).

Let $T \geq 1$ and let $t_{0}, ζ, C \in R$ . Let $θ$ be an observable-data estimator of the admissible class used in Definition 14: for each observed-alphabet size and policy pair, $θ$ is a measurable real-valued function of the observed path taking values in $[- 1, 1]$ . For every $M = (X, H, K, b, e, L (O_{T})) \in M_{T} (t_{0}, ζ, C)$ as in Definition 10, with $θ$ evaluated at the observed-alphabet size and policies of $M$ , the squared risk for the target value $θ (K, e)$ of Definition 12 satisfies $E_{M} [(θ - θ (K, e))^{2}] \leq 4.$

⊢ Lean

Proof of Lemma 38.

Fix $M = (X, H, K, b, e, L (O_{T})) \in M_{T} (t_{0}, ζ, C),$ and write an observed path as $ω_{obs} = ((X_{t}, A_{t}, Y_{t}))_{t = 0}^{T - 1} \in O_{T} .$ For this path set $r_{err} (ω_{obs}) := θ (n_{X}, b, e, ω_{obs}) - θ (K, e) .$ The admissibility condition in Definition 14 gives $- 1 \leq θ (n_{X}, b, e, ω_{obs}) \leq 1,$ and Lemma 33 gives $- 1 \leq θ (K, e) \leq 1.$ Hence $- 2 \leq r_{err} (ω_{obs}) \leq 2.$ Equivalently, $0 \leq r_{err} (ω_{obs}) + 2, 0 \leq 2 - r_{err} (ω_{obs}) .$ Multiplying these two nonnegative quantities yields $0 \leq (r_{err} (ω_{obs}) + 2) (2 - r_{err} (ω_{obs})) = 4 - r_{err} (ω_{obs})^{2},$ so $(θ (n_{X}, b, e, ω_{obs}) - θ (K, e))^{2} \leq 4 for every ω_{obs} \in O_{T} .$
The function $ω_{obs} ⟼ (θ (n_{X}, b, e, ω_{obs}) - θ (K, e))^{2}$ is measurable because $θ (n_{X}, b, e, \cdot)$ is measurable by admissibility and the target value is constant in the observed path. The observed-path law $L_{M}^{obs}$ is a probability law, and the pointwise bound from the previous step gives integrability together with $\int (θ (n_{X}, b, e, ω_{obs}) - θ (K, e))^{2} d L_{M}^{obs} (ω_{obs}) \leq \int 4 d L_{M}^{obs} (ω_{obs}) = 4.$ By the squared-risk definition in Definition 14, the left-hand side is exactly $E_{M} [(θ - θ (K, e))^{2}] .$ Therefore $E_{M} [(θ - θ (K, e))^{2}] \leq 4.$

∎

Lemma 38 places all admissible estimator losses on a common bounded scale. This supplies the integrability and range control used when the two-point comparison is embedded into the minimax risk.

Lemma 39 [lem:two-point-risk-floor] (Two-point risk floor).

Let $T \geq 1$ , $t_{0}, ζ, C \in R$ , and let $M_{0}, M_{1} \in M_{T} (t_{0}, ζ, C)$ as in Definition 10. After identifying their common observed alphabet, write $P_{j}$ for the observed-data law of $M_{j}$ and $θ_{j}$ for the target value of $M_{j}$ from Definition 12, $j \in {0, 1}$ . Let $s$ be a nonnegative separation radius and let $χ$ be a nonnegative KL upper bound. Assume:

(Common observable structure.) The two models have the same observed-alphabet cardinality, the same behavior policy, and the same target policy.

(Information bound.) $P_{0} ≪ P_{1}$ , $KL (P_{0} ∥ P_{1}) \leq χ, KL (P_{0} ∥ P_{1}) < \infty.$
(Target separation.) $2 s \leq ∣ θ_{0} - θ_{1} ∣.$

Then every raw observable-data estimator $θ$ , viewed as a real-valued measurable function of the observed-alphabet size, the policy pair, and the observed path, with $(θ - θ_{j})^{2}$ integrable under $P_{j}$ at $M_{j}$ for $j = 0, 1$ , satisfies $\frac{s ^{2} exp ( - χ )}{4} \leq max {E_{M_{0}} [(θ - θ_{0})^{2}], E_{M_{1}} [(θ - θ_{1})^{2}]} .$ Consequently, the minimax squared-error risk of Definition 14 satisfies $\frac{s ^{2} exp ( - χ )}{4} \leq R_{T} (t_{0}, ζ, C) .$

⊢ Lean

Proof of Lemma 39.

Write the common observed alphabet size as $n_{X}$ , and use the common observable structure to identify the behavior and target policy carriers of the two models. Put $P_{j} := P_{j}, θ_{j} := θ (M_{j}), j \in {0, 1},$ where $θ (M_{j})$ is the target value from Definition 12. The information bound and $χ \geq 0$ give $KL (P_{0} ∥ P_{1})_{R} \leq χ,$ where $KL (P_{0} ∥ P_{1})_{R}$ denotes the real value of the finite extended-real KL divergence. Hence $exp (- χ) \leq exp {- KL (P_{0} ∥ P_{1})_{R}} .$ By the Bretagnolle–Huber affinity inequality, using $P_{0} ≪ P_{1}$ and finite KL divergence, $\frac{1}{2} exp {- KL (P_{0} ∥ P_{1})_{R}} \leq 1 - TV (P_{0}, P_{1}) .$ Combining the two displays yields $\frac{exp ( - χ )}{4} \leq \frac{1 - TV ( P _{0} , P _{1} )}{2} .$

Fix a raw observable-data estimator $θ$ satisfying the measurability and integrability hypotheses. With the common observable inputs fixed, define $f (o) := θ (n_{X}, b, e, o), o \in O_{T},$ and define the two error events $E_{j} := {o \in O_{T} : s \leq ∣ f (o) - θ_{j} ∣}, j \in {0, 1} .$ The separation condition $2 s \leq ∣ θ_{0} - θ_{1} ∣$ and the two-point testing bound for real-valued parameters give $\frac{1 - TV ( P _{0} , P _{1} )}{2} \leq max {P_{0} (E_{0}), P_{1} (E_{1})} .$

Since $s \geq 0$ , for each $j \in {0, 1}$ , $E_{j} = {o \in O_{T} : s^{2} \leq (f (o) - θ_{j})^{2}} .$ The squared-error integrability assumptions therefore allow the elementary tail-to-mean comparison $s^{2} P_{j} (E_{j}) \leq \int (f (o) - θ_{j})^{2} d P_{j} (o), j \in {0, 1} .$

Combining Equations 1 and 2 gives $\frac{exp ( - χ )}{4} \leq max {P_{0} (E_{0}), P_{1} (E_{1})} .$ If $P_{0} (E_{0}) \leq P_{1} (E_{1})$ , then the last maximum is $P_{1} (E_{1})$ , and Equation 3 gives $\frac{s ^{2} exp ( - χ )}{4} \leq s^{2} P_{1} (E_{1}) \leq \int (f (o) - θ_{1})^{2} d P_{1} (o) \leq j \in {0, 1} max \int (f (o) - θ_{j})^{2} d P_{j} (o) .$ If $P_{1} (E_{1}) \leq P_{0} (E_{0})$ , the same argument gives $\frac{s ^{2} exp ( - χ )}{4} \leq s^{2} P_{0} (E_{0}) \leq \int (f (o) - θ_{0})^{2} d P_{0} (o) \leq j \in {0, 1} max \int (f (o) - θ_{j})^{2} d P_{j} (o) .$ These two cases prove $\frac{s ^{2} exp ( - χ )}{4} \leq max {E_{M_{0}} [(θ - θ_{0})^{2}], E_{M_{1}} [(θ - θ_{1})^{2}]} .$

It remains to pass from the fixed two models to the minimax risk in Definition 14. The observable-estimator class is nonempty, for example it contains the constant-zero estimator. We first record the bounded-risk input needed for the minimax reduction. Fix an admissible observable estimator $θ$ . For every model $M \in M_{T} (t_{0}, ζ, C)$ , with observed alphabet size $n_{X}^{M}$ , behavior carrier $b_{M}$ , target carrier $e_{M}$ , observed law $P_{M}$ , and target value $θ (M)$ , Lemma 38 gives $E_{M} [(θ (n_{X}^{M}, b_{M}, e_{M}, O_{T}) - θ (M))^{2}] = \int (θ (n_{X}^{M}, b_{M}, e_{M}, o) - θ (M))^{2} d P_{M} (o) \leq 4.$ Consequently, ${E_{M} [(θ (n_{X}^{M}, b_{M}, e_{M}, O_{T}) - θ (M))^{2}] : M \in M_{T} (t_{0}, ζ, C)} \subset (- \infty, 4],$ which is the boundedness hypothesis needed in the two-point minimax reduction. For such an estimator, the two integrability requirements used above hold at $M_{0}$ and $M_{1}$ . These are two separate statements, one for each model, and each is taken at that model’s own observable inputs before any identification of carriers: at $M_{j}$ the estimator is measurable by admissibility and its range condition gives $∣ θ (n_{X}, b_{M_{j}}, e_{M_{j}}, o) ∣ \leq 1, o \in O_{T},$ so the fixed-model squared error is dominated by the constant $(1 + ∣ θ (M_{j}) ∣)^{2}$ , since $∣ θ (n_{X}, b_{M_{j}}, e_{M_{j}}, o) - θ (M_{j}) ∣ \leq 1 + ∣ θ (M_{j}) ∣, j \in {0, 1} .$ Only after both integrability statements are in hand do the common-policy hypotheses $b_{M_{0}} = b_{M_{1}}$ and $e_{M_{0}} = e_{M_{1}}$ rewrite the two carriers into the single pair $(b, e)$ used for $f$ above, so that the two integrals are integrals of one function against the two observed laws. Applying Equation 4 to each admissible observable estimator and then the standard two-point reduction to the minimax value yields $\frac{s ^{2} exp ( - χ )}{4} \leq R_{T} (t_{0}, ζ, C),$ which is the claimed minimax lower bound.

∎

Lemma 36 sets the value separation for the singleton-state Rademacher pair, and Lemma 37 gives the corresponding observed KL budget. The bounded-risk statement in Lemma 38 fixes the loss scale for admissible clipped estimators, and the generic reduction in Lemma 39 then translates a common-policy, low-divergence pair with target separation into a lower bound for every observable-data estimator.

Lemma 40 [lem:uniform-parametric-floor] (Uniform parametric floor).

Fix a positive mixing scale $t_{0} > 0$ and a positive log-overlap scale $ζ > 0$ . For every horizon $T \in N$ with $T \geq 1$ and every radius $C \geq 1$ , let $M_{T}^{-}$ and $M_{T}^{+}$ be the two raw POMDP experiments generated by the singleton-state, singleton-observation Rademacher construction with rewards in ${- 1, + 1}$ , fair behavior and target policies $b = e$ , and the two opposite signs of the construction’s parametric reward tilt. Then:

(Class membership.) Both $M_{T}^{-}$ and $M_{T}^{+}$ belong to the latent-overlap model class $M_{T} (t_{0}, ζ, C)$ of Definition 10.
(Common policies.) Each experiment has equal behavior and target policies, and the two experiments share these policies: $b_{M_{T}^{-}} = e_{M_{T}^{-}}, b_{M_{T}^{+}} = e_{M_{T}^{+}}, b_{M_{T}^{-}} = b_{M_{T}^{+}}, e_{M_{T}^{-}} = e_{M_{T}^{+}} .$
(Uniform minimax floor.) The minimax squared-error risk $R_{T} (t_{0}, ζ, C)$ of Definition 14 satisfies $\frac{exp ( - 1/4 )}{64 T} \leq R_{T} (t_{0}, ζ, C) .$

(Pointwise two-experiment floor.) For every observable-data estimator $θ$ , viewed as a measurable function of the revealed alphabet and the policies, under the usual squared-error integrability conditions for the observed trajectory laws of $M_{T}^{-}$ and $M_{T}^{+}$ , $\frac{exp ( - 1/4 )}{64 T} \leq max {E_{M_{T}^{-}}^{obs} [(θ (1, b_{M_{T}^{-}}, e_{M_{T}^{-}}; O_{T}) - θ (K_{M_{T}^{-}}, e_{M_{T}^{-}}))^{2}], E_{M_{T}^{+}}^{obs} [(θ (1, b_{M_{T}^{+}}, e_{M_{T}^{+}}; O_{T}) - θ (K_{M_{T}^{+}}, e_{M_{T}^{+}}))^{2}]},$ where $θ (K, e)$ is the stationary target-policy mean reward of Definition 12.

⊢ Lean

Proof of Lemma 40.

Fix $T \geq 1$ and put $η_{T} := \frac{1}{4 T} .$ Since $T \geq 1$ , $0 \leq η_{T} \leq 1/4$ . For each sign $σ \in {- 1, + 1}$ , define a singleton-state experiment with $X = {x_{⋆}}$ , $H = {h_{⋆}}$ , fair policies $b_{σ} (a ∣ x_{⋆}) = e_{σ} (a ∣ x_{⋆}) = \frac{1}{2}, a \in {0, 1},$ and reward-transition kernel $σ Pr {Y_{t} = y, X_{t + 1} = x_{⋆}, H_{t + 1} = h_{⋆} ∣ X_{t} = x_{⋆}, H_{t} = h_{⋆}, A_{t} = a} = \frac{1 + σ y η _{T}}{2}, y \in {- 1, + 1} .$ Let $M_{T}^{-}$ be the experiment with $σ = - 1$ , and $M_{T}^{+}$ the experiment with $σ = + 1$ . The weights above are nonnegative and sum to one. The hypotheses give the positive parameter-scale requirements $0 < t_{0}$ and $0 < ζ$ appearing in Definition 10. The Markov-kernel, sequential-ignorability, stationary-start, and bounded-reward requirements in Assumptions 1, 2, 3, and 4 hold by this generated singleton construction. The policy-overlap condition in Assumption 5 holds because $b_{σ} = e_{σ}$ and $exp (ζ) \geq 1$ . For either policy $p \in {b_{σ}, e_{σ}}$ , $ν P_{p} = ν^{'} P_{p} = δ_{(x_{⋆}, h_{⋆})}$ for all probability laws $ν, ν^{'}$ on the joint state space. Put $α_{t_{0}} := exp (- 1/ t_{0}) .$ Since $t_{0} > 0$ , $α_{t_{0}} > 0$ , and therefore $α_{t_{0}} ∥ ν - ν^{'} ∥_{TV} \geq 0$ . Thus $∥ ν P_{p} - ν^{'} P_{p} ∥_{TV} = 0 \leq α_{t_{0}} ∥ ν - ν^{'} ∥_{TV},$ which is the uniform-contraction requirement in Assumption 6. Finally, the latent stationary overlap condition in Assumption 7 is immediate: $d_{e_{σ}} = d_{b_{σ}} = δ_{(x_{⋆}, h_{⋆})}$ , and hence $d_{e_{σ}} (s) \leq C d_{b_{σ}} (s)$ for every state whenever $C \geq 1$ . Thus both alternatives belong to $M_{T} (t_{0}, ζ, C)$ as defined in Definition 10. The same displayed definition of the policies also gives $b_{M_{T}^{-}} = e_{M_{T}^{-}}, b_{M_{T}^{+}} = e_{M_{T}^{+}}, b_{M_{T}^{-}} = b_{M_{T}^{+}}, e_{M_{T}^{-}} = e_{M_{T}^{+}} .$
The singleton alternatives just constructed are the two alternatives covered by Lemma 36. With the sign convention of Step 1, that result gives $θ_{-} := θ (K_{M_{T}^{-}}, e_{M_{T}^{-}}) = - η_{T}, θ_{+} := θ (K_{M_{T}^{+}}, e_{M_{T}^{+}}) = η_{T} .$ Consequently $2 η_{T} = ∣ θ_{-} - θ_{+} ∣.$ For the divergence calculation, let $r \in {0, 1}$ be the reward code and set $y (r) := 2 r - 1, y (0) = - 1, y (1) = + 1.$ Define the one-epoch coded observed law $Λ_{T}^{σ} \in Δ ({x_{⋆}} \times {0, 1} \times {0, 1})$ by $Λ_{T}^{σ} {(x_{⋆}, a, r)} := \frac{1}{2} \frac{1 + σ y ( r ) η _{T}}{2}, a, r \in {0, 1} .$ Since $0 \leq η_{T} \leq 1/4$ , every displayed cell has positive mass. Thus $Λ_{T}^{-} ≪ Λ_{T}^{+}$ , and the log likelihood ratio is integrable. If $Ξ_{T}^{-}$ and $Ξ_{T}^{+}$ denote the reward-code marginals of $Λ_{T}^{-}$ and $Λ_{T}^{+}$ , then $Ξ_{T}^{-} {1} = \frac{1 - η _{T}}{2}, Ξ_{T}^{+} {1} = \frac{1 + η _{T}}{2} .$ The fair action factor is common to the two signs, so the product rule for KL with a common second factor gives $KL (Λ_{T}^{-} ∥ Λ_{T}^{+}) = KL (Ξ_{T}^{-} ∥ Ξ_{T}^{+}) .$ Writing out the two Bernoulli masses gives $KL (Ξ_{T}^{-} ∥ Ξ_{T}^{+}) = \frac{1 - η _{T}}{2} lo g \frac{1 - η _{T}}{1 + η _{T}} + \frac{1 + η _{T}}{2} lo g \frac{1 + η _{T}}{1 - η _{T}} = η_{T} lo g \frac{1 + η _{T}}{1 - η _{T}} .$ For $0 \leq u \leq 1/4$ , $lo g \frac{1 + u}{1 - u} = lo g (1 + u) - lo g (1 - u) \leq u + \frac{u}{1 - u} \leq 4 u,$ using $lo g (1 + u) \leq u$ and $- lo g (1 - u) \leq u / (1 - u)$ . Applying this with $u = η_{T}$ yields the one-epoch bound $KL (Λ_{T}^{-} ∥ Λ_{T}^{+}) \leq 4 η_{T}^{2} .$ Let $Π_{σ, T}^{code} := i = 1 ⨂ T Λ_{T}^{σ} on ({x_{⋆}} \times {0, 1} \times {0, 1})^{T} .$ The generated finite trajectory construction has this product as its coded observed-word law. By finite-product KL tensorization, with the absolute continuity and integrability supplied by the positive one-epoch masses, $KL (Π_{-, T}^{code} ∥ Π_{+, T}^{code}) \leq T KL (Λ_{T}^{-} ∥ Λ_{T}^{+}) \leq 4 T η_{T}^{2} .$ Because $T \geq 1$ , $T > 0$ , and $η_{T} = 1/ (4 T)$ , $4 T η_{T}^{2} = \frac{1}{4} .$ Finally, let $d_{T}$ be the decoding map that replaces each reward code $r$ by $y (r)$ and keeps the observed state and action coordinates. The observed laws in the statement are the pushforwards $P_{M_{T}^{-}}^{obs} = (d_{T})_{#} Π_{-, T}^{code}, P_{M_{T}^{+}}^{obs} = (d_{T})_{#} Π_{+, T}^{code} .$ KL divergence is monotone under measurable pushforward, so the preceding coded product bound gives $KL (P_{M_{T}^{-}}^{obs} P_{M_{T}^{+}}^{obs}) \leq KL (Π_{-, T}^{code} ∥ Π_{+, T}^{code}) \leq \frac{1}{4} .$ This is the observed KL budget in Lemma 37. The positive one-epoch masses give $Π_{-, T}^{code} ≪ Π_{+, T}^{code}$ , hence also $P_{M_{T}^{-}}^{obs} ≪ P_{M_{T}^{+}}^{obs}$ , and the displayed upper bound gives finite KL divergence.
Let $θ$ be any observable-data estimator satisfying the measurability and two integrability assumptions in the statement, and write $φ (o) := θ (1, b_{M_{T}^{-}}, e_{M_{T}^{-}}; o) = θ (1, b_{M_{T}^{+}}, e_{M_{T}^{+}}; o),$ where the equality uses the common policies from Step 1. The two-point risk inequality applied here is Lemma 39, used in its raw-estimator form for the estimator $θ$ . Its structural hypotheses are available here: the observed alphabets both have one state and the behavior and target policies are common by Step 1. Step 2 supplies $2 η_{T} \leq ∣ θ_{-} - θ_{+} ∣, P_{M_{T}^{-}}^{obs} ≪ P_{M_{T}^{+}}^{obs}, KL (P_{M_{T}^{-}}^{obs} P_{M_{T}^{+}}^{obs}) \leq \frac{1}{4},$ together with finite KL divergence. The present step supplies the required measurability of $θ (n_{X}, b, e; \cdot)$ for every input $(n_{X}, b, e)$ , and the two squared-error integrability assumptions for the two displayed risks.

Apply Lemma 39 directly to the raw estimator $θ$ , the two observed laws $P_{M_{T}^{-}}^{obs}, P_{M_{T}^{+}}^{obs},$ the target values $θ_{-} = θ (K_{M_{T}^{-}}, e_{M_{T}^{-}}), θ_{+} = θ (K_{M_{T}^{+}}, e_{M_{T}^{+}}),$ and the numerical choices $s_{⋆} = η_{T}, χ_{KL} = \frac{1}{4} .$ The separation condition is supplied by Step 2, and the absolute-continuity and finite-KL hypotheses are supplied by the final display of Step 2. Hence $\frac{η _{T}^{2} exp ( - 1/4 )}{4} \leq max {E_{M_{T}^{-}}^{obs} [{φ (O_{T}) - θ_{-}}^{2}], E_{M_{T}^{+}}^{obs} [{φ (O_{T}) - θ_{+}}^{2}]} .$ Finally, $\frac{η _{T}^{2} exp ( - 1/4 )}{4} = \frac{exp ( - 1/4 )}{64 T},$ which is exactly the pointwise two-experiment floor in the statement.
It remains to record the minimax lower bound. For the supplied radius $C$ , the two singleton alternatives with the same kernels and policies satisfy the hypotheses of the minimax conclusion in Lemma 39: Step 1 gives their common observable cardinality, common behavior policy, common target policy, and membership in $M_{T} (t_{0}, ζ, C)$ ; Step 2 gives $2 η_{T} \leq ∣ θ_{-} - θ_{+} ∣, P_{M_{T}^{-}}^{obs} ≪ P_{M_{T}^{+}}^{obs}, KL (P_{M_{T}^{-}}^{obs} P_{M_{T}^{+}}^{obs}) \leq \frac{1}{4},$ with finite KL divergence. Applying Lemma 39 with $s_{⋆} = η_{T}, χ_{KL} = \frac{1}{4}$ yields $\frac{η _{T}^{2} exp ( - 1/4 )}{4} \leq R_{T} (t_{0}, ζ, C),$ where $R_{T} (t_{0}, ζ, C)$ is the observable minimax risk of Definition 14. Since $\frac{η _{T}^{2} exp ( - 1/4 )}{4} = \frac{exp ( - 1/4 )}{64 T},$ the desired minimax floor follows. Together with the membership and policy identities from Step 1 and the pointwise inequality from Step 3, this proves all four asserted conclusions.

∎

Lemma 40 complements the signed-depth lower construction by giving a radius-uniform benchmark at the parametric scale. The experiments share behavior and target policies, so the displayed floor reflects value uncertainty in the observed rewards themselves and applies throughout the latent-overlap model class.

Verification note

This note records the machine-verification scope of the paper.

What is checked. Every displayed assumption, definition, lemma, proposition, and theorem of the main text and of this appendix is either mapped to a declaration of an accompanying Lean 4 development, listed under Statement map below, or flagged under Presentation-level definitions as notation introduced for the prose alone. Each displayed proof renders the corresponding Lean proof. The Lean development is the verified artifact; the displayed statements and proofs are human-readable transcriptions of it, and the constants, exponents, and inequalities they display are the ones that appear in the machine-checked declarations.

Toolchain and artifact version. The development compiles under the Lean 4 toolchain leanprover/lean4:v4.33.0 against mathlib4 at revision db584cd6d46c92f209a44c0f1c829460d327499d, in the module tree CausalSmith/Stat/STAT_PomdpLatentOverlapMinimax_Research, from the artifact source tree at commit f2e7e979167a2f3f9242e93c680c7ba1667b6dcc, which is the commit recorded in the verification contract accompanying this paper.

What the receipts certify. The receipts behind this note are declaration-level: for each declaration listed below, the recorded check is that it compiles under the stated toolchain and that its source carries no sorry and no admit. That is the exact scope of the trust claim made here. Readers who additionally want the kernel-level axiom footprint of a particular declaration can obtain it directly by running #print axioms on that declaration in the accompanying development.

External inputs. No result in this paper is stated conditionally on a published theorem: the cited literature supplies framing, motivation, and comparison only, and no bibliographic item enters any displayed statement as a hypothesis.

Declarations from outside the module tree. Lemma 5 is carried by Causalean.Mathlib.Probability.CertifiedFiniteMarkovExpectation.stationaryRewardInterval_sound_of_chunked, reached through Helpers/InsulinGrid.lean. It belongs to the same verified library and compiles under the same toolchain.

Statement map. Each anchored object and the declaration it is mapped to, with the file of the module tree that contains it:

Definition 1: policyFactor in Basic.lean.
Definition 2: FullTrajectory in Basic.lean.
Assumption 1: PomdpKernelLaw in Basic.lean.
Definition 3: Policy in Basic.lean.
Assumption 2: SequentialIgnorability in Basic.lean.
Definition 4: policyKernel in Basic.lean.
Definition 5: JointState in Basic.lean.
Assumption 3: StationaryStart in Basic.lean.
Assumption 4: BoundedReward in Basic.lean.
Assumption 5: PolicyOverlap in Basic.lean.
Definition 7: mixingAlpha in Basic.lean.
Assumption 6: UniformContraction in Basic.lean.
Assumption 7: LatentStationaryOverlap in Basic.lean.
Definition 8: ObsView in Basic.lean.
Definition 10: LatentOverlapClass in Basic.lean.
Definition 11: rewardRegression in Basic.lean.
Definition 12: targetValue in Basic.lean.
Definition 14: minimaxRisk in Basic.lean.
Definition 15: ratio in Basic.lean.
Definition 16: phiwEstimator in Basic.lean.
Definition 17: historyDepth in Basic.lean.
Definition 18: signedDepthFamily in Helpers/SignedDepth.lean.
Definition 19: overlapRadius in Basic.lean.
Definition 20: radiusAdaptiveDepth in Basic.lean.
Theorem 1: uniform_overlap_frontier in TUniformOverlapFrontier.lean.
Theorem 2: fixed_c_minimax in TFixedCMinimax.lean.
Theorem 3: unit_overlap_boundary in TUnitOverlapBoundary.lean.
Theorem 4: shrinking_overlap_frontier in TShrinkingOverlapFrontier.lean.
Definition 24: insulinGrid in Helpers/InsulinGridCore/Model.lean.
Lemma 1: insulinPolicy_contraction in Helpers/InsulinGridCore/Model.lean.
Lemma 2: insulinStationaryLaw_isStationary in Helpers/InsulinGridCore/Model.lean.
Lemma 3: insulinTargetRewardRegression in Helpers/InsulinGridCore/Semantic.lean.
Lemma 4: insulinImmediateRewardRegression in Helpers/InsulinGridCore/Semantic.lean.
Lemma 6: insulinBiasInterval_bounds in Helpers/InsulinChunkAssembly.lean.
Lemma 7: insulinGrid_bias_certificate in Helpers/InsulinGrid.lean.
Theorem 5: finite_insulin_demonstration in TFiniteInsulinDemonstration.lean.
Definition 25: filterQuantities in Helpers/ObservedFilter.lean.
Lemma 8: integral_sq_phiwScore_le in Helpers/PhiwWindowMoments.lean.
Lemma 9: abs_covariance_phiwScore_le_overlapDecay in Helpers/PhiwWindowMoments.lean.
Lemma 10: integral_phiwScore_mul_eq_futureFunction in Helpers/PhiwFutureEndpoint.lean.
Lemma 11: markovOperatorIter_congr_on_behaviorSupport in Helpers/StationaryRewardSupport.lean.
Lemma 12: abs_covariance_phiwScore_le_disjoint_sharp in Helpers/PhiwFutureEndpoint.lean.
Lemma 13: sum_abs_covariance_future_le in Helpers/PhiwVariance.lean.
Lemma 14: variance_phiwRaw_le in Helpers/PhiwVariance.lean.
Lemma 15: phiw_upper in Helpers/PhiwUpper.lean.
Lemma 16: signedDepth_uniformContraction_actionOne in Helpers/SignedDepthStationarity.lean.
Lemma 17: signedDepth_stationaryLaw_apply in Helpers/SignedDepthStationarity.lean.
Lemma 18: signedDepth_membership in Helpers/SignedDepthMembership.lean.
Lemma 19: pow_one_sub_le_signedDepthPosteriorLowerWindow in Helpers/TerminalPosteriorBound.lean.
Lemma 20: signedDepthRawPosterior_le_product_of_le in Helpers/TerminalPosteriorBound.lean.
Lemma 21: signedDepthRawPosterior_le_product_of_lt in Helpers/TerminalPosteriorBound.lean.
Lemma 22: signedDepthRawPosterior_le_ratio_pow in Helpers/TerminalPosteriorBound.lean.
Lemma 23: signedDepthRawPosterior_le_refined in Helpers/TerminalPosteriorBound.lean.
Lemma 24: rareAction_secondMoment_eq in Helpers/SignedDepthObservedChain.lean.
Lemma 25: signedDepth_finiteObs_klDiv_le in Helpers/ObservedKL.lean.
Lemma 26: exists_uniform_signedDepth_bootstrap_factor in Helpers/TerminalPosteriorBound.lean.
Lemma 27: signedDepth_terminalMass_invariant in Helpers/SignedDepthFilterInvariant.lean.
Lemma 28: signedDepth_prefix_nextRewardNumerator in Helpers/SignedDepthFilterRecursion.lean.
Lemma 29: terminalPrefixMass_eq_sum_signedDepthFilterMass in Helpers/ObservedFilter.lean.
Lemma 30: conditionalRewardMean_signedDepth_all in Helpers/ObservedFilter.lean.
Definition 28: signedDepthObsLaw in Helpers/ObservedKL.lean.
Lemma 31: observed_path_kl in Helpers/ObservedKL.lean.
Lemma 32: integral_phiwScore_eq_init_targetIter in Helpers/BiasVariance.lean.
Lemma 33: targetValue_mem_unit in Helpers/Kernels.lean.
Lemma 34: radius_sensitive_phiw in Helpers/BiasVariance.lean.
Lemma 35: radius_explicit_observed_kl in Helpers/ObservedKL.lean.
Lemma 36: parametricPair_targetValue in Helpers/TwoPoint.lean.
Lemma 37: parametricPair_observed_klDiv_le in Helpers/TwoPoint.lean.
Lemma 38: observedRisk_le_four in Helpers/Kernels.lean.
Lemma 39: two_point_floor_explicit in Helpers/TwoPoint.lean.
Lemma 40: uniform_parametric_floor in Helpers/TwoPoint.lean.

Presentation-level definitions. The following definitions correspond to no single Lean declaration. They name objects the prose needs in order to display the argument, and which the Lean development carries inline rather than through a named definition; no claim is attached to them beyond the notation they fix: Definition 6, Definition 9, Definition 13, Definition 21, Definition 22, Definition 23, Definition 26, Definition 27.

Proofs of the main results

Proof of Theorem 1.

Write $α = exp (- 1/ t_{0}), L = exp (ζ), β = \frac{2}{2 + t _{0} ζ} .$ Then $0 < α < 1$ , $1 < L$ , and $0 < β < 1$ .

Step 1. Adaptive-depth calibration. Set $D = 2 lo g (1/ α) + lo g L = \frac{2}{t _{0}} + ζ, q = q_{C}, ξ = T q^{2} .$ There are $c_{rad} > 0$ and $T_{rad} \geq 1$ , depending only on $t_{0}, ζ$ , such that, for all $T \geq T_{rad}$ and all $C \geq 1$ , $q_{C}^{2} α^{2 κ_{T}^{rad} (C)} + \frac{L ^{κ_{T}^{rad} (C)}}{T} \leq c_{rad} {T^{- 1} + T^{- β} q_{C}^{2 (1 - β)}} .$ Indeed, since $lo g T = o (T)$ , enlarge $T_{rad}$ so that $∣ lo g T ∣ \leq \frac{D}{2} T (T \geq T_{rad}) .$ If $ξ \leq 1$ , then $κ_{T}^{rad} (C) = 0$ by Definition 20, and $q_{C}^{2} \leq T^{- 1}$ , hence $q_{C}^{2} + \frac{1}{T} \leq 2 T^{- 1} \leq 2 {T^{- 1} + T^{- β} q_{C}^{2 (1 - β)}} .$ If $ξ > 1$ , then $q_{C} > 0$ , $0 < ξ \leq T$ , and the same logarithmic bound gives $⌊ \frac{lo g ξ}{D} ⌋ \leq \frac{T}{2} .$ Thus $κ_{T}^{rad} (C) = ⌊ lo g ξ / D ⌋$ . The floor bounds $\frac{lo g ξ}{D} - 1 < κ_{T}^{rad} (C) \leq \frac{lo g ξ}{D}$ give $α^{2 κ_{T}^{rad} (C)} \leq e^{2/ t_{0}} ξ^{- β}, L^{κ_{T}^{rad} (C)} \leq ξ^{1 - β} .$ Consequently, $q_{C}^{2} α^{2 κ_{T}^{rad} (C)} + \frac{L ^{κ_{T}^{rad} (C)}}{T} \leq {e^{2/ t_{0}} + 1} T^{- β} q_{C}^{2 (1 - β)} .$ Taking $c_{rad} = max {2, e^{2/ t_{0}} + 1}$ proves the calibration.

Step 2. Upper bound. Let $A = max {4, 2 L (1 + \frac{4}{L - 1}), \frac{8}{1 - α}}, c_{att} = A (c_{rad} + 1) .$ For $T \geq T_{rad}$ , put $k = κ_{T}^{rad} (C)$ . By Definition 20, $k \leq T /2$ , and Lemma 34 gives, uniformly over $M \in M_{T} (t_{0}, ζ, C)$ , $E_{M} [(θ_{k} - θ (K, e))^{2}] \leq 4 q_{C}^{2} α^{2 k} + \frac{2}{T} {L^{k + 1} (1 + \frac{4}{L - 1}) + \frac{4}{1 - α}} .$ By the definition of $A$ , $4 q_{C}^{2} α^{2 k} + \frac{2}{T} {L^{k + 1} (1 + \frac{4}{L - 1}) + \frac{4}{1 - α}} \leq A {q_{C}^{2} α^{2 k} + \frac{L ^{k}}{T} + \frac{1}{T}} .$ The calibration in Step 1 and the inequality $T^{- 1} \leq T^{- 1} + T^{- β} q_{C}^{2 (1 - β)}$ therefore imply $M \in M_{T} (t_{0}, ζ, C) sup E_{M} [(θ_{κ_{T}^{rad} (C)} - θ (K, e))^{2}] \leq c_{att} r_{T} (t_{0}, ζ, C) .$ Since $R_{T} (t_{0}, ζ, C)$ is the infimum over observable-data estimators in Definition 14, this also yields $R_{T} (t_{0}, ζ, C) \leq c_{att} r_{T} (t_{0}, ζ, C) .$

Step 3. Constants for the lower comparison. From Lemma 35, choose $K_{0} = K_{0} (t_{0}) > 0$ . With $c_{0} = (1 - α) /4$ , define $B = \frac{4 c _{0}^{2} K _{0}^{2}}{1 - c _{0}^{2}} > 0, x_{⋆} = max {1, \frac{1}{4 B}} .$ Then $1 < 8 B x_{⋆}$ . Put $c_{par} = \frac{e ^{- 1/4}}{64}, c_{hid} = {\frac{c _{0} ( 1 - α ) α}{2}}^{2} (8 B)^{- β} \frac{e ^{- 1/8}}{4},$ $s_{⋆} = 1 + x_{⋆}^{1 - β}, c_{lo} = min {\frac{c _{par}}{2 s _{⋆}}, \frac{c _{hid}}{2}} .$ All these constants are positive and depend only on $t_{0}, ζ$ .

Step 4. Lower bound in the small-radius branch. By Lemma 40, $\frac{c _{par}}{T} \leq R_{T} (t_{0}, ζ, C) (T \geq 1, C \geq 1) .$ If $ξ = T q_{C}^{2} < x_{⋆}$ , then $T^{- β} q_{C}^{2 (1 - β)} = T^{- 1} ξ^{1 - β} \leq T^{- 1} x_{⋆}^{1 - β},$ with both sides equal to $0$ when $q_{C} = 0$ . Hence $r_{T} (t_{0}, ζ, C) \leq \frac{s _{⋆}}{T},$ and therefore $c_{lo} r_{T} (t_{0}, ζ, C) \leq \frac{c _{par}}{T} \leq R_{T} (t_{0}, ζ, C) .$

Step 5. Lower bound in the large-radius branch. Assume $ξ \geq x_{⋆}$ . Then $C > 1$ . Define $Q = ⌈ \frac{lo g ( 8 B ξ )}{D} ⌉ .$ Because $8 B ξ \geq 8 B x_{⋆} > 1$ , we have $Q \geq 1$ , and $D Q \geq lo g (8 B ξ), Q < \frac{lo g ( 8 B ξ )}{D} + 1.$ Let $M_{+}$ and $M_{-}$ be the two signed-depth alternatives of depth $Q$ from Definition 18, and write $θ_{+}$ and $θ_{-}$ for their target values. By Lemma 18, both alternatives belong to $M_{T} (t_{0}, ζ, C)$ . With $w_{Q}$ denoting the terminal depth mass in that construction, the same result gives $θ_{+} = c_{0} ε_{C} w_{Q}, θ_{-} = - c_{0} ε_{C} w_{Q} .$ Define the separation radius $σ_{Q} := c_{0} ε_{C} w_{Q} .$ The target-value identities give the separation condition needed for the two-point bound, $2 σ_{Q} \leq ∣ θ_{+} - θ_{-} ∣.$ Also, by Lemma 35, $\frac{q _{C}}{2} \leq ε_{C} \leq q_{C}$ and, orienting the two observed laws as $P_{Q, +}^{obs, T}$ against $P_{Q, -}^{obs, T}$ , $KL (P_{Q, +}^{obs, T} ∥ P_{Q, -}^{obs, T}) \leq B T ε_{C}^{2} α^{2 Q} L^{- Q} .$ Since $α^{2 Q} L^{- Q} = e^{- D Q}$ , the choice of $Q$ gives $B T ε_{C}^{2} α^{2 Q} L^{- Q} \leq B T q_{C}^{2} (8 B T q_{C}^{2})^{- 1} = \frac{1}{8} .$ The two alternatives have the same observed alphabet and the same revealed policies by their construction; the displayed KL bound, in the same orientation, supplies the information and finiteness hypotheses with $χ = 1/8$ . Applying Lemma 39 therefore gives $R_{T} (t_{0}, ζ, C) \geq σ_{Q}^{2} \frac{e ^{- 1/8}}{4} = (c_{0} ε_{C} w_{Q})^{2} \frac{e ^{- 1/8}}{4} .$ The depth mass in Lemma 18 satisfies $w_{Q} = \frac{( 1 - α ) α ^{Q}}{1 - α ^{Q + 1}} \geq (1 - α) α^{Q},$ and the ceiling upper bound implies $α^{Q} \geq α (8 B ξ)^{- 1/ (t_{0} D)} .$ Thus $(c_{0} ε_{C} w_{Q})^{2} \frac{e ^{- 1/8}}{4} \geq c_{hid} q_{C}^{2} ξ^{- β} = c_{hid} T^{- β} q_{C}^{2 (1 - β)} .$ Combining this hidden lower bound with the parametric floor and using $c_{lo} \leq c_{par} /2$ and $c_{lo} \leq c_{hid} /2$ , we get $c_{lo} r_{T} (t_{0}, ζ, C) \leq \frac{1}{2} R_{T} (t_{0}, ζ, C) + \frac{1}{2} R_{T} (t_{0}, ζ, C) = R_{T} (t_{0}, ζ, C) .$

Step 6. Assemble the frontier. Set $c_{hi} = max {c_{att}, c_{lo}}, T_{⋆} = T_{rad} .$ Here $c_{att} = A (c_{rad} + 1)$ uses the same calibration constant fixed in Step 1. Thus $0 < c_{lo} \leq c_{hi}$ and $c_{att} \leq c_{hi}$ . Since $r_{T} (t_{0}, ζ, C) \geq 0$ , the two upper bounds from Step 2 with constant $c_{att}$ also hold with constant $c_{hi}$ . Together, Steps 2, 4, and 5 prove, for all $C \geq 1$ and $T \geq T_{⋆}$ , $c_{lo} r_{T} (t_{0}, ζ, C) \leq R_{T} (t_{0}, ζ, C) \leq c_{hi} r_{T} (t_{0}, ζ, C),$ together with $M \in M_{T} (t_{0}, ζ, C) sup E_{M} [(θ_{κ_{T}^{rad} (C)} - θ (K, e))^{2}] \leq c_{hi} r_{T} (t_{0}, ζ, C) .$

Step 7. Immediate estimator in the $T q_{C}^{2} = O (1)$ regime. Let $C_{T} \geq 1$ and suppose there is $D_{imm} \geq 0$ such that, eventually, $T q_{C_{T}}^{2} \leq D_{imm} .$ Define $V = L (1 + \frac{4}{L - 1}) + \frac{4}{1 - α}, c_{imm} = 4 D_{imm} + 2 V > 0.$ For all sufficiently large $T$ , apply Lemma 34 with $k = 0$ . Since $q_{C_{T}}^{2} \leq D_{imm} / T$ , $M \in M_{T} (t_{0}, ζ, C_{T}) sup E_{M} [(θ_{0} - θ (K, e))^{2}] \leq 4 q_{C_{T}}^{2} + \frac{2}{T} V \leq \frac{4 D _{imm} + 2 V}{T} = \frac{c _{imm}}{T} .$ This is the asserted eventual parametric bound for the immediate estimator.

∎

Proof of Theorem 2.

Set $α := exp (- 1/ t_{0}), L := exp (ζ), D := \frac{2}{t _{0}} + ζ .$ By $t_{0} > 0$ and $ζ > 0$ , the constants satisfy $0 < α < 1, L > 1, D > 0,$ and, using the definitions in Definitions 7 and 1, $lo g α = - \frac{1}{t _{0}}, lo g L = ζ .$ The exponent in the theorem can be written as $β = \frac{2}{2 + t _{0} ζ} = \frac{2/ t _{0}}{D} .$ Apply Lemma 15 with $C \geq 1$ . It gives constants $c_{up, 0} > 0$ and $T_{up}$ such that, for all $T \geq T_{up}$ , $M \in M_{T} (t_{0}, ζ, C) sup E_{M} [{θ_{k_{T}} - θ (M)}^{2}] \leq c_{up, 0} T^{- β} .$ For the signed-depth family, the behavior policy in Definition 18 is a memoryless function of the single observed state, independent of the latent coordinate and the past. Consequently, for every horizon $T$ , terminal depth $Q \geq 1$ , and sign $v \in {- 1, + 1}$ , the action kernel satisfies Assumption 2. The same alternatives start from the behavior stationary law for every horizon, including the zero-horizon case. Indeed, for $r := L^{- 1},$ the signed-depth constant-policy stationary law at action-one probability $r$ has masses $w_{j} \frac{1 + u ε _{C} r ^{j}}{2}, w_{j} = \frac{( 1 - α ) α ^{j}}{1 - α ^{Q + 1}},$ and the behavior policy in Definition 18 chooses action one with probability $L^{- 1} = r$ . Hence its stationary law is precisely the initial hidden-state law $w_{j} \frac{1 + u ε _{C} L ^{- j}}{2} .$ The chronological path law is generated from this initial distribution, so the time-zero joint-state marginal is this stationary law; when $T = 0$ , the same identity is the whole marginal check, and for $T \geq 1$ it agrees with the behavior-stationary masses displayed in Lemma 18. Thus Assumption 3 holds for every horizon. These are the two structural hypotheses needed for Lemma 31. Therefore that lemma gives a constant $B_{0} > 0$ , depending on $t_{0}, ζ, C$ , such that for all $T$ and $Q \geq 1$ , $KL (P_{Q, + 1}^{obs, T} P_{Q, - 1}^{obs, T}) \leq B_{0} T α^{2 Q} L^{- Q} .$ Let $ε_{C} := min {\frac{C - 1}{2}, \frac{1}{2}}, c_{0} := \frac{1 - α}{4},$ and define $c_{⋆} := {c_{0} ε_{C} (1 - α) α}^{2} (8 B_{0})^{- β} \frac{exp ( - 1/8 )}{4} .$ Since $C > 1$ , every factor in this product is positive, so $c_{⋆} > 0$ . Finally set $c^{⋆} := max {c_{up, 0}, c_{⋆}} .$ Since $c_{up, 0} > 0$ and $c_{⋆} > 0$ , the maximum is positive; moreover the second argument is bounded by the maximum. Thus $0 < c^{⋆}, 0 < c_{⋆} \leq c^{⋆} .$
Define $T_{pos} := ⌈ \frac{1}{8 B _{0}} ⌉ + 1, T_{⋆} := max {T_{up}, T_{pos}, 1} .$ Fix an integer $T \geq T_{⋆}$ . Then $T \geq 1$ , $T \geq T_{up}$ , and $1 < 8 B_{0} T .$ Indeed, $\frac{1}{8 B _{0}} \leq ⌈ \frac{1}{8 B _{0}} ⌉ < T,$ and multiplication by $8 B_{0} > 0$ gives the displayed inequality. Put $τ_{T} := \frac{lo g ( 8 B _{0} T )}{D}, Q := ⌈ τ_{T} ⌉ .$ Since $lo g (8 B_{0} T) > 0$ and $D > 0$ , one has $Q \geq 1$ . Moreover, $τ_{T} \leq Q < τ_{T} + 1$ and therefore $D Q \geq lo g (8 B_{0} T) .$
The preceding depth choice makes the two signed-depth alternatives statistically close. First, $α^{2 Q} L^{- Q} = exp {Q (2 lo g α - lo g L)} = exp (- D Q),$ because $D = 2/ t_{0} + ζ$ , $lo g α = - 1/ t_{0}$ , and $lo g L = ζ$ . Hence $exp (- D Q) \leq exp {- lo g (8 B_{0} T)} = (8 B_{0} T)^{- 1} .$ Combining this inequality with the KL bound from Lemma 31 gives $KL (P_{Q, + 1}^{obs, T} P_{Q, - 1}^{obs, T}) \leq \frac{1}{8} .$
Let $M_{Q, + 1}^{T}$ and $M_{Q, - 1}^{T}$ be the two signed-depth experiments of Definition 18. They have one observed state and $2 (Q + 1)$ latent states. Their revealed behavior and target policies are common, because the policy part of the construction is independent of the sign $v$ . By Lemma 18, both experiments belong to $M_{T} (t_{0}, ζ, C)$ , and their target values are $θ (M_{Q, + 1}^{T}) = c_{0} ε_{C} w_{Q}, θ (M_{Q, - 1}^{T}) = - c_{0} ε_{C} w_{Q},$ where $w_{Q} := \frac{( 1 - α ) α ^{Q}}{1 - α ^{Q + 1}} .$ Set $s_{T} := c_{0} ε_{C} w_{Q} .$ Then $s_{T} \geq 0$ , and the two target values satisfy the separation bound $2 s_{T} \leq θ (M_{Q, + 1}^{T}) - θ (M_{Q, - 1}^{T}) .$ The two embedded observed laws have the same finite real-reward support, $Ω_{T}^{\circ} := {((0, a_{t}, y_{t}))_{t = 1}^{T} : a_{t} \in {0, 1}, y_{t} \in {- 1, + 1}} .$ They assign total mass one to $Ω_{T}^{\circ}$ , because the construction has one observed state, Boolean actions, and rewards in ${- 1, + 1}$ . We now check that every atom in this common support has positive mass under both signs. Fix $o = ((0, a_{t}, y_{t}))_{t = 1}^{T} \in Ω_{T}^{\circ}$ , and choose a fixed latent sign $u_{0} \in {- 1, + 1}$ . Consider the full hidden lift that keeps the latent state at depth zero and sign $u_{0}$ : $S_{t} = (0, (0, u_{0})), A_{t} = a_{t}, Y_{t} = y_{t}, t = 1, l d o t s, T .$ Since $0 < α < 1$ and $Q \geq 1$ , $1 - α^{Q + 1} > 0, w_{0} = \frac{1 - α}{1 - α ^{Q + 1}} > 0.$ Also $0 < ε_{C} \leq 1/2$ , so the initial mass of the chosen hidden state is positive: $w_{0} \frac{1 + u _{0} ε _{C}}{2} > 0.$ For each action in the observed word, $b (1 ∣ 0) = \frac{1}{L} > 0, b (0 ∣ 0) = 1 - \frac{1}{L} > 0,$ because $L > 1$ . Along the displayed hidden lift, the current depth is $0 < Q$ . Hence the reward factor is $\frac{1 + y _{t} v c _{0} u _{0} 1 { 0 = Q }}{2} = \frac{1}{2} > 0,$ and the hidden transition that resets to the same depth-zero sign has weight $(1 - α) \frac{1 + u _{0} ε _{C}}{2} > 0.$ Thus this single compatible hidden lift has strictly positive finite-path probability for each sign $v \in {- 1, + 1}$ , and summing over all lifts gives $P_{Q, v}^{obs, T} ({o}) > 0, o \in Ω_{T}^{\circ}, v \in {- 1, + 1} .$ The reward decoding used in the embedding is the same for the two signs, so the embedded laws preserve this common atomic support. Therefore any measurable set with zero mass under one embedded observed law contains no atom of $Ω_{T}^{\circ}$ , and consequently has zero mass under the other embedded observed law. Hence $P_{Q, + 1}^{obs, T} ≪ P_{Q, - 1}^{obs, T}, P_{Q, - 1}^{obs, T} ≪ P_{Q, + 1}^{obs, T},$ and the finite KL displayed above is an ordinary KL bound between mutually absolutely continuous observed laws.
Apply Lemma 39 to the present pair with $s = s_{T}$ , taking the $+ 1$ alternative as the lemma’s first model and the $- 1$ alternative as its second: $M_{0} = M_{Q, + 1}^{T}$ and $M_{1} = M_{Q, - 1}^{T}$ . With this assignment the lemma’s divergence hypothesis $KL (P_{0} ∥ P_{1}) \leq χ$ reads $KL (P_{Q, + 1}^{obs, T} P_{Q, - 1}^{obs, T}) \leq \frac{1}{8},$ which is exactly the direction bounded above, and its absolute-continuity hypothesis reads $P_{Q, + 1}^{obs, T} ≪ P_{Q, - 1}^{obs, T}$ , which the preceding step supplies. The common observable structure was checked in the preceding step, the KL bound gives the information hypothesis with $χ = 1/8$ , and the displayed target-value calculation gives $2 s_{T} \leq θ (M_{Q, + 1}^{T}) - θ (M_{Q, - 1}^{T}) .$ Therefore every measurable observable estimator $θ$ , under the stated squared-error integrability conditions for both alternatives, satisfies $s_{T}^{2} \frac{exp ( - 1/8 )}{4} \leq max {E_{M_{Q, + 1}^{T}} [{θ - θ (M_{Q, + 1}^{T})}^{2}], E_{M_{Q, - 1}^{T}} [{θ - θ (M_{Q, - 1}^{T})}^{2}]} .$ Commuting the two entries of the maximum, this is exactly $s_{T}^{2} \frac{exp ( - 1/8 )}{4} \leq v \in {- 1, + 1} max E_{M_{Q, v}^{T}} [{θ - θ (M_{Q, v}^{T})}^{2}] .$ The two signed-depth alternatives belong to $M_{T} (t_{0}, ζ, C)$ and share the revealed inputs $(n_{X}, b, e)$ . For the witness clause of the theorem the two squared-error integrability conditions are hypotheses, imposed on the estimator and discharged by whoever supplies it; nothing above derives them. They are derived only when $θ$ is further restricted to the admissible carrier of Definition 14, whose members are measurable and uniformly $[- 1, 1]$ -valued by definition: for such an estimator the squared error at either alternative is a measurable function bounded by $(1 + ∣ θ_{\pm} ∣)^{2} \leq 4$ on a probability space, hence integrable, so the preceding pointwise lower bound applies to every admissible estimator. Taking the supremum over the model class, then the infimum over these admissible estimators, yields $s_{T}^{2} \frac{exp ( - 1/8 )}{4} \leq R_{T} (t_{0}, ζ, C) .$
It remains to compare $s_{T}$ with $T^{- β /2}$ . Since $0 < α < 1$ and $Q \geq 1$ , also $Q + 1 \geq 1$ . Thus $0 < α^{Q + 1} < 1,$ so the denominator in the depth mass is strictly positive and satisfies $0 < 1 - α^{Q + 1} \leq 1.$ The numerator $(1 - α) α^{Q}$ is nonnegative, and division by a positive number at most one gives $w_{Q} = \frac{( 1 - α ) α ^{Q}}{1 - α ^{Q + 1}} \geq (1 - α) α^{Q} .$ The upper ceiling inequality $Q < τ_{T} + 1$ and the identity $lo g α = - 1/ t_{0}$ give $α^{Q} = exp (- \frac{Q}{t _{0}}) \geq exp (- \frac{τ _{T} + 1}{t _{0}}) = α (8 B_{0} T)^{- (1/ t_{0}) / D} .$ Therefore $s_{T} \geq c_{0} ε_{C} (1 - α) α (8 B_{0} T)^{- (1/ t_{0}) / D} .$ Squaring and using $β = (2/ t_{0}) / D$ yields $s_{T}^{2} \geq {c_{0} ε_{C} (1 - α) α}^{2} (8 B_{0} T)^{- β} = {c_{0} ε_{C} (1 - α) α}^{2} (8 B_{0})^{- β} T^{- β} .$ Multiplying by $exp (- 1/8) /4$ gives $c_{⋆} T^{- β} \leq s_{T}^{2} \frac{exp ( - 1/8 )}{4} .$ Combining this with the two-point floor proves $c_{⋆} T^{- β} \leq R_{T} (t_{0}, ζ, C),$ and, for every measurable observable-data estimator satisfying the two integrability conditions, $c_{⋆} T^{- β} \leq v \in {- 1, + 1} max E_{M_{Q, v}^{T}} [{θ - θ (M_{Q, v}^{T})}^{2}] .$
The upper bound follows from the estimator bound in the first step. Since $T \geq T_{up}$ , $M \in M_{T} (t_{0}, ζ, C) sup E_{M} [{θ_{k_{T}} - θ (M)}^{2}] \leq c_{up, 0} T^{- β} \leq c^{⋆} T^{- β} .$ By Definition 14, the minimax risk is bounded above by the worst-case risk of any admissible observable estimator, in particular by that of $θ_{k_{T}}$ . Thus $R_{T} (t_{0}, ζ, C) \leq c^{⋆} T^{- β} .$ Together with the lower bound from the preceding step, this proves all asserted inequalities for every $T \geq T_{⋆}$ , and the chosen $Q = ⌈ lo g (8 B_{0} T) / D ⌉$ supplies the stated signed-depth witnesses with one observed state and $2 (Q + 1)$ latent states.

∎

Proof of Theorem 3.

Write $α := exp (- 1/ t_{0}), L := exp (ζ) .$ Since $t_{0} > 0$ and $ζ > 0$ , one has $0 < α < 1$ and $1 < L$ .

1. Define $c_{imm} := 2 {L (1 + \frac{4}{L - 1}) + \frac{4}{1 - α}} .$ The preceding inequalities imply $c_{imm} > 0$ . For any $T \geq 1$ , Lemma 34 applied at depth $k = 0$ and overlap constant $C = 1$ gives, uniformly over $M \in M_{T} (t_{0}, ζ, 1)$ , $E_{M} [(θ_{0} - θ (K, e))^{2}] \leq \frac{2}{T} {L (1 + \frac{4}{L - 1}) + \frac{4}{1 - α}} = \frac{c _{imm}}{T} .$ Taking the supremum over the model class yields $M \in M_{T} (t_{0}, ζ, 1) sup E_{M} [(θ_{0} - θ (K, e))^{2}] \leq \frac{c _{imm}}{T} .$

2. Let $M \in M_{T} (t_{0}, ζ, 1)$ . Write $d_{b}$ and $d_{e}$ for the behavior and target stationary laws on the finite joint state space $S$ , and define $∥ d_{b} - d_{e} ∥_{TV} := \frac{1}{2} s \in S \sum ∣ d_{b} (s) - d_{e} (s) ∣.$ The stationary-law total-variation bound at overlap radius $q_{C}$ gives $∥ d_{b} - d_{e} ∥_{TV} \leq q_{1},$ and at unit overlap Definition 19 gives $q_{1} = \frac{1 - 1}{1} = 0.$ Thus $∥ d_{b} - d_{e} ∥_{TV} \leq 0.$ Since the displayed total-variation quantity is a half-sum of absolute values, it is nonnegative, so $∥ d_{b} - d_{e} ∥_{TV} = 0.$ Unfolding the definition of total variation yields $s \in S \sum ∣ d_{b} (s) - d_{e} (s) ∣ = 0.$ For each joint state $s$ , the nonnegative summand is bounded by the whole finite sum, $0 \leq ∣ d_{b} (s) - d_{e} (s) ∣ \leq u \in S \sum ∣ d_{b} (u) - d_{e} (u) ∣ = 0,$ and hence $∣ d_{b} (s) - d_{e} (s) ∣ = 0.$ Therefore $d_{b} (s) = d_{e} (s)$ for every $s \in S$ , and consequently $d_{e} = d_{b}$ .

3. Set $c_{lo} := \frac{exp ( - 1/4 )}{64}, c_{hi} := max {c_{imm}, c_{lo}}, T_{⋆} := 1.$ Then $c_{lo} > 0$ , $c_{lo} \leq c_{hi}$ , and therefore $c_{hi} > 0$ ; also $T_{⋆} \geq 1$ . For every $T \geq T_{⋆}$ , Lemma 40 at $C = 1$ gives $\frac{exp ( - 1/4 )}{64 T} \leq R_{T} (t_{0}, ζ, 1),$ which is $\frac{c _{lo}}{T} \leq R_{T} (t_{0}, ζ, 1) .$ For the upper bound, the minimax risk is bounded above by the worst-case risk of any admissible estimator with nonnegative squared risk; applying this to $θ_{0}$ and using Step 1 gives $R_{T} (t_{0}, ζ, 1) \leq M \in M_{T} (t_{0}, ζ, 1) sup E_{M} [(θ_{0} - θ (K, e))^{2}] \leq \frac{c _{imm}}{T} \leq \frac{c _{hi}}{T} .$

4. Finally, $t_{0} ζ > 0, hence 2 + t_{0} ζ > 2.$ Therefore $\frac{2}{2 + t _{0} ζ} < 1,$ which is the asserted bound on the hidden-state exponent.

∎

Proof of Theorem 4.

Put $α = exp (- 1/ t_{0}), L = exp (ζ), β = \frac{2}{2 + t _{0} ζ}, γ = 2 (1 - β) .$ Since $t_{0} > 0$ and $ζ > 0$ , one has $0 < α < 1$ , $1 < L$ , $0 < β < 1$ , and hence $γ > 0$ .

1. A radius-depth calibration. Let $Λ := 2 lo g (1/ α) + lo g L = \frac{2}{t _{0}} + ζ .$ Then $Λ > 0$ , $β = \frac{2/ t _{0}}{Λ}, 1 - β = \frac{ζ}{Λ} .$ There are $c_{R} > 0$ and $T_{R} \geq 1$ , depending only on $t_{0}, ζ$ , such that for every $T \geq T_{R}$ and every $C \geq 1$ , $q_{C}^{2} α^{2 κ_{T}^{rad} (C)} + \frac{L ^{κ_{T}^{rad} (C)}}{T} \leq c_{R} {T^{- 1} + T^{- β} q_{C}^{2 (1 - β)}} .$ Indeed, choose $T_{R}$ so that $∣ lo g T ∣ \leq (Λ/2) T$ for all $T \geq T_{R}$ . Fix such a $T$ , put $q = q_{C}$ , and set $x = T q^{2}$ . The relation $C \geq 1$ and Definition 19 give $0 \leq q \leq 1$ , hence $0 \leq x \leq T$ . If $x \leq 1$ , then $κ_{T}^{rad} (C) = 0$ by Definition 20, and $q^{2} \leq T^{- 1}$ , so $q^{2} + \frac{1}{T} \leq 2 T^{- 1} \leq 2 {T^{- 1} + T^{- β} q^{2 (1 - β)}} .$ If $x > 1$ , then $q > 0$ . With $k = ⌊ \frac{lo g x}{Λ} ⌋,$ the choice of $T_{R}$ gives $k \leq T /2$ , so the cap in Definition 20 is inactive and $κ_{T}^{rad} (C) = k$ . The floor bounds $\frac{lo g x}{Λ} - 1 < k \leq \frac{lo g x}{Λ}$ imply $α^{2 k} = exp (- \frac{2 k}{t _{0}}) \leq exp (2/ t_{0}) x^{- β}, \frac{L ^{k}}{T} \leq \frac{x ^{1 - β}}{T} .$ Since $q^{2} x^{- β} = T^{- β} q^{2 (1 - β)}, \frac{x ^{1 - β}}{T} = T^{- β} q^{2 (1 - β)},$ the $x > 1$ case is bounded by $(exp (2/ t_{0}) + 1) T^{- β} q^{2 (1 - β)} .$ Taking $c_{R} = max {2, exp (2/ t_{0}) + 1}$ proves Equation 5.

2. Constants. Apply Theorem 1 to obtain constants $c_{lo}, c_{hi} > 0$ , with $c_{lo} \leq c_{hi}$ , and a threshold $T_{U} \geq 1$ , all depending only on $t_{0}, ζ$ , such that its two-sided risk comparison and radius-calibrated estimator bound hold for all $C \geq 1$ and $T \geq T_{U}$ . Define $η_{rad} := 2^{- γ},$ and $A_{bd} := max {4, 2 L (1 + \frac{4}{L - 1}), \frac{8}{1 - α}} .$ Finally set $c_{loc} := c_{lo} η_{rad}, c_{att} := A_{bd} (c_{R} + 1), C_{loc} := max {c_{hi}, c_{att}} .$ Then $0 < c_{loc} \leq C_{loc}$ , because $0 < η_{rad} \leq 1$ , $c_{lo} \leq c_{hi}$ , and $A_{bd} > 0$ .

3. Comparing the two rate surfaces. Fix a deterministic sequence $(δ_{T})_{T \geq 1}$ with $δ_{T} \geq 0$ and $δ_{T} \to 0$ . Choose $T_{Δ}$ such that $0 \leq δ_{T} < 1 for all T \geq T_{Δ} .$ Let $T_{⋆} = max {T_{U}, T_{R}, T_{Δ}}$ . For $T \geq T_{⋆}$ , put $C_{T} = 1 + δ_{T}$ . Then $C_{T} \geq 1$ , and $q_{C_{T}} = \frac{C _{T} - 1}{C _{T}} = \frac{δ _{T}}{1 + δ _{T}} .$ Since $0 \leq δ_{T} < 1$ , $0 \leq q_{C_{T}} \leq δ_{T}, \frac{δ _{T}}{2} \leq q_{C_{T}} .$ Raising these inequalities to the positive power $γ$ gives $η_{rad} δ_{T}^{γ} \leq q_{C_{T}}^{γ} \leq δ_{T}^{γ} .$ Thus the theorem’s local rate, evaluated along the fixed sequence $(δ_{T})$ , and the fixed-radius rate at $C_{T}$ satisfy $η_{rad} {T^{- 1} + T^{- β} δ_{T}^{γ}} \leq r_{T} (t_{0}, ζ, C_{T}) \leq T^{- 1} + T^{- β} δ_{T}^{γ} .$ Equivalently, since $r_{T} (δ)$ in the statement denotes this fixed-sequence quantity, $η_{rad} r_{T} (δ) \leq r_{T} (t_{0}, ζ, C_{T}) \leq r_{T} (δ) .$ Combining Equation 6 with the two-sided comparison from Theorem 1 gives $c_{loc} r_{T} (δ) \leq R_{T} (t_{0}, ζ, 1 + δ_{T}) \leq C_{loc} r_{T} (δ) .$

4. Attainment by the displayed shrinking depth. For $T \geq T_{⋆}$ , define $C_{T}^{†} := \frac{1}{1 - δ _{T}} .$ Then $C_{T}^{†} \geq 1$ , and $q_{C_{T}^{†}} = \frac{C _{T}^{†} - 1}{C _{T}^{†}} = δ_{T} .$ Therefore Definition 20 gives $κ_{T}^{rad} (C_{T}^{†}) = κ_{T} .$ where $κ_{T}$ is the depth displayed in the theorem statement. In particular $κ_{T} \leq T /2$ . Applying Equation 5 with $C = C_{T}^{†}$ yields $δ_{T}^{2} α^{2 κ_{T}} + \frac{L ^{κ_{T}}}{T} \leq c_{R} r_{T} (δ) .$ Now fix $M \in M_{T} (t_{0}, ζ, 1 + δ_{T})$ . By Lemma 34, the estimator from Definition 16 satisfies $E_{M} [(θ_{κ_{T}} - θ (M))^{2}] \leq 4 q_{C_{T}}^{2} α^{2 κ_{T}} + \frac{2}{T} {L^{κ_{T} + 1} (1 + \frac{4}{L - 1}) + \frac{4}{1 - α}} .$ Using $q_{C_{T}} \leq δ_{T}$ and the definition of $A_{bd}$ , $E_{M} [(θ_{κ_{T}} - θ (M))^{2}] \leq A_{bd} {δ_{T}^{2} α^{2 κ_{T}} + \frac{L ^{κ_{T}}}{T} + T^{- 1}} .$ Together with Equation 9 and $T^{- 1} \leq r_{T} (δ)$ , this gives $E_{M} [(θ_{κ_{T}} - θ (M))^{2}] \leq c_{att} r_{T} (δ) \leq C_{loc} r_{T} (δ) .$ Taking the supremum over $M \in M_{T} (t_{0}, ζ, 1 + δ_{T})$ proves the asserted estimator bound.

5. The bounded local regime. Assume that there is a constant $D_{bd} \geq 0$ such that $T δ_{T}^{2} \leq D_{bd}$ for all sufficiently large $T$ . Since $q_{1 + δ_{T}} \leq δ_{T}$ , the same tail of $T$ ’s satisfies $T q_{1 + δ_{T}}^{2} \leq D_{bd} .$ Applying the final assertion of Theorem 1 to the sequence $C_{T} = 1 + δ_{T}$ gives a constant $c_{imm} > 0$ such that, for all sufficiently large $T$ , $M \in M_{T} (t_{0}, ζ, 1 + δ_{T}) sup E_{M} [(θ_{0} - θ (M))^{2}] \leq \frac{c _{imm}}{T} .$ This is the claimed immediate-depth conclusion.

∎

Proof of Lemma 1.

Fix $T$ , a policy $p$ , and probability vectors $ν, ν^{'}$ on $S_{grid}$ . We prove the displayed total-variation bound; the $ℓ^{1}$ formulation follows from $∥ η ∥_{1} = 2∥ η ∥_{TV}$ .

1. Let $U (s^{'}) = 1/360$ be the uniform law on $S_{grid}$ . For $s = (g, d_{1}, d_{2}, x_{1}, x_{2}, a_{1})$ , $a \in {0, 1}$ , and $s^{'} = (g^{'}, d_{1}^{'}, d_{2}^{'}, x_{1}^{'}, x_{2}^{'}, a_{1}^{'})$ , define the non-refresh transition weight $D (s, a; s^{'}) = γ_{s, a} (g^{'}) π_{d} (d_{1}^{'}) π_{x} (x_{1}^{'}) 1 {d_{2}^{'} = d_{1}} 1 {x_{2}^{'} = x_{1}} 1 {a_{1}^{'} = a},$ where $π_{d} (0) = 4/5$ , $π_{d} (78) = 1/5$ , $π_{x} (0) = 2/5$ , $π_{x} (31) = 2/5$ , $π_{x} (850) = 1/5$ , and $γ_{s, a} (g^{⋆}) = Pr {round_{{50, 100, 150, 200, 250}} clip_{[50, 250]} (10 + 0.9 g + 0.1 d_{1} + 0.1 d_{2} - 0.01 x_{1} - 0.01 x_{2} - 2 a - 4 a_{1} + ξ) = g^{⋆}}$ for $ξ \sim N (0, 25)$ . The refresh construction in Definition 24 gives, after marginalizing over the reward, $K_{T} {(q, s^{''}) : s^{''} = s^{'} ∣ s, a} = \frac{1}{720} + \frac{1}{2} D (s, a; s^{'}) .$ Therefore the policy-induced matrix can be written as $P_{p}^{(T)} (s, s^{'}) = \frac{1}{720} + \frac{1}{2} R_{p} (s, s^{'}), R_{p} (s, s^{'}) = a \in {0, 1} \sum p (a ∣ x (s)) D (s, a; s^{'}) .$ Equivalently, $P_{p}^{(T)} (s, s^{'}) = (1 - \frac{1}{2}) U (s^{'}) + \frac{1}{2} R_{p} (s, s^{'}) .$

2. For each fixed $s$ , $R_{p} (s, \cdot)$ is a probability vector. Indeed, all factors in $D (s, a; s^{'})$ are nonnegative. Also, $s^{'} \in S_{grid} \sum D (s, a; s^{'}) = g^{⋆} \sum γ_{s, a} (g^{⋆}) d^{⋆} \sum π_{d} (d^{⋆}) x^{⋆} \sum π_{x} (x^{⋆}) = 1,$ because the rounded Gaussian cells partition the clipped glucose outcome, the diet probabilities sum to $1$ , the activity probabilities sum to $1$ , and the three indicator factors merely select the shifted memory coordinates. Thus $s^{'} \in S_{grid} \sum R_{p} (s, s^{'}) = a \in {0, 1} \sum p (a ∣ x (s)) s^{'} \in S_{grid} \sum D (s, a; s^{'}) = a \in {0, 1} \sum p (a ∣ x (s)) = 1.$

3. It remains to apply the common-refresh contraction calculation. For any finite state space, any probability vectors $η, η^{'}$ , and any stochastic matrix $R$ , a kernel of the form $M_{α} (s, t) = (1 - α) u (t) + α R (s, t), α \geq 0,$ satisfies $(η M_{α} - η^{'} M_{α}) (t) = α s \sum (η (s) - η^{'} (s)) R (s, t),$ since the refresh term contributes $(1 - α) u (t) (\sum_{s} η (s) - \sum_{s} η^{'} (s)) = 0$ . Hence $∥ η M_{α} - η^{'} M_{α} ∥_{TV} = \frac{1}{2} t \sum α s \sum (η (s) - η^{'} (s)) R (s, t) \leq α \frac{1}{2} s \sum ∣ η (s) - η^{'} (s) ∣ t \sum R (s, t) = α ∥ η - η^{'} ∥_{TV} .$ With $α = 1/2$ , $u = U$ , $R = R_{p}$ , and $η = ν, η^{'} = ν^{'}$ , this gives $∥ ν P_{p}^{(T)} - ν^{'} P_{p}^{(T)} ∥_{TV} \leq \frac{1}{2} ∥ ν - ν^{'} ∥_{TV} .$ Multiplying by $2$ yields the stated $ℓ^{1}$ version.

∎

Proof of Lemma 2.

Write $P = P_{p}^{(T)}$ for the transition matrix induced by the policy $p$ on the finite state space $S_{grid}$ . We verify the hypotheses of the finite-state contraction fixed-point criterion and then identify the selected fixed point.

Since $p (\cdot ∣ x)$ is a probability law on ${0, 1}$ for each observed grid state $x$ , and since the kernel of $I_{grid}$ is a Markov kernel, the policy mixture $P (s, s^{'}) = a \in {0, 1} \sum p (a ∣ x (s)) K_{T} (s, a; s^{'})$ is stochastic. Thus, for every probability vector $ν$ on $S_{grid}$ , $(ν P) (s^{'}) = s \in S_{grid} \sum ν (s) P (s, s^{'})$ is again a probability vector on $S_{grid}$ . The initial law specified for $I_{grid}$ is also a probability vector.
By the stochasticity verified in the preceding step, the map $ν \mapsto ν P$ sends the probability simplex into itself. The displayed hypothesis of Lemma 2 gives $∥ ν P - ν^{'} P ∥_{TV} \leq \frac{1}{2} ∥ ν - ν^{'} ∥_{TV}$ for all probability vectors $ν, ν^{'}$ . The constant $1/2$ is nonnegative and strictly smaller than $1$ . Since the probability simplex over the finite set $S_{grid}$ is complete in total variation distance, the contraction mapping theorem gives a probability vector $d$ such that $d P = d .$ Equivalently, $d (s^{'}) = s \in S_{grid} \sum d (s) P (s, s^{'}), s^{'} \in S_{grid} .$
The selected law $d_{p}$ is defined as the stationary law chosen for $P$ whenever such a stationary probability vector exists. The preceding step supplies that existence, so the selected law is itself a probability vector and satisfies $d_{p} P = d_{p} .$ Reading this coordinatewise yields $d_{p} (s^{'}) = s \in S_{grid} \sum d_{p} (s) P_{p}^{(T)} (s, s^{'}), s^{'} \in S_{grid},$ which is the asserted stationarity equation.

∎

Proof of Lemma 3.

Fix $s = (g, d_{1}, d_{2}, x_{1}, x_{2}, a_{1}) \in S_{grid}$ and abbreviate $g_{e} (s) := a \in {0, 1} \sum e (a ∣ x (s)) \int p_{1} K_{T} (d p ∣ s, a) .$ By the definition of the target-policy reward regression, the left-hand side of the claimed identity is $g_{e} (s)$ .

For each $a \in {0, 1}$ , the embedded insulin kernel assigns the reward coordinate according to the current glucose component of $s$ . Hence its conditional reward mean is $\int p_{1} K_{T} (d p ∣ s, a) = r_{ins} (s) .$ Indeed, Definition 24 makes the reward coordinate a deterministic function of the present state, with the displayed values defining $r_{ins}$ ; the subsequent randomization affects the next state coordinates but not this reward coordinate.

Substituting this identity into the regression gives $g_{e} (s) = a \in {0, 1} \sum e (a ∣ x (s)) r_{ins} (s) = a \in {0, 1} \sum e (a ∣ x (s)) r_{ins} (s) .$ The target policy in Definition 24 is a probability vector on ${0, 1}$ : if $g \geq 125$ , then $e (1 ∣ x (s)) = 1$ and $e (0 ∣ x (s)) = 0$ , while if $g < 125$ , then $e (1 ∣ x (s)) = 0$ and $e (0 ∣ x (s)) = 1$ . Therefore $a \in {0, 1} \sum e (a ∣ x (s)) = 1,$ and consequently $g_{e} (s) = r_{ins} (s)$ .

This is the asserted identity.

∎

Proof of Lemma 4.

Fix $s = (g, d_{1}, d_{2}, x_{1}, x_{2}, a_{1})$ and write $x = x (s)$ .

Let $cat_{rew} (s) \in {0, 1, 2, 3}$ be the reward category selected by the current glucose coordinate, and let $val_{rew} (0) = - 1, val_{rew} (1) = - \frac{2}{3}, val_{rew} (2) = - \frac{1}{3}, val_{rew} (3) = 0.$ Thus $val_{rew} (cat_{rew} (s)) = r_{ins} (s)$ . In the finite insulin kernel of Definition 24, for each action $a$ the next-state mass may be written as $ω_{s, a} (σ)$ , with $ω_{s, a} (σ) \geq 0, σ \in S_{grid} \sum ω_{s, a} (σ) = 1,$ and the reward coordinate is deterministic at the current-state reward category: $K_{T} ({(val_{rew} (cat_{rew} (s)), σ) : σ \in S_{grid}} ∣ s, a) = 1.$ Therefore finite-space integration gives $\int p_{1} K_{T} (d p ∣ s, a) = σ \in S_{grid} \sum ω_{s, a} (σ) val_{rew} (cat_{rew} (s)) = val_{rew} (cat_{rew} (s)) σ \in S_{grid} \sum ω_{s, a} (σ) = r_{ins} (s) .$
The behavior policy in Definition 24 satisfies $b (1 ∣ x) = \frac{3}{10}, b (0 ∣ x) = \frac{7}{10},$ so $b (a ∣ x) > 0$ for both $a \in {0, 1}$ . By the zero-cell convention in the definition of $ρ$ , the positive-denominator branch applies and hence $b (a ∣ x) ρ (x, a) = b (a ∣ x) \frac{e ( a ∣ x )}{b ( a ∣ x )} = e (a ∣ x) .$
Substituting the two preceding identities into the weighted regression gives $a \in {0, 1} \sum b (a ∣ x) ρ (x, a) \int p_{1} K_{T} (d p ∣ s, a) = a \in {0, 1} \sum e (a ∣ x) r_{ins} (s) = a \in {0, 1} \sum e (a ∣ x) r_{ins} (s) .$ The target policy in Definition 24 injects exactly when $g \geq 125$ , so $e (1 ∣ x) = 1 {g \geq 125}, e (0 ∣ x) = 1 - 1 {g \geq 125}, a \in {0, 1} \sum e (a ∣ x) = 1.$ Consequently $a \in {0, 1} \sum b (a ∣ x) ρ (x, a) \int p_{1} K_{T} (d p ∣ s, a) = r_{ins} (s),$ as claimed.

∎

Proof of Lemma 5.

Fix $k = 0, \dots, m$ and $s^{'} \in S$ . For this pair $(k, s^{'})$ , write the chunked-recurrence data as $G_{1}^{k, s^{'}}, \dots, G_{J_{k, s^{'}}}^{k, s^{'}}, Ξ_{1}^{k, s^{'}} (s^{'}), \dots, Ξ_{J_{k, s^{'}}}^{k, s^{'}} (s^{'}), Σ_{1}^{k, s^{'}}, \dots, Σ_{J_{k, s^{'}} + 1}^{k, s^{'}} .$ These are the block partition, block intervals, and partial-sum intervals attached to this fixed $(k, s^{'})$ . The chunked recurrence first gives the ordinary interval recurrence $s \in S \sum T_{k} (s) \cdot A (s, s^{'}) \subseteq T_{k + 1} (s^{'}) .$ Indeed, take arbitrary choices from the interval summands on the left and group them by the disjoint covering blocks $G_{1}^{k, s^{'}}, \dots, G_{J_{k, s^{'}}}^{k, s^{'}}$ . The contribution of block $G_{j}^{k, s^{'}}$ lies in $Ξ_{j}^{k, s^{'}} (s^{'})$ . Starting from the tail value $0 \in Σ_{J_{k, s^{'}} + 1}^{k, s^{'}}$ , the inclusions $Ξ_{j}^{k, s^{'}} (s^{'}) + Σ_{j + 1}^{k, s^{'}} \subseteq Σ_{j}^{k, s^{'}}, j = J_{k, s^{'}}, \dots, 1,$ show by backward induction that the sum of the block contributions from $j$ through $J_{k, s^{'}}$ lies in $Σ_{j}^{k, s^{'}}$ . At $j = 1$ this is the full sum, and $Σ_{1}^{k, s^{'}} = T_{k + 1} (s^{'})$ . If the block list is empty, the same argument is just the terminal condition $0 \in Σ_{1}^{k, s^{'}} = T_{k + 1} (s^{'})$ . Thus the initial row together with these coordinate recurrences is the ordinary finite-iterate certificate with rows $T_{0}, \dots, T_{m + 1}$ used in the remaining argument.
Define the true iterates $η_{n} : S \to R$ , for all $n \in N$ , by $η_{0} (s) = μ_{0} (s), η_{n + 1} (s^{'}) = s \in S \sum η_{n} (s) P (s, s^{'}) .$ Here $μ_{0}$ is the given initial probability vector, while $η_{n}$ denotes its $n$ -step image under $P$ . The initial condition gives $η_{0} (s) \in T_{0} (s)$ . If $η_{n} (s) \in T_{n} (s)$ for every $s$ , then $P (s, s^{'}) \in A (s, s^{'})$ and exact interval multiplication give $η_{n} (s) P (s, s^{'}) \in T_{n} (s) \cdot A (s, s^{'}),$ and exact interval summation together with the recurrence from the previous step gives $η_{n + 1} (s^{'}) \in T_{n + 1} (s^{'}) .$ Thus, for $n = 0, \dots, m + 1$ , $η_{n} (s) \in T_{n} (s) for every s \in S .$ In particular, $η_{m} (s) \in T_{m} (s), η_{m + 1} (s) \in T_{m + 1} (s) .$ Since $μ_{0}$ is a probability vector and $P$ is stochastic, every $η_{n}$ is a probability vector.
The terminal interval expectation contains the true reward expectation at time $m$ : $s \in S \sum η_{m} (s) r (s) \in s \in S \sum T_{m} (s) \cdot r (s) = E .$ This follows coordinatewise from $η_{m} (s) \in T_{m} (s)$ and $r (s) \in r (s)$ , followed by exact interval products and exact finite interval addition.
The one-step residual of $η_{m}$ is bounded by $δ$ . Since $η_{m + 1} = η_{m} P$ , exact interval subtraction gives $η_{m} (s) - η_{m + 1} (s) \in T_{m} (s) ⊖ T_{m + 1} (s),$ and hence $∣ η_{m} (s) - η_{m + 1} (s) ∣ \leq T_{m} (s) ⊖ T_{m + 1} (s)_{m a x} .$ Summing over $s$ yields $∥ η_{m} - η_{m} P ∥_{1} = ∥ η_{m} - η_{m + 1} ∥_{1} \leq δ .$ Because $π$ is stationary and both $η_{m}$ and $π$ are probability vectors, $∥ η_{m} - π ∥_{1} \leq ∥ η_{m} - η_{m} P ∥_{1} + ∥ η_{m} P - π P ∥_{1} \leq δ + ρ ∥ η_{m} - π ∥_{1} .$ Since $1 - ρ > 0$ , this gives $∥ η_{m} - π ∥_{1} \leq \frac{δ}{1 - ρ} .$
For every state $s$ , $∣ r (s) ∣ \leq max {∣ ℓ (r (s)) ∣, ∣ u (r (s)) ∣} = ∣ r (s) ∣_{m a x} \leq B .$ Therefore $s \in S \sum π (s) r (s) - s \in S \sum η_{m} (s) r (s) = s \in S \sum (π (s) - η_{m} (s)) r (s) \leq s \in S \sum ∣ π (s) - η_{m} (s) ∣ ∣ r (s) ∣ \leq B ∥ π - η_{m} ∥_{1} \leq \frac{B δ}{1 - ρ} .$
Combining the preceding two enclosures, let $x_{m} = s \in S \sum η_{m} (s) r (s), x_{π} = s \in S \sum π (s) r (s) .$ From $x_{m} \in E$ and $∣ x_{π} - x_{m} ∣ \leq \frac{B δ}{1 - ρ},$ we obtain $ℓ (E) - \frac{B δ}{1 - ρ} \leq x_{π} \leq u (E) + \frac{B δ}{1 - ρ} .$ By the definition of $J$ , this is exactly $s \in S \sum π (s) r (s) \in J .$

∎

Proof of Lemma 6.

All arithmetic below is exact rational interval arithmetic with the operations described in Lemma 5.

The residual sums over the $360$ states of Definition 24 evaluate to $δ_{b} = s \in S_{grid} \sum T_{b, 0} (s) ⊖ T_{b, 1} (s)_{m a x} = \frac{2596399843}{200000000000000},$ and $δ_{e} = s \in S_{grid} \sum T_{e, 0} (s) ⊖ T_{e, 1} (s)_{m a x} = \frac{16950307061}{1000000000000000} .$
Since $r_{ins} (s) = [r_{ins} (s), r_{ins} (s)]$ , the interval expectations computed from the terminal rows are $E_{b} = s \in S_{grid} \sum T_{b, 0} (s) \cdot r_{ins} (s) = [- \frac{157888864566941}{300000000000000}, - \frac{63155545824091}{120000000000000}],$ and $E_{e} = s \in S_{grid} \sum T_{e, 0} (s) \cdot r_{ins} (s) = [- \frac{312236715379949}{600000000000000}, - \frac{312236715366659}{600000000000000}] .$
Because $1 - 1/2 = 1/2$ , the expansion radius in each case is $2 δ_{⋆}$ . Hence $J_{b} = [- \frac{157888864566941}{300000000000000} - 2 \frac{2596399843}{200000000000000}, - \frac{63155545824091}{120000000000000} + 2 \frac{2596399843}{200000000000000}] = [- \frac{15789665376647}{30000000000000}, - \frac{315762150721397}{600000000000000}],$ and $J_{e} = [- \frac{312236715379949}{600000000000000} - 2 \frac{16950307061}{1000000000000000}, - \frac{312236715366659}{600000000000000} + 2 \frac{16950307061}{1000000000000000}] = [- \frac{1561285278742111}{3000000000000000}, - \frac{1561081874990929}{3000000000000000}] .$
Taking the interval difference, $B = J_{b} ⊖ J_{e} = [ℓ (J_{b}) - u (J_{e}), u (J_{b}) - ℓ (J_{e})],$ so $B = [- \frac{17884662673771}{3000000000000000}, - \frac{2920912477479}{500000000000000}] .$ The endpoint comparisons are then the rational inequalities $- \frac{6}{1000} = - \frac{18000000000000}{3000000000000000} < - \frac{17884662673771}{3000000000000000} = ℓ (B),$ and $u (B) = - \frac{2920912477479}{500000000000000} < - \frac{2900000000000}{500000000000000} = - \frac{58}{10000} .$ This proves both asserted endpoint bounds.

∎

Proof of Lemma 7.

Fix $T \in N$ . Write $P_{b}^{(T)}$ and $P_{e}^{(T)}$ for the policy-induced transition matrices on $S_{grid}$ obtained from Lemma 1 by taking $p = b$ and $p = e$ , respectively, and write $d_{b}^{(T)}$ and $d_{e}^{(T)}$ for their selected stationary laws.

The behavior and target kernels both satisfy the one-half contraction estimate $∥ ν P_{b}^{(T)} - ν^{'} P_{b}^{(T)} ∥_{1} \leq \frac{1}{2} ∥ ν - ν^{'} ∥_{1}, ∥ ν P_{e}^{(T)} - ν^{'} P_{e}^{(T)} ∥_{1} \leq \frac{1}{2} ∥ ν - ν^{'} ∥_{1}$ for all probability vectors $ν, ν^{'}$ on $S_{grid}$ . This is exactly the $ℓ^{1}$ -form supplied by Lemma 1, instantiated first at $p = b$ and then at $p = e$ . The corresponding selected laws are probability vectors and satisfy $d_{b}^{(T)} P_{b}^{(T)} = d_{b}^{(T)}, d_{e}^{(T)} P_{e}^{(T)} = d_{e}^{(T)},$ by Lemma 2.
Let $r_{ins} (s) = ⎩ ⎨ ⎧ - 1, - 2/3, - 1/3, 0, g \leq 70, g > 150, 70 < g \leq 80 or 120 < g \leq 150, otherwise, s = (g, d_{1}, d_{2}, x_{1}, x_{2}, a_{1}) .$ For $⋆ \in {b, e}$ , with $P_{⋆}^{(T)}$ denoting $P_{b}^{(T)}$ or $P_{e}^{(T)}$ , let $A_{⋆}$ be the exact rational interval matrix recorded by the finite-grid certificate. Its entrywise enclosure is $P_{⋆}^{(T)} (s, s^{'}) \in A_{⋆} (s, s^{'}), s, s^{'} \in S_{grid} .$ The same certificate supplies an exact rational initial probability row $μ_{⋆, 0}$ , interval rows $T_{⋆, 0}$ and $T_{⋆, 1}$ , and an exact chunked recurrence from the first interval row to the second. In the notation of Lemma 5, its rational inclusions verify $[μ_{⋆, 0} (s), μ_{⋆, 0} (s)] \subseteq T_{⋆, 0} (s), s \in S_{grid} .$ For every output state, the recorded blocks partition $S_{grid}$ , every block has at most $40$ states, and the block and partial-sum intervals satisfy exactly the inclusion chain in Lemma 5. Because the certificate has zero approximation steps, this verified recurrence links its initial interval row directly to its successor row. Passing from the chunked certificate to the finite recurrence used by that lemma preserves both rows and all the rational inclusions. For the reward certificate, put $r_{ins} (s) = [r_{ins} (s), r_{ins} (s)] .$ Since $∣ r_{ins} (s) ∣ \leq 1$ for every grid state, the reward bound in Lemma 5 is $B = 1$ . Define $δ_{⋆} = s \in S_{grid} \sum T_{⋆, 0} (s) ⊖ T_{⋆, 1} (s)_{m a x}, E_{⋆} = s \in S_{grid} \sum T_{⋆, 0} (s) \cdot r_{ins} (s) .$ After that conversion, the terminal row is $T_{⋆, 0}$ and the successor row is $T_{⋆, 1}$ , so the residual and stationary-reward interval are $δ_{⋆} = s \in S_{grid} \sum T_{⋆, 0} (s) ⊖ T_{⋆, 1} (s)_{m a x}, J_{⋆} = [ℓ (E_{⋆}) - \frac{δ _{⋆}}{1 - 1/2}, u (E_{⋆}) + \frac{δ _{⋆}}{1 - 1/2}] .$ The initial-row, recurrence, reward-bound, contraction, and stationarity hypotheses of Lemma 5 are therefore satisfied for both policies after passing each chunked recurrence certificate to its associated finite recurrence certificate. Under this conversion, the terminal interval row is $T_{⋆, 0}$ , the successor interval row is $T_{⋆, 1}$ , and the stationary-reward interval produced by Lemma 5 is exactly $J_{⋆} = [ℓ (E_{⋆}) - \frac{δ _{⋆}}{1 - 1/2}, u (E_{⋆}) + \frac{δ _{⋆}}{1 - 1/2}],$ which is the chunked interval $J_{⋆}$ used here. Hence $s \in S_{grid} \sum d_{b}^{(T)} (s) r_{ins} (s) \in J_{b}, s \in S_{grid} \sum d_{e}^{(T)} (s) r_{ins} (s) \in J_{e} .$
Define the subtraction interval $B := J_{b} ⊖ J_{e} = [ℓ (J_{b}) - u (J_{e}), u (J_{b}) - ℓ (J_{e})] .$ Interval subtraction is sound, so the two inclusions from the previous step give $s \sum d_{b}^{(T)} (s) r_{ins} (s) - s \sum d_{e}^{(T)} (s) r_{ins} (s) \in B .$
The reward-regression identities in Lemmas 4 and 3 give, for every $s \in S_{grid}$ , $a \in {0, 1} \sum b (a ∣ x (s)) ρ (x (s), a) \int p_{1} K_{T} (d p ∣ s, a) = r_{ins} (s),$ and $a \in {0, 1} \sum e (a ∣ x (s)) \int p_{1} K_{T} (d p ∣ s, a) = r_{ins} (s) .$ The finite-grid embedding preserves the policy-induced kernels entry by entry: $P_{b}^{(T)} (s, s^{'}) = a \sum b (a ∣ x (s)) K_{T} (R \times {s^{'}} ∣ s, a), P_{e}^{(T)} (s, s^{'}) = a \sum e (a ∣ x (s)) K_{T} (R \times {s^{'}} ∣ s, a) .$ Consequently the stationary laws in the interval certificate are exactly the behavior- and target-policy stationary laws entering the two regression identities. Combining these kernel identifications, the two regression identities, and Definition 12 yields $Δ_{imm} (T) = s \sum d_{b}^{(T)} (s) r_{ins} (s) - s \sum d_{e}^{(T)} (s) r_{ins} (s) .$ Therefore $Δ_{imm} (T) \in B .$
By the exact rational interval evaluation in Lemma 6, the endpoints of the subtraction interval $B$ satisfy $- \frac{6}{1000} < ℓ (B), u (B) < - \frac{58}{10000} .$ Since $Δ_{imm} (T) \in B$ , we have $ℓ (B) \leq Δ_{imm} (T) \leq u (B) .$ The last two displays imply $- \frac{6}{1000} < Δ_{imm} (T) < - \frac{58}{10000},$ as claimed.

∎

Proof of Theorem 5.

Fix $T$ . We verify the five asserted clauses.

By Definition 24, the observed coordinate has $∣ X ∣ = ∣ {50, 100, 150, 200, 250} ∣ ∣ {0, 31, 850} ∣^{2} ∣ {0, 1} ∣ = 5 \cdot 3^{2} \cdot 2 = 90,$ and the latent diet-memory coordinate has $∣ H ∣ = ∣ {0, 78} ∣^{2} = 2^{2} = 4.$ Thus the joint-state alphabet $S = X \times H$ has $∣ S ∣ = 90 \cdot 4 = 360.$
For a memoryless policy $p \in {b, e}$ , let $P_{p}^{(T)} : S \times S \to [0, 1]$ be the reward-marginalized joint-state transition matrix $P_{p}^{(T)} (s, s^{'}) := a \in {0, 1} \sum p (a ∣ s_{X}) K_{T} {(r_{out}, \overset{s}{ˉ}) : \overset{s}{ˉ} = s^{'} ∣ s, a}, s, s^{'} \in S .$ By the construction in Definition 24, the embedded policy kernel of the refreshed insulin experiment is this matrix. Hence, for every pair of probability laws $ν, ν^{'}$ on $S$ , Lemma 1 gives $∥ ν P_{p}^{(T)} - ν^{'} P_{p}^{(T)} ∥_{TV} \leq \frac{1}{2} ∥ ν - ν^{'} ∥_{TV} .$ Applying this with $p = b$ and $p = e$ is exactly Assumption 6 with $α = 1/2$ .
The behavior probabilities are $b (1 ∣ x) = \frac{3}{10}, b (0 ∣ x) = \frac{7}{10},$ and the target policy is a probability law with $0 \leq e (a ∣ x) \leq 1$ . Therefore $e (1 ∣ x) \leq 1 = \frac{10}{3} \frac{3}{10} = \frac{10}{3} b (1 ∣ x),$ and $e (0 ∣ x) \leq 1 \leq \frac{7}{3} = \frac{10}{3} \frac{7}{10} = \frac{10}{3} b (0 ∣ x) .$ Thus Assumption 5 holds with $L = 10/3$ .
Let $d_{b}$ and $d_{e}$ be the selected stationary laws of $P_{b}^{(T)}$ and $P_{e}^{(T)}$ . The behavior and target policies in Definition 24 are probability vectors; together with the contraction estimate supplied by Lemma 1, Lemma 2 gives $d_{p} (s^{'}) = s \in S \sum d_{p} (s) P_{p}^{(T)} (s, s^{'}), s \in S \sum d_{p} (s) = 1, d_{p} (s) \geq 0,$ for $p \in {b, e}$ and all $s^{'} \in S$ . Write $S_{t}$ for the joint state at time $t$ . For $a \in {0, 1}$ and $s, s^{'} \in S$ , define the nonrefresh action kernel as follows. If $s = (g, d_{1}, d_{2}, x_{1}, x_{2}, a_{1}), s^{'} = (g^{'}, d_{1}^{'}, d_{2}^{'}, x_{1}^{'}, x_{2}^{'}, a_{1}^{'}),$ let $φ_{s, a} (g^{'})$ be the probability that the Gaussian clipping-and-rounding rule in Definition 24 produces next glucose value $g^{'}$ , and set this probability to zero when $g^{'}$ is outside the glucose grid. Let $π_{D} (0) = \frac{4}{5}, π_{D} (78) = \frac{1}{5}, π_{E} (0) = π_{E} (31) = \frac{2}{5}, π_{E} (850) = \frac{1}{5},$ with $π_{D}$ and $π_{E}$ equal to zero off their displayed supports. Conditional on the nonrefresh branch and action $a$ , Definition 24 draws the new diet and activity coordinates with laws $π_{D}$ and $π_{E}$ , shifts the old memory coordinates, and stores the current action. Hence the nonrefresh probability of moving from $s$ to $s^{'}$ under action $a$ is $ω_{a} (s, s^{'}) := φ_{s, a} (g^{'}) π_{D} (d_{1}^{'}) π_{E} (x_{1}^{'}) 1 {d_{2}^{'} = d_{1}} 1 {x_{2}^{'} = x_{1}} 1 {a_{1}^{'} = a} .$ All factors in this display are nonnegative, so $ω_{a} (s, s^{'}) \geq 0$ . The refresh branch in Definition 24 has probability $1/2$ and draws the next full state uniformly from the $360$ -point set $S$ , so its contribution to any fixed next state $s^{'}$ is $\frac{1}{2} \cdot \frac{1}{360} = \frac{1}{720} .$ The remaining branch has probability $1/2$ , and the policy-induced kernel averages the action-specific nonrefresh probabilities with weights $p (a ∣ s_{X})$ . Thus, for every memoryless policy $p$ on ${0, 1}$ , $P_{p}^{(T)} (s, s^{'}) = \frac{1}{720} + \frac{1}{2} a \in {0, 1} \sum p (a ∣ s_{X}) ω_{a} (s, s^{'}), s, s^{'} \in S .$ Since the behavior policy has nonnegative action probabilities, this identity gives the pointwise minorization $P_{b}^{(T)} (s, s^{'}) = \frac{1}{720} + \frac{1}{2} a \in {0, 1} \sum b (a ∣ s_{X}) ω_{a} (s, s^{'}) \geq \frac{1}{720}, s, s^{'} \in S .$ The stationary-minorization calculation is then $d_{b} (s^{'}) = s \in S \sum d_{b} (s) P_{b}^{(T)} (s, s^{'}) \geq s \in S \sum d_{b} (s) \frac{1}{720} = \frac{1}{720}, s^{'} \in S .$ Since $d_{e}$ is a probability vector, $d_{e} (s^{'}) \leq \overset{s}{ˉ} \in S \sum d_{e} (\overset{s}{ˉ}) = 1.$ Combining the last two displays yields $d_{e} (s^{'}) \leq 1 \leq 720 d_{b} (s^{'}), s^{'} \in S,$ and $720 \geq 1$ . This proves Assumption 7 with $C = 720$ .
For the same horizon $T$ , write $K$ for the transition kernel of $I_{grid}$ , $b$ and $e$ for its behavior and target policies, and $d_{b}$ for the stationary law induced by $b$ . The stationary immediate-weighting bias diagnostic is $Δ_{imm} (T) = s \sum d_{b} (s) a \in {0, 1} \sum b (a ∣ s_{X}) \frac{e ( a ∣ s _{X} )}{b ( a ∣ s _{X} )} \int r K (d s^{'}, d r ∣ s, a) - θ (K, e) .$ For this diagnostic on the refreshed finite insulin adaptation $I_{grid}$ of Definition 24, Lemma 7 supplies the interval certificate $- \frac{6}{1000} < Δ_{imm} (T) < - \frac{58}{10000},$ which is the displayed bound $- 0.006 < Δ_{imm} (T) < - 0.0058.$

The five displayed conclusions are precisely the conjunction asserted for the refreshed insulin experiment.

∎

References

Hu, Yuchen and Wager, Stefan (2023). Off-Policy Evaluation in Partially Observed Markov Decision Processes under Sequential Ignorability. The Annals of Statistics. arXiv
Mehrabi, Mohammad and Wager, Stefan (2025). Off-Policy Evaluation in Markov Decision Processes under Weak Distributional Overlap. . arXiv
Zhang, Yuheng and Jiang, Nan (2024). On the Curses of Future and History in Future-dependent Value Functions for Off-policy Evaluation. Advances in Neural Information Processing Systems. arXiv
Tennenholtz, Guy and Shalit, Uri and Mannor, Shie (2020). Off-Policy Evaluation in Partially Observable Environments. Proceedings of the AAAI Conference on Artificial Intelligence. doi
Kallus, Nathan and Uehara, Masatoshi (2020). Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes. Journal of Machine Learning Research. arXiv
Liao, Peng and Klasnja, Predrag and Murphy, Susan A. (2021). Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Health. Journal of the American Statistical Association. doi
Luckett, Daniel J. and Laber, Eric B. and Kahkoska, Anna R. and Maahs, David M. and Mayer-Davis, Elizabeth J. and Kosorok, Michael R. (2020). Estimating Dynamic Treatment Regimes in Mobile Health Using {V}-Learning. Journal of the American Statistical Association. doi
Maahs, David M. and Mayer-Davis, Elizabeth J. and Bishop, Freya K. and Wang, Lin and Mangan, Margo and McMurray, Robert G. (2012). Outpatient Assessment of Determinants of Glucose Excursions in Adolescents with Type 1 Diabetes: Proof of Concept. Diabetes Technology \& Therapeutics. doi
Robins, James M. (1986). A New Approach to Causal Inference in Mortality Studies with a Sustained Exposure Period---Application to Control of the Healthy Worker Survivor Effect. Mathematical Modelling. doi
Rosenbaum, Paul R. and Rubin, Donald B. (1983). The Central Role of the Propensity Score in Observational Studies for Causal Effects. Biometrika. doi
Robins, James M. and Hern{\'a}n, Miguel A. and Brumback, Babette (2000). Marginal Structural Models and Causal Inference in Epidemiology. Epidemiology. doi
Murphy, Susan A. (2003). Optimal Dynamic Treatment Regimes. Journal of the Royal Statistical Society: Series B (Statistical Methodology). doi
Smallwood, Richard D. and Sondik, Edward J. (1973). The Optimal Control of Partially Observable Markov Processes over a Finite Horizon. Operations Research. doi
Kaelbling, Leslie Pack and Littman, Michael L. and Cassandra, Anthony R. (1998). Planning and Acting in Partially Observable Stochastic Domains. Artificial Intelligence. doi
Sutton, Richard S. and Barto, Andrew G. (2018). Reinforcement Learning: An Introduction. MIT Press.
Precup, Doina and Sutton, Richard S. and Singh, Satinder P. (2000). Eligibility Traces for Off-Policy Policy Evaluation. Proceedings of the Seventeenth International Conference on Machine Learning.
Jiang, Nan and Li, Lihong (2016). Doubly Robust Off-policy Value Evaluation for Reinforcement Learning. Proceedings of the 33rd International Conference on Machine Learning. arXiv
Thomas, Philip S. and Brunskill, Emma (2016). Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning. Proceedings of the 33rd International Conference on Machine Learning. arXiv
Liu, Qiang and Li, Lihong and Tang, Ziyang and Zhou, Dengyong (2018). Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation. Advances in Neural Information Processing Systems. arXiv
Nachum, Ofir and Chow, Yinlam and Dai, Bo and Li, Lihong (2019). {DualDICE}: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections. Advances in Neural Information Processing Systems. arXiv
Uehara, Masatoshi and Huang, Jiawei and Jiang, Nan (2020). Minimax Weight and {Q}-Function Learning for Off-Policy Evaluation. Proceedings of the 37th International Conference on Machine Learning. arXiv
Bennett, Andrew and Kallus, Nathan (2021). Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processes. . arXiv
Shi, Chengchun and Uehara, Masatoshi and Huang, Jiawei and Jiang, Nan (2022). A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processes. Proceedings of the 39th International Conference on Machine Learning. arXiv
Miao, Rui and Qi, Zhengling and Zhang, Xiaoke (2022). Off-Policy Evaluation for Episodic Partially Observable Markov Decision Processes under Non-Parametric Models. Advances in Neural Information Processing Systems. arXiv
Uehara, Masatoshi and Kiyohara, Haruka and Bennett, Andrew and Chernozhukov, Victor and Jiang, Nan and Kallus, Nathan and Shi, Chengchun and Sun, Wen (2023). Future-Dependent Value-Based Off-Policy Evaluation in {POMDPs}. Advances in Neural Information Processing Systems. arXiv
Zhang, Yuheng and Jiang, Nan (2025). Statistical Tractability of Off-Policy Evaluation of History-Dependent Policies in {POMDPs}. International Conference on Learning Representations. arXiv
Kuang, Qi and Wang, Jiayi and Zhou, Fan and Qi, Zhengling (2025). Breaking the Order Barrier: Off-Policy Evaluation for Confounded {POMDPs}. Advances in Neural Information Processing Systems.
Liao, Peng and Qi, Zhengling and Wan, Runzhe and Klasnja, Predrag and Murphy, Susan A. (2022). Batch Policy Learning in Average Reward Markov Decision Processes. The Annals of Statistics. doi
Nahum-Shani, Inbal and Smith, Shawna N. and Spring, Bonnie J. and Collins, Linda M. and Witkiewitz, Katie and Tewari, Ambuj and Murphy, Susan A. (2018). Just-in-Time Adaptive Interventions ({JITAIs}) in Mobile Health: Key Components and Design Principles for Ongoing Health Behavior Support. Annals of Behavioral Medicine. doi

Formal layer — Lean correspondence

Every object below is formalized in Lean 4. Click a verified row to view its statement; the rest link to the Lean source.

Setups (3)

Setup S-1 — A single trajectory $(X_t,A_t,Y_t)_{t=1}^{T}$ from a time-homogeneous partially observed Markov decision process on a finite joint state space $\mathcal S=\mathcal X\times\mathcal H$ with binary action alphabet $\mathcal A=\{0,1\}$ and rewards bounded by one in absolute value. The primitives are a joint reward-transition kernel $K(dy,dx',dh'\mid x,h,a)$ , a known memoryless behavior policy $b$ and a known memoryless target policy $e$ ; derived from them are the policy-induced joint-state kernels $P_p$ , their stationary laws $d_p$ , the target-policy reward regression $g_e$ , the stationary target value $\theta(K,e)$ , and the observable one-step policy ratio $\rho_t$ . The regime parameters are the mixing scale $t_0$ , the log policy-overlap bound $\zeta$ and the latent stationary-overlap radius $C$ , with derived constants $\alpha=e^{-1/t_0}$ and $L=e^{\zeta}$ . The latent coordinates $H_t$ , the joint states $S_t=(X_t,H_t)$ and both stationary laws are unobserved; only $\mathsf O_T$ — the observed states $X_t$ , the actions $A_t$ and the reward VALUES $Y_t\in[-1,1]$ — together with $b$ , $e$ and the observed alphabet is revealed. A trajectory of length $T$ consists of the joint states $S_1,\ldots,S_{T+1}$ together with the action-reward pairs $(A_t,Y_t)$ for $t=1,\ldots,T$ : at each epoch $t$ the action is drawn from $b(\cdot\mid X_t)$ and the kernel $K$ generates the pair $(Y_t,S_{t+1})$ from $(S_t,A_t)$ , so the terminal state $S_{T+1}$ belongs to the trajectory while the observation records only the first $T$ states, and of those only their observed coordinates. An estimator sees each epoch exactly through the triple $(X_t,A_t,Y_t)$ : its argument is the sequence of those triples itself, and the reward enters only through its real value in $[-1,1]$ . No restriction is placed on the reward law beyond that bound: the conditional law of $Y_t$ under $K$ may be any probability law on $[-1,1]$ , discrete, continuous or mixed. — Current P15 architecture. JointState nX nH := Fin nX × Fin nH. Step nX nH := ℝ × JointState nX nH. FullTrajectory T nX nH := (Fin (T + 1) → JointState nX nH) × (Fin T → Bool × ℝ). ObsView T nX := Fin T → (Fin nX × Bool × ℝ) is the projection to (X_t, A_t, Y_t) from the first T states and the T action/reward coordinates; the terminal state is not observed. K : JointState nX nH → Bool → Measure (ℝ × JointState nX nH) has an arbitrary real reward law supported on [-1,1]. There is no class-level reward alphabet or coding map.
Setup S-3 — The signed-depth two-point sub-experiment inside $\mathcal M_T(t_0,\zeta,C)$ for $C>1$ . The observed state is constant, the hidden state is $H_t=(J_t,U_t)$ with depth $J_t\in\{0,\ldots,Q\}$ and sign $U_t\in\{-1,+1\}$ , the behavior policy plays action one with probability $1/L$ and the target policy plays action one surely. A reset draws the sign with bias $\varepsilon_C=\min\{(C-1)/2,1/2\}$ ; an advance retains the sign under action one and refreshes it under action zero; at terminal depth the reward is Rademacher with mean $vc_0U_t$ , where $c_0=(1-\alpha)/4$ and $v\in\{-1,+1\}$ indexes the two alternatives. The depth marginal is stationary with masses $w_j=(1-\alpha)\alpha^j/(1-\alpha^{Q+1})$ , and the two induced observed-path laws are $\mathbb P_{Q,v}^{\mathrm{obs},T}$ . Attached to them is the observed filtration $\mathcal F_{t-1}^{\mathrm{obs}}=\sigma(A_1,Y_1,\ldots,A_{t-1},Y_{t-1})$ , with observed prefix length $\ell_t=\min\{Q,t-1\}$ , predictive terminal posterior $q_t^v=\Pr_v(J_t=Q\mid\mathcal F_{t-1}^{\mathrm{obs}})$ , rare-action-history factor $a_t^{(Q)}$ , and the two constants $B_0$ (observed divergence) and $K_0$ (uniform terminal posterior). — PROCESS. ROUND-9 FRAMING (decision P14 part (5)), which is what keeps this whole block intact: the signed-depth family is built as a FiniteRewardModel T 1 (2*(Q+1)) 2 and then embeded into the real-reward class of S1, so every computation described below stays a FINITE computation and only the final KL budget and the absolute-continuity discharge are transported to the real-reward observed law -- by InformationTheory.klDiv_map_le (data processing, the <= direction the converse needs) and MeasureTheory.Measure.AbsolutelyContinuous.map respectively, through the Helpers/FiniteEncoding.lean bridges embed_klDiv_le and embed_absolutelyContinuous. The witness is a genuine member of the real-reward class (its rewards are Rademacher, a bounded real law that happens to be finitely supported), so nothing about the converse is weakened by the world change; what WAS weakened before round 9 was the upper bracket, which is a claim over the whole class and is repaired in S1/S2. Lean carrier (at the FiniteRewardModel level): latent alphabet Fin (Q+1) x Bool encoded as Fin (2*(Q+1)); observed alphabet Fin 1; reward alphabet Fin 2 with rew = ![-1,1] (injective, which is why conditioning on the symbol prefix and on the real reward prefix generate the same information and the filter realization below is faithful for the embedded law). $\mathcal F_{t-1}^{\mathrm{obs}}$ is realized as conditioning on the observed prefix Fin t -> (Bool x Fin 2) (a Fintype), so q_t^v and a_t^{(Q)} are ordinary real-valued functions of a prefix rather than sigma-algebra objects -- the standard finite-alphabet realization of a filtration, and the only encoding that keeps the chain-rule step elementary. REUSE for the divergence step: the observed-path KL is factored with Mathlib's InformationTheory.klDiv_compProd_eq_add (Mathlib/InformationTheory/KullbackLeibler/ChainRule.lean, the full chain rule KL(mu (x) kappa, nu (x) eta) = KL(mu,nu) + integral of KL(kappa,eta)) and InformationTheory.klDiv_compProd_left; the action conditionals cancel through Causalean.Mathlib.InformationTheory.Measure.klDiv_compProd_left. The testing floor is Causalean.Stat.bretagnolle_huber_affinity (Causalean/Stat/Minimax/BretagnolleHuber.lean:373), used DIRECTLY: it states (1/2) * Real.exp (-(klDiv mu nu).toReal) <= 1 - tvDist mu nu, which is verbatim the note's (24) overlap inequality with an EXPLICIT constant, and it composes with Causalean.Stat.real_two_point_lower_bound (Causalean/Stat/Minimax/MinimaxRisk.lean:61, the note's sign-classification step) and Mathlib's mul_meas_ge_le_integral_of_nonneg (the note's (25) squared-loss event bound) to give the explicit two-point floor. So the note's 'Jensen + Cauchy-Schwarz + sign classification' paragraph is REPLACED by reuse and does not need reproving, but the composition is done in the paper-local primitive two_point_floor_explicit (Helpers/TwoPoint.lean) rather than by calling Causalean.Stat.Minimax.le_cam_two_point_mse. ROUND-3 CORRECTION (decision P8): the earlier version of this note claimed le_cam_two_point_mse 'already packages exp(-K)/32 as the constant' -- that is FALSE at the level of its statement, which concludes only exists cK, 0 < cK and ..; the witness exp(-K)/32 is picked inside its proof (LeCamTwoPoint.lean:239) and is invisible to consumers. That reuse therefore stays available for the EXISTENTIAL-rate converses only, and the explicit-constant floors go through the three public ingredients just named. ROUND-4 CORRECTION (decision P9): bretagnolle_huber_affinity states its floor as (1/2) * Real.exp (-(klDiv mu nu).toReal) <= 1 - tvDist mu nu, so converting a budget written as klDiv <= ENNReal.ofReal K into the displayed Real.exp (-K) needs (klDiv).toReal <= K, i.e. ENNReal.toReal_mono plus ENNReal.toReal_ofReal, and the latter REQUIRES 0 <= K (for K < 0 the step is false, not merely unproved). The primitive two_point_floor_explicit therefore carries an explicit (hK : 0 <= K) premise, threaded through both tiers and through every consumer; it is discharged by norm_num at the two concrete budgets this world supplies, K = 1/4 (the equal-policy Rademacher pair, writeup.tex eq. (32)/(10)) and K = 1/8 (the signed-depth pair, writeup.tex eqs. (23)/(14)/(34)). ROUND-5 CORRECTION (decision P10): bretagnolle_huber_affinity also requires hac : mu << nu and hfin : klDiv mu nu <> top, and these are NOT free on this world's Fintype observed-path space -- finiteness of FiniteObsView T 1 2 gives measurability and llr-integrability (MeasureTheory.Integrable.of_finite) but says nothing about which observed words carry mass, and a degenerate in-class member can charge a word its partner does not. What actually discharges them here is the STRICT POSITIVITY this world does have, and the writeup already proves: b(1) = 1/L in (0,1) so both actions have positive behaviour probability, and every conditional reward probability lies in [(1 - c_0)/2, (1 + c_0)/2] with c_0 = (1 - alpha)/4 in (0, 1/4), so every observed word has positive mass. That sentence becomes the lemma signedDepth_fullSupportObs : FullSupportObs (signedDepthFamily ..) in Helpers/SignedDepth.lean, fed to the generic bridges obsLaw_absolutelyContinuous_of_fullSupport and klDiv_ne_top_of_fullSupport (Helpers/Kernels.lean, over PMF.toMeasure_apply_singleton and InformationTheory.klDiv_ne_top_iff). The same lemma is what makes this world's KL BUDGET meaningful at all: without it klDiv is top and the budget inequality is false, so the support work is load-bearing regardless of whether F2 keeps hac/hfin as premises or re-derives them from the budget the way LeCamTwoPoint.lean:266-269 does. No other part of this block changes: the chain-rule reuse, the filter realization and the two-point composition are exactly as at round 3. ROUND-9 ADDENDUM (decision P14): the affinity and testing steps are now applied at the EMBEDDED observed laws, i.e. to measures on ObsView T 1, rather than at the finite FiniteObsView T 1 2, so their side conditions are the TRANSPORTED forms of the same finite work -- hac is embed_absolutelyContinuous applied to signedDepth_fullSupportObs, hfin follows from the transported budget through embed_klDiv_le, and the estimator's measurability (previously free from the Fintype) is supplied by the converse's own premise per decision P14 part (6). The finite chain-rule assembly, the posterior recursion and the full-support lemma are all unchanged statements about the FiniteRewardModel. SEARCH TRACE: lean_local_search 'klDiv_compProd' -> InformationTheory.klDiv_compProd_eq_add + klDiv_compProd_left (Mathlib), Causalean.Mathlib.InformationTheory.Measure.klDiv_compProd_left; lean_leansearch 'Kullback-Leibler divergence chain rule for composition product of measure and kernel' -> confirms Mathlib coverage; lean_local_search 'bretagnolle_huber' -> Causalean.Stat.bretagnolle_huber_affinity. ROUND-3 RE-SURVEY: Grep of Causalean/Stat/Minimax/BretagnolleHuber.lean and LeCamTwoPoint.lean for 'theorem|lemma' with context (read the full statement of bretagnolle_huber_affinity and the full proof of le_cam_two_point_mse, lines 228-337) and Grep of /Causalean for 'half_one_sub_tvDist_le_max_error' -> Causalean.Stat.half_one_sub_tvDist_le_max_error (LeCam.lean:76, metric-space form) and its real specialization Causalean.Stat.real_two_point_lower_bound (MinimaxRisk.lean:61), both public. ROUND-10 ADDENDUM (decision P15 part (6)): this world's Lean carrier is respelled to the two-block finite trajectory -- signedDepthFinite .. : FiniteRewardModel T 1 (2*(Q+1)) 2 with law : PMF (FiniteTrajectory T 1 (2*(Q+1)) 2), a Fin (T+1)-indexed joint-state block (constant observed coordinate, latent (J,U) packed into Fin (2*(Q+1))) paired with T (action, reward-symbol) coordinates, built by the PMF.bind/Fin.snoc epoch fold -- and its observed-symbol space is FiniteObsView T 1 2. NOTHING in this block's mathematics moves: the reset/advance depth dynamics, the sign dynamics, the Rademacher terminal reward, w_j, eps_C, c_0, the filtration realization as conditioning on ObsPrefix t := Fin t -> (Bool x Fin 2), the chain-rule assembly, the posterior recursion, the full-support lemma and the two-point composition are all unchanged; the transport to the real-reward observed law is still one embed_klDiv_le / embed_absolutelyContinuous step. What the round-10 shape fixes here is the same defect as everywhere: the finite fold produces a genuine successor state for the LAST epoch, so this witness satisfies ass:pomdp-kernel's round-10 equation at every t : Fin T rather than at all but one.
Setup S-4 — A refreshed finite-state insulin-policy adaptation $\mathcal I_{\mathrm{grid}}$ : a $360$ -state controlled Markov model whose state $s=(g,d_1,d_2,x_1,x_2,a_1)$ records a discretized glucose level, two lagged diet indicators, two lagged activity levels and the previous action. The diet coordinates form the latent state $H_t$ and all remaining coordinates form the observed state $X_t$ ; the behavior policy injects insulin with probability $3/10$ and the target policy injects exactly when glucose is at least $125$ . With probability one half the next full state is uniform, and otherwise the next glucose is a Gaussian autoregressive update rounded to the grid while the memories shift. The reward is a deterministic four-valued function of glucose. The diagnostic $\Delta_{\mathrm{imm}}$ is the difference between the stationary expectation of $Y_te(A_t\mid X_t)/b(A_t\mid X_t)$ and $\theta(K,e)$ in this model. — PROCESS. REVISED at F2.5: this S-block is a LOCAL finite insulin-POMDP WORLD, not a reuse of the interval-arithmetic package. The earlier entry mis-set disposition:"reuse" with reuse: ...RatInterval, which named a certificate-layer numeric helper as if it were the ambient world -- RatInterval binds neither $\mathcal I_{\mathrm{grid}}$ nor $\Delta_{\mathrm{imm}}$ and cannot, since it is a rational-interval type with no experiment, policy, kernel or stationary law in it. BYPASS-JUSTIFIED (one-line reason): Causalean carries no finite-POMDP experiment world at this abstraction, so the world is realised locally over the S1 carrier. The two bound symbols become two local defs in Helpers/InsulinGrid.lean. ROUND-9 SPELLING (decision P14 part (5)): the model is built as noncomputable def insulinGridFinite (T : Nat) : FiniteRewardModel T 90 4 4 (its reward is a deterministic four-valued function of glucose, so the finite device applies verbatim) and the world instance is noncomputable def insulinGrid (T : Nat) : RawPomdpExperiment T 90 4 := embed (insulinGridFinite T), whose reward kernel is a Dirac at the reward value; embed_policyKernel_eq and embed_stationaryLaw_eq make the four structural conjuncts (360 states, Dobrushin <= 1/2, L = 10/3, d_e <= 720 d_b) the SAME finite-vector computations as before, and embed_targetValue_eq does the same for the certificate conjunct, so nothing in this block's mathematics moves. The finite data are unchanged (observed alphabet 5 glucose x 3 x 3 activity x 2 previous action = 90, latent diet alphabet {0,78}^2 = 4, reward alphabet {-1,-2/3,-1/3,0} of size 4, kernel = (1/2) uniform refresh + (1/2) action-specific Gaussian-rounded update at the cut points 75/125/175/225, b(1|x) = 3/10, e(1|x) = indicator g >= 125), and noncomputable def immediateWeightBias (T : Nat) : R realising $\Delta_{\mathrm{imm}}$ as the stationary expectation of Y_t * ratio b e X_t A_t minus targetValue (insulinGrid T) -- i.e. both are instances of the S1 world's own RawPomdpExperiment / policyKernel / stationaryLaw / targetValue machinery, which is why this block only NARROWS S1 and adds no new abstraction. DEMOTED REUSE (certificate layer only, not the world): RatInterval (Causalean/Mathlib/Analysis/CertifiedContourIntervalArithmetic/Basic.lean:22, rational closed intervals with a Contains real-containment predicate) plus the proved arithmetic-enclosure operations in .Operations, and expScalar with expScalar_sound / expScalar_width (.../Exponential.lean:50,114,335) for the rational enclosure of Real.exp at rational points, are reused ONLY inside the numeric certificate for conjunct (5) of prop:finite-insulin-demonstration; they replace the note's bespoke Machin pi bound, 1/sqrt(2*pi) enclosure and truncated erf series, and F3 builds the Gaussian-CDF enclosure Phi on top of expScalar rather than reproving series tails. Nothing structural in this block depends on them: conjuncts (1)-(4) of that proposition (360 states, Dobrushin <= 1/2, L = 10/3, d_e <= 720 d_b) are pure finite-vector algebra over S1. SEARCH TRACE (world layer, re-run at F2.5): lean_local_search 'insulin', 'glucose', 'POMDP', 'controlled Markov', 'finite experiment' and Grep of <repo-root>/Causalean for 'insulin|glucose|POMDP|MarkovDecision|controlledMarkov' -> zero hits; the only Markov-chain file in Causalean is Causalean/Mathlib/TwoStateMarkov.lean (two-state special case, no general finite-state kernel or stationary-law API); lean_leansearch 'finite state controlled Markov chain with a behaviour policy and stationary distribution' -> Mathlib has only ProbabilityTheory.Kernel.Invariant, no POMDP/experiment world. Hence local. SEARCH TRACE (certificate layer, unchanged): Grep '/Causalean/Mathlib' for 'interval|Interval|rational enclosure|Gaussian CDF|gaussianCdf|erf' -> the CertifiedContourIntervalArithmetic package (reused); Glob of Mathlib/Analysis/SpecialFunctions/Gaussian/ -> only GaussianIntegral / FourierTransform / PoissonSummation, i.e. Mathlib has NO rational-enclosure Gaussian CDF, confirming the local build on top of expScalar. ROUND-10 ADDENDUM (decision P15 part (6)): insulinGridFinite T : FiniteRewardModel T 90 4 4 is respelled onto the two-block finite trajectory (a Fin (T+1)-indexed state block plus T (action, reward-symbol) coordinates, built by the same epoch fold) and insulinGrid T := embed (insulinGridFinite T) is the world instance; every finite datum, every structural conjunct and the whole interval certificate are unchanged, and the world remains an instance of S1's own RawPomdpExperiment / policyKernel / stationaryLaw / targetValue machinery, so this block still only NARROWS S1 and adds no abstraction.

Definitions (10)

Definition P-1 verified CausalSmith.Stat.PomdpLatentOverlapMinimax.LatentOverlapClass $\mathcal M_T(t_0,\zeta,C):=\{(\mathcal X,\mathcal H,K,b,e,\mathcal L(\mathsf O_T)): T\in\mathbb N,\ |\mathcal X|<\infty,\ |\mathcal H|<\infty,\ |\mathcal A|=2,\text{ and }\mathrm{A}_{\mathrm{pomdp}},\mathrm{A}_{\mathrm{ign}},\mathrm{A}_{\mathrm{start}},\mathrm{A}_{\mathrm{reward}},\mathrm{A}_{\mathrm{act}},\mathrm{A}_{\mathrm{mix}},\mathrm{A}_{\mathrm{stat}}\text{ hold}\}$ , where the seven predicates are exactly $\mathrm{ass{:}pomdp\text{-}kernel}$ through $\mathrm{ass{:}latent\text{-}stationary\text{-}overlap}$ ; the union ranges over all finite cardinalities and all revealed policy pairs.
Definition P-2 verified CausalSmith.Stat.PomdpLatentOverlapMinimax.targetValue $\theta(K,e):=\sum_{s=(x,h)\in\mathcal S}d_e(s)g_e(s)=\sum_{s=(x,h)\in\mathcal S}d_e(s)\sum_{a\in\mathcal A}e(a\mid x)E_K[Y_t\mid S_t=s,A_t=a]$ .
Definition P-3 verified CausalSmith.Stat.PomdpLatentOverlapMinimax.minimaxRisk $R_T(t_0,\zeta,C):=\inf_{\widehat\theta\in\sigma(\mathsf O_T,b,e,t_0,\zeta,C)}\sup_{M\in\mathcal M_T(t_0,\zeta,C)}E_M[(\widehat\theta-\theta(K,e))^2]$ , where the infimum is over all measurable estimators and the supplied finite observed alphabets are included in the estimator's information.
Definition P-4 verified CausalSmith.Stat.PomdpLatentOverlapMinimax.phiwEstimator For $0\le k<T$ , define $\widehat\theta_k:=\operatorname{clip}_{[-1,1]}\{(T-k)^{-1}\sum_{t=k+1}^{T}Z_t(k)\}$ , with $Z_t(k)=Y_t\prod_{j=t-k}^{t}\rho_j$ and the zero-cell convention in the definition of $\rho_t$ .
Definition P-5 verified CausalSmith.Stat.PomdpLatentOverlapMinimax.historyDepth $k_T:=\min\{\lfloor T/2\rfloor,\max\{0,\lfloor\log(T)/(2\log(1/\alpha)+\log L)\rfloor\}\}$ .
Definition P-6 verified CausalSmith.Stat.PomdpLatentOverlapMinimax.signedDepthFamily For $C>1$ , $Q\ge1$ , and $v\in\{-1,+1\}$ , take $X_t$ constant, $H_t=(J_t,U_t)\in\{0,\ldots,Q\}\times\{-1,+1\}$ , $b(1)=1/L$ , and $e(1)=1$ . Draw a reset sign from $\Pr(U= u)=(1+u\varepsilon_C)/2$ . If $J_t=Q$ , set $J_{t+1}=0$ and draw a reset sign. If $J_t<Q$ , reset in the same way with probability $1-\alpha$ ; otherwise set $J_{t+1}=J_t+1$ , retaining $U_t$ when $A_t=1$ and replacing it by an independent fair sign when $A_t=0$ . Independently of the next-state draw conditional on $(J_t,U_t,A_t)$ , draw $Y_t\in\{-1,+1\}$ with $\Pr_v(Y_t=y\mid J_t,U_t,A_t)=\{1+yvc_0U_t\mathbf 1(J_t=Q)\}/2$ . Initialize from the behavior-stationary law $d_b(j,u)=w_j(1+u\varepsilon_C L^{-j})/2$ .
Definition P-7 verified CausalSmith.Stat.PomdpLatentOverlapMinimax.filterQuantities Let $\mathcal F_{t-1}^{\mathrm{obs}}:=\sigma(A_1,Y_1,\ldots,A_{t-1},Y_{t-1})$ , $\ell_t:=\min\{Q,t-1\}$ , $q_t^v:=\Pr_v(J_t=Q\mid\mathcal F_{t-1}^{\mathrm{obs}})$ , and $a_t^{(Q)}:=L^{-(Q-\ell_t)}\prod_{j=t-\ell_t}^{t-1}\mathbf 1\{A_j=1\}$ , with an empty product equal to one.
Definition P-9 verified CausalSmith.Stat.PomdpLatentOverlapMinimax.insulinGrid Let the state be $s=(g,d_1,d_2,x_1,x_2,a_1)$ in $\{50,100,150,200,250\}\times\{0,78\}^2\times\{0,31,850\}^2\times\{0,1\}$ , giving $360$ states. The diet coordinates $(d_1,d_2)$ form $H_t$ ; all other coordinates form $X_t$ . Put $b(1\mid X_t)=0.3$ and $e(1\mid X_t)=\mathbf 1\{g\ge125\}$ . With probability $1/2$ , draw the next full state uniformly. Otherwise draw $d_0\in\{0,78\}$ with probabilities $(0.8,0.2)$ , draw $x_0\in\{0,31,850\}$ with probabilities $(0.4,0.4,0.2)$ , draw $\xi\sim N(0,25)$ , set $g'=\operatorname{round}_{\{50,100,150,200,250\}}\operatorname{clip}_{[50,250]}(10+0.9g+0.1d_1+0.1d_2-0.01x_1-0.01x_2-2A_t-4a_1+\xi)$ , and shift the memories to $(g',d_0,d_1,x_0,x_1,A_t)$ . Set $Y_t=-1$ for $g\le70$ , $Y_t=-2/3$ for $g>150$ , $Y_t=-1/3$ for $70<g\le80$ or $120<g\le150$ , and $Y_t=0$ otherwise. This defines $\mathcal I_{\mathrm{grid}}$ .
Definition P-10 verified CausalSmith.Stat.PomdpLatentOverlapMinimax.overlapRadius For $C\ge1$ , define $q_C:=\frac{C-1}{C}.$
Definition P-11 verified CausalSmith.Stat.PomdpLatentOverlapMinimax.radiusAdaptiveDepth For $T\in\mathbb N$ and $C\ge1$ , define the radius-adaptive history depth $\kappa_T^{\mathrm{rad}}(C):= \begin{cases} 0, & Tq_C^2\le1,\\[3pt] \displaystyle \min\!\left\{\left\lfloor\frac T2\right\rfloor, \left\lfloor\frac{\log(Tq_C^2)}{2\log(1/\alpha)+\log L}\right\rfloor\right\}, & Tq_C^2>1. \end{cases}$

Assumptions (7)

Assumption A-1 verified CausalSmith.Stat.PomdpLatentOverlapMinimax.PomdpKernelLaw $\mathcal L(Y_t,X_{t+1},H_{t+1}\mid X_{1:t},H_{1:t},A_{1:t},Y_{1:t-1})=K(\cdot\mid X_t,H_t,A_t),\quad t=1,\ldots,T$ .
Assumption A-2 verified CausalSmith.Stat.PomdpLatentOverlapMinimax.SequentialIgnorability $\Pr(A_t=a\mid X_{1:t},H_{1:t},A_{1:t-1},Y_{1:t-1})=b(a\mid X_t),\quad (a,t)\in\mathcal A\times\{1,\ldots,T\}$ .
Assumption A-3 verified CausalSmith.Stat.PomdpLatentOverlapMinimax.StationaryStart $S_1\sim d_b$ .
Assumption A-4 verified CausalSmith.Stat.PomdpLatentOverlapMinimax.BoundedReward $\Pr(|Y_t|\le1)=1,\quad t=1,\ldots,T$ .
Assumption A-5 verified CausalSmith.Stat.PomdpLatentOverlapMinimax.PolicyOverlap $e(a\mid x)\le Lb(a\mid x),\quad (a,x)\in\mathcal A\times\mathcal X$ .
Assumption A-6 verified CausalSmith.Stat.PomdpLatentOverlapMinimax.UniformContraction $\|\nu P_p-\nu'P_p\|_{\mathrm{TV}}\le\alpha\|\nu-\nu'\|_{\mathrm{TV}},\quad (p,\nu,\nu')\in\{b,e\}\times\Delta(\mathcal S)^2$ .
Assumption A-7 verified CausalSmith.Stat.PomdpLatentOverlapMinimax.LatentStationaryOverlap $d_e(s)\le C d_b(s),\quad s\in\mathcal S$ .

Lemmas (40)

Lemma L-1 verified CausalSmith.Stat.PomdpLatentOverlapMinimax.phiw_upper For every fixed $t_0>0$ , $\zeta>0$ , and $C\ge1$ , there is $c^\star<\infty$ , independent of $T$ , $|\mathcal X|$ , and $|\mathcal H|$ , such that $\sup_{M\in\mathcal M_T(t_0,\zeta,C)}E_M[(\widehat\theta_{k_T}-\theta(K,e))^2]\le c^\star T^{-\beta}$ for all sufficiently large $T$ . In particular, the overlapping-history covariance sum is geometric and introduces no logarithmic loss.
Lemma L-2 verified CausalSmith.Stat.PomdpLatentOverlapMinimax.signedDepth_membership For every fixed $t_0>0$ , $\zeta>0$ , $C>1$ , $Q\ge1$ , and $v\in\{-1,+1\}$ , the signed-depth construction belongs to $\mathcal M_T(t_0,\zeta,C)$ for every positive horizon $T$ . Its policy-induced stationary laws satisfy $d_r(j,u)=w_j(1+u\varepsilon_C r^j)/2$ when action one is taken with probability $r$ , $\max_{j,u}d_e(j,u)/d_b(j,u)\le1+\varepsilon_C\le C$ , and $\theta(K,e)=vc_0\varepsilon_Cw_Q$ .
Lemma L-3 verified CausalSmith.Stat.PomdpLatentOverlapMinimax.observed_path_kl For the signed-depth alternatives, there exists $B_0<\infty$ , independent of $Q$ and $T$ , such that $\operatorname{KL}(\mathbb P_{Q,+1}^{\mathrm{obs},T}\,\|\,\mathbb P_{Q,-1}^{\mathrm{obs},T})\le B_0T\alpha^{2Q}L^{-Q}$ . More precisely, for every observed history, $E_v[Y_t\mid\mathcal F_{t-1}^{\mathrm{obs}}]=vc_0\varepsilon_Ca_t^{(Q)}q_t^v$ and $q_t^v\le K_0\alpha^Q$ for a finite $K_0$ depending only on $t_0$ , so the bound includes all stationary initial windows $t\le Q$ without revealing reset times.
Lemma L-4 verified CausalSmith.Stat.PomdpLatentOverlapMinimax.radius_sensitive_phiw For every $T\in\mathbb N$ , $C\ge1$ , $M\in\mathcal M_T(t_0,\zeta,C)$ , and integer $0\le k\le\lfloor T/2\rfloor$ , $E_M[(\widehat\theta_k-\theta(K,e))^2] \le4q_C^2\alpha^{2k} +\frac2T\left\{L^{k+1}\left(1+\frac4{L-1}\right) +\frac4{1-\alpha}\right\}.$ The bound is uniform in $C$ and in both state-space cardinalities.
Lemma L-5 verified CausalSmith.Stat.PomdpLatentOverlapMinimax.radius_explicit_observed_kl For every $C>1$ , $Q\ge1$ , and $T\in\mathbb N$ , the signed-depth alternatives satisfy $\frac{q_C}{2}\le\varepsilon_C\le q_C$ and $\operatorname{KL}(\mathbb P_{Q,+1}^{\mathrm{obs},T}\,\|\, \mathbb P_{Q,-1}^{\mathrm{obs},T}) \le \frac{4c_0^2K_0^2}{1-c_0^2} T\varepsilon_C^2\alpha^{2Q}L^{-Q}.$ The displayed coefficient depends only on $t_0$ , not on $C,Q,T$ , or any alphabet cardinality.
Lemma L-6 verified CausalSmith.Stat.PomdpLatentOverlapMinimax.uniform_parametric_floor For every fixed $t_0>0$ , $\zeta>0$ , every $C\ge1$ , and every $T\in\mathbb N$ , $R_T(t_0,\zeta,C)\ge\frac{e^{-1/4}}{64T}.$ This lower bound holds on a common singleton-state, equal-policy subexperiment and is therefore uniform in $C$ .
Lemma lem:phiw-raw-variance verified CausalSmith.Stat.PomdpLatentOverlapMinimax.variance_phiwRaw_le Let $T\ge1$ , let $M\in\mathcal M_T(t_0,\zeta,C)$ be a raw POMDP experiment in the model class of def:model-class, and write $L=e^{\zeta}$ and $\alpha=e^{-1/t_0}$ . Then, for every history depth $k\le\lfloor T/2\rfloor$ , the unclipped partial-history average $\frac{1}{T-k}\sum_{t=k}^{T-1}Z_t(k)$ formed from the depth- $k$ scores $Z_t(k)$ of def:phiw-estimator has, under the observed trajectory law of $M$ , variance $\operatorname{Var}_M\!\left(\frac{1}{T-k}\sum_{t=k}^{T-1}Z_t(k)\right) \le \frac{2}{T}\left\{L^{k+1}\left(1+\frac{4}{L-1}\right)+\frac{4}{1-\alpha}\right\}.$ The constant comes from the exact finite covariance assembly: each diagonal second moment is at most $L^{k+1}$ , the covariances with overlapping windows sum to at most $2L^{k+1}/(L-1)$ per index, and the covariances with separated windows sum to at most $2/(1-\alpha)$ per index, so the geometric aggregate leaves no logarithmic factor.
Lemma lem:phiw-score-stationary-mean verified CausalSmith.Stat.PomdpLatentOverlapMinimax.integral_phiwScore_eq_init_targetIter Let $M\in\mathcal M_T(t_0,\zeta,C)$ be a raw POMDP experiment in the model class of def:model-class, let $k$ be a history depth, and let $t$ be a time index with $k\le t$ , so that the depth- $k$ window of the score $Z_t(k)$ of def:phiw-estimator is mature. Then, under the observed trajectory law of $M$ , $\mathbb E_M[Z_t(k)]=\sum_{s}d_b(s)\,(P_e^kg_e)(s),$ where $d_b$ is the behavior stationary law selected for $M$ , $P_e$ is the target-policy transition operator, and $g_e$ is the target-policy reward regression. The right-hand side does not depend on $t$ : conditioning the observed window backwards, the one-step policy ratios cancel the behavior action probabilities, which turns the observed score into the $k$ -step target-kernel iterate of $g_e$ integrated against the stationary behavior law.
Lemma lem:refined-terminal-posterior verified CausalSmith.Stat.PomdpLatentOverlapMinimax.signedDepthRawPosterior_le_refined Let $t_0>0$ , $\zeta>0$ , and $C>1$ , and write $\alpha=e^{-1/t_0}$ and $c_0=(1-\alpha)/4$ . Fix a horizon $N$ , a terminal depth $Q$ , an alternative index $v\in\{-1,+1\}$ , and an observed prefix $o=(a_1,y_1,\ldots,a_k,y_k)$ of arbitrary length $k$ , and let $\pi_{N,Q,v}(o)=\Pr_v(J_{k+1}=Q\mid O_{1:k}=o)$ be the raw terminal-depth posterior of the signed-depth family of def:signed-depth-family, that is, the unnormalized filter mass carried at terminal depth divided by the total unnormalized filter mass of the prefix. Then $\pi_{N,Q,v}(o) \le \frac{\alpha^Q}{\left\{1-c_0\left(\dfrac{\alpha}{1-c_0}\right)^Q\right\}^Q}.$ This is the bootstrapped refinement of the crude estimate $\pi_{N,Q,v}\le\{\alpha/(1-c_0)\}^Q$ : applying the crude estimate at every strict prefix of the final window lower-bounds the product of one-step correction factors by $\{1-c_0(\alpha/(1-c_0))^Q\}^Q$ , and substituting that lower bound into the window identity for the posterior gives the displayed inequality, in both the mature case $Q\le k$ and the initial-window case $k<Q$ .
Lemma lem:finite-signed-depth-kl verified CausalSmith.Stat.PomdpLatentOverlapMinimax.signedDepth_finiteObs_klDiv_le Let $t_0>0$ , $\zeta>0$ , $C>1$ , and $K_0>0$ , and write $\alpha=e^{-1/t_0}$ , $L=e^{\zeta}$ , $c_0=(1-\alpha)/4$ , and $\varepsilon_C=\min\{(C-1)/2,1/2\}$ . Fix a terminal depth $Q$ and assume the uniform terminal-posterior bound $q_t^v(o)\le K_0\alpha^Q$ of def:filter-quantities, valid for every horizon, every alternative index $v\in\{-1,+1\}$ , every time $t$ , and every observed prefix $o$ . Then, for every horizon $T$ , the finite observed-word laws $\mathbb P_{Q,v}^{\mathrm{fin},T}$ of the signed-depth alternatives of def:signed-depth-family, taken before the reward symbols are decoded as real rewards, satisfy $\operatorname{KL}\!\left(\mathbb P_{Q,+1}^{\mathrm{fin},T}\middle\|\mathbb P_{Q,-1}^{\mathrm{fin},T}\right) \le \frac{4c_0^2K_0^2}{1-c_0^2}\,T\,\varepsilon_C^2\alpha^{2Q}L^{-Q}.$ The estimate iterates the finite chain rule for relative entropy over the horizon. Each one-step conditional divergence is controlled by the chi-square quantity built from the conditional reward means, $(m_t^{+1}-m_t^{-1})^2/\{1-(m_t^{-1})^2\}\le4c_0^2K_0^2(1-c_0^2)^{-1}\varepsilon_C^2\alpha^{2Q}(a_t^{(Q)})^2$ , and the rare-action second moment of the factors $a_t^{(Q)}$ of def:filter-quantities satisfies $\sum_o\Pr_{Q,+1}\{O_{1:k}=o\}\,(a_{k+1}^{(Q)}(o))^2\le L^{-Q}$ at every horizon $k$ .
Lemma lem:insulin-bias-certificate verified CausalSmith.Stat.PomdpLatentOverlapMinimax.insulinGrid_bias_certificate For every horizon $T$ , the immediate-weighting bias diagnostic $\Delta_{\mathrm{imm}}(T)$ of the refreshed finite insulin adaptation $\mathcal I_{\mathrm{grid}}$ of def:insulin-grid, that is, the difference between the behavior-stationary mean of the immediately reweighted reward and the target value $\theta(K,e)$ of def:target-value, obeys the strict two-sided enclosure $-\frac{6}{1000}<\Delta_{\mathrm{imm}}(T)<-\frac{58}{10000}.$ The enclosure is a rational certificate. Both policy-induced transition matrices of $\mathcal I_{\mathrm{grid}}$ contract in $L^1$ with modulus $1/2$ ; chunked rational interval iterates of these matrices, combined with an interval enclosure of the state reward vector, enclose the behavior-stationary and target-stationary reward expectations, and the interval difference of those two enclosures lies inside the displayed rational window.
Lemma lem:insulin-policy-contraction verified CausalSmith.Stat.PomdpLatentOverlapMinimax.insulinPolicy_contraction Fix a horizon $T$ and let $\mathcal I_{\mathrm{grid}}$ be the refreshed finite insulin adaptation of def:insulin-grid, with $360$ -point joint state space $\mathsf S_{\mathrm{grid}}$ , observed coordinate map $s\mapsto x(s)$ onto the $90$ -point observed alphabet, and transition kernel $K_T$ . For any policy $p$ assigning to each observed state a probability law on $\{0,1\}$ , write $P_p^{(T)}(s,s')=\sum_{a\in\{0,1\}}p(a\mid x(s))\,K_T\{(q,s''):\ s''=s'\mid s,a\}$ for the induced transition matrix on $\mathsf S_{\mathrm{grid}}$ . Then $P_p^{(T)}$ contracts total variation with modulus $1/2$ : for all probability vectors $\nu,\nu'$ on $\mathsf S_{\mathrm{grid}}$ , $\|\nu P_p^{(T)}-\nu'P_p^{(T)}\|_{\mathrm{TV}}\le\tfrac12\|\nu-\nu'\|_{\mathrm{TV}},$ equivalently $\|\nu P_p^{(T)}-\nu'P_p^{(T)}\|_1\le\tfrac12\|\nu-\nu'\|_1$ for the $\ell^1$ norm $\|\cdot\|_1=2\|\cdot\|_{\mathrm{TV}}$ . The modulus $1/2$ is exactly the refresh probability of $\mathcal I_{\mathrm{grid}}$ : the kernel splits as $P_p^{(T)}=\tfrac12u+\tfrac12\Gamma_p$ with uniform refresh law $u(s')=1/360$ and residual part $\Gamma_p$ a Markov kernel, so the common refresh term cancels in $\nu P_p^{(T)}-\nu'P_p^{(T)}$ and the residual application is nonexpansive in total variation.
Lemma lem:insulin-stationary-law verified CausalSmith.Stat.PomdpLatentOverlapMinimax.insulinStationaryLaw_isStationary Fix a horizon $T$ and let $p$ be a policy of the refreshed finite insulin adaptation $\mathcal I_{\mathrm{grid}}$ of def:insulin-grid, that is, an assignment of a probability law on $\{0,1\}$ to each observed state. Assume the induced transition matrix $P_p^{(T)}$ on the $360$ -point joint state space $\mathsf S_{\mathrm{grid}}$ contracts total variation with modulus $1/2$ , that is, $\|\nu P_p^{(T)}-\nu'P_p^{(T)}\|_{\mathrm{TV}}\le\frac12\|\nu-\nu'\|_{\mathrm{TV}}$ for all probability vectors $\nu,\nu'$ . Then the stationary law $d_p$ selected for $P_p^{(T)}$ is a genuine stationary probability vector: its entries are nonnegative and sum to one, and $d_p(s')=\sum_{s\in\mathsf S_{\mathrm{grid}}}d_p(s)P_p^{(T)}(s,s'),\qquad s'\in\mathsf S_{\mathrm{grid}} .$ The selection is legitimate because a contraction of modulus $1/2<1$ applied repeatedly to the initial law of $\mathcal I_{\mathrm{grid}}$ produces a Cauchy sequence of probability vectors whose limit is a fixed point, so the selector ranges over a nonempty set. Since both policy-induced matrices of $\mathcal I_{\mathrm{grid}}$ contract with modulus $1/2$ , the hypothesis holds for the behavior policy $b$ and for the target policy $e$ .
Lemma lem:insulin-target-reward-regression verified CausalSmith.Stat.PomdpLatentOverlapMinimax.insulinTargetRewardRegression Fix a horizon $T$ and let $\mathcal I_{\mathrm{grid}}$ be the refreshed finite insulin adaptation of def:insulin-grid, with transition kernel $K_T$ , target policy $e$ , observed coordinate map $s\mapsto x(s)$ , and grid states $s=(g,d_1,d_2,x_1,x_2,a_1)$ . Write $p=(p_1,p_2)$ for a pair drawn by $K_T$ , whose first coordinate $p_1$ is the numeric reward and whose second coordinate is the successor state, and set $r_{\mathrm{ins}}(s)= \begin{cases} -1,&g\le70,\\ -2/3,&g>150,\\ -1/3,&70<g\le80\ \text{or}\ 120<g\le150,\\ 0,&\text{otherwise}, \end{cases}$ so that $r_{\mathrm{ins}}(s)$ depends on $s$ only through its glucose coordinate $g$ . Then the target-policy reward regression of $\mathcal I_{\mathrm{grid}}$ , which enters the target value $\theta(K_T,e)$ of def:target-value, is the reward vector itself: $\sum_{a\in\{0,1\}}e(a\mid x(s))\int p_1\,K_T(dp\mid s,a)=r_{\mathrm{ins}}(s)$ for every grid state $s$ . Indeed $\int p_1\,K_T(dp\mid s,a)=r_{\mathrm{ins}}(s)$ for both actions, because the reward emitted by $K_T$ is a deterministic function of the current glucose coordinate, and the two target probabilities $e(0\mid x(s))$ and $e(1\mid x(s))$ sum to one.
Lemma lem:insulin-immediate-reward-regression verified CausalSmith.Stat.PomdpLatentOverlapMinimax.insulinImmediateRewardRegression Fix a horizon $T$ and let $\mathcal I_{\mathrm{grid}}$ be the refreshed finite insulin adaptation of def:insulin-grid, with transition kernel $K_T$ , behavior policy $b$ , target policy $e$ , and observed coordinate map $s\mapsto x(s)$ . Let $\rho(x,a)=e(a\mid x)/b(a\mid x)$ when $b(a\mid x)\ne0$ and $\rho(x,a)=0$ otherwise, and let $r_{\mathrm{ins}}$ be the insulin reward vector $r_{\mathrm{ins}}(s)= \begin{cases} -1,&g\le70,\\ -2/3,&g>150,\\ -1/3,&70<g\le80\ \text{or}\ 120<g\le150,\\ 0,&\text{otherwise}, \end{cases}$ Then the immediate weighted reward regression of $\mathcal I_{\mathrm{grid}}$ is again the reward vector: $\sum_{a\in\{0,1\}}b(a\mid x(s))\,\rho(x(s),a)\int p_1\,K_T(dp\mid s,a)=r_{\mathrm{ins}}(s)$ for every grid state $s$ , where $p_1$ is the numeric reward coordinate of a reward/state pair $p$ drawn by $K_T$ . The behavior probabilities $b(1\mid x)=3/10$ and $b(0\mid x)=7/10$ are strictly positive, so the ratio convention gives $b(a\mid x(s))\rho(x(s),a)=e(a\mid x(s))$ for both actions; since $\int p_1\,K_T(dp\mid s,a)=r_{\mathrm{ins}}(s)$ does not depend on $a$ , summing over the two actions and using $e(0\mid x(s))+e(1\mid x(s))=1$ gives the identity.
Lemma lem:phiw-window-second-moment verified CausalSmith.Stat.PomdpLatentOverlapMinimax.integral_sq_phiwScore_le Let $T\ge1$ and $k\in\mathbb N$ , and let $M$ be a raw POMDP experiment on finite observed and hidden alphabets satisfying ass:pomdp-kernel, ass:sequential-ignorability, ass:bounded-reward, and ass:policy-overlap with a constant $L\ge1$ . Write $\mathbb E_{\mathrm{obs}}$ and $\operatorname{Cov}_{\mathrm{obs}}$ for expectation and covariance under the observed trajectory law of $M$ , and let $Z_t(k)$ be the depth- $k$ scores of def:phiw-estimator. Then, for every index $t$ with $k\le t\le T-1$ , $\mathbb E_{\mathrm{obs}}\bigl[Z_t(k)^2\bigr]\le L^{k+1}.$ The bound is the diagonal term of the variance assembly. Bounded rewards give $|Y_t|\le1$ , so $Z_t(k)^2\le L^{k+1}\Gamma_t^{(k)}$ pointwise, where $\Gamma_t^{(k)}=\prod_{j=t-k}^{t}\rho_j$ is the depth- $k$ window likelihood ratio, one factor of the overlap envelope $L^{k+1}$ cancelling against the ratio block; the remaining window product has observed mean $\mathbb E_{\mathrm{obs}}\Gamma_t^{(k)}=1$ .
Lemma lem:phiw-overlapping-window-covariance verified CausalSmith.Stat.PomdpLatentOverlapMinimax.abs_covariance_phiwScore_le_overlapDecay Let $T\ge1$ and $k\in\mathbb N$ , and let $M$ be a raw POMDP experiment on finite observed and hidden alphabets satisfying ass:pomdp-kernel, ass:sequential-ignorability, ass:bounded-reward, and ass:policy-overlap with a constant $L\ge1$ . Write $\mathbb E_{\mathrm{obs}}$ and $\operatorname{Cov}_{\mathrm{obs}}$ for expectation and covariance under the observed trajectory law of $M$ , and let $Z_t(k)$ be the depth- $k$ scores of def:phiw-estimator. Then, for all indices $t,u$ with $k\le t\le T-1$ and $u=t+h\le T-1$ for an overlap lag $h$ with $1\le h\le k$ , $\bigl|\operatorname{Cov}_{\mathrm{obs}}\bigl(Z_t(k),Z_u(k)\bigr)\bigr|\le2L^{k+1-h}.$ This is the overlapping-window estimate: the two windows share $k+1-h$ epochs, the union of the two likelihood-ratio blocks has observed mean one, the product envelope leaves $L^{k+1-h}$ for the mixed absolute moment, and each of the two first absolute moments is at most one, so the crude covariance bound $|\operatorname{Cov}_{\mathrm{obs}}(U,V)|\le\mathbb E_{\mathrm{obs}}|UV|+\mathbb E_{\mathrm{obs}}|U|\,\mathbb E_{\mathrm{obs}}|V|$ gives $L^{k+1-h}+1\le2L^{k+1-h}$ .
Lemma lem:phiw-separated-window-covariance verified CausalSmith.Stat.PomdpLatentOverlapMinimax.abs_covariance_phiwScore_le_disjoint_sharp Let $T\ge1$ and $k\in\mathbb N$ , let $M\in\mathcal M_T(t_0,\zeta,C)$ be a raw POMDP experiment in the model class of def:model-class, and write $\alpha=e^{-1/t_0}$ . Then, for all indices $t,u$ with $k\le t\le T-1$ and $u=t+h\le T-1$ for a separation lag $h\ge k+1$ , the depth- $k$ scores $Z_t(k)$ of def:phiw-estimator satisfy, under the observed trajectory law of $M$ , $\bigl|\operatorname{Cov}_{\mathrm{obs}}\bigl(Z_t(k),Z_u(k)\bigr)\bigr|\le2\alpha^{h-k-1}.$ The exponent $h-k-1$ counts the transitions strictly between the two disjoint windows, and only the contraction of ass:uniform-contraction is spent: conditioning the later score on the joint state reached just after the earlier window expresses it as a bounded function of that state with oscillation at most $2\alpha^{h-k-1}$ , the earlier score has first absolute moment at most one, and centering the bounded function against its mean yields the displayed covariance bound.
Lemma lem:phiw-future-covariance-sum verified CausalSmith.Stat.PomdpLatentOverlapMinimax.sum_abs_covariance_future_le Let $T\ge1$ and $k\in\mathbb N$ , let $M\in\mathcal M_T(t_0,\zeta,C)$ be a raw POMDP experiment in the model class of def:model-class, and write $L=e^{\zeta}$ , $\alpha=e^{-1/t_0}$ , and $\mathscr T_k=\{t:k\le t\le T-1\}$ . Then, for every $t\in\mathscr T_k$ , the depth- $k$ scores of def:phiw-estimator satisfy, under the observed trajectory law of $M$ , $\sum_{\substack{u\in\mathscr T_k\\ t<u}}\bigl|\operatorname{Cov}_{\mathrm{obs}}\bigl(Z_t(k),Z_u(k)\bigr)\bigr| \le\frac{2L^{k+1}}{L-1}+\frac{2}{1-\alpha}.$ The future indices split into the overlapping block $\{u:1\le u-t\le k\}$ and the separated block $\{u:u-t\ge k+1\}$ , which are disjoint and exhaust $\{u\in\mathscr T_k:t<u\}$ . On the overlapping block the lag map $u\mapsto k-(u-t)$ is injective into $\{0,\ldots,k-1\}$ , so those terms are dominated by $2L\sum_{j=0}^{k-1}L^{j}\le2L^{k+1}/(L-1)$ ; on the separated block the lag map $u\mapsto u-t-k-1$ is injective into $\{0,\ldots,T-1\}$ , so those terms are dominated by $2\sum_{j=0}^{T-1}\alpha^{j}\le2/(1-\alpha)$ . Both dominations reindex finite sums of nonnegative terms along an injection, so no term is counted twice and the geometric partial sums bound the aggregate.
Lemma lem:signed-depth-constant-policy-contraction verified CausalSmith.Stat.PomdpLatentOverlapMinimax.signedDepth_uniformContraction_actionOne Let $t_0>0$ , $\zeta>0$ , $C\ge1$ and $Q\ge1$ , write $\alpha=e^{-1/t_0}$ , and fix an alternative index $v\in\{-1,+1\}$ . For $r\in[0,1]$ let $a_r$ be the constant policy with $\Pr(A_t=1)=r$ , and let $P^{(r)}$ be the transition kernel induced by $a_r$ on the joint state space $\mathcal S_Q=\{0\}\times(\{0,\ldots,Q\}\times\{-1,+1\})$ of the signed-depth alternative of def:signed-depth-family with terminal depth $Q$ and index $v$ . Then $P^{(r)}$ contracts total variation with modulus $\alpha$ : for every $r\in[0,1]$ and all probability vectors $\mu,\mu'$ on $\mathcal S_Q$ , $\|\mu P^{(r)}-\mu'P^{(r)}\|_{\mathrm{TV}}\le\alpha\|\mu-\mu'\|_{\mathrm{TV}} .$ The bound is uniform in the policy parameter $r$ and in particular covers the behavior policy $r=L^{-1}$ and the target policy $r=1$ of the signed-depth family.
Lemma lem:signed-depth-stationary-law verified CausalSmith.Stat.PomdpLatentOverlapMinimax.signedDepth_stationaryLaw_apply Let $t_0>0$ , $\zeta>0$ , $C\ge1$ , $Q\ge1$ , $v\in\{-1,+1\}$ and $r\in[0,1]$ , write $\alpha=e^{-1/t_0}$ and $\varepsilon_C=\min\{(C-1)/2,1/2\}$ , and let $w_j=(1-\alpha)\alpha^{j}/(1-\alpha^{Q+1})$ , $j\in\{0,\ldots,Q\}$ , be the truncated geometric depth masses. Let $P^{(r)}$ be the constant-policy kernel of the signed-depth alternative of def:signed-depth-family with terminal depth $Q$ and index $v$ , and let $d_r$ be the stationary law selected for $P^{(r)}$ . Then $d_r$ is given in closed form: for every hidden state $h$ with depth coordinate $j(h)$ and sign coordinate $u(h)$ , $d_r(0,h)=w_{j(h)}\,\frac{1+u(h)\varepsilon_Cr^{j(h)}}{2}.$ This identifies the abstract selector with the explicit law: the displayed right-hand side is a stationary probability vector for $P^{(r)}$ , the selector is stationary as soon as some stationary probability law exists, and a kernel contracting total variation with modulus $\alpha<1$ has at most one stationary probability vector, so the two coincide.
Lemma lem:signed-depth-bootstrap-factor verified CausalSmith.Stat.PomdpLatentOverlapMinimax.exists_uniform_signedDepth_bootstrap_factor Let $t_0>0$ and write $\alpha=e^{-1/t_0}$ , $c_0=(1-\alpha)/4$ and $\eta_0=\alpha/(1-c_0)$ . Then there is a constant $K_0>0$ , depending only on $t_0$ , such that $\frac{1}{\bigl(1-c_0\eta_0^{Q}\bigr)^{Q}}\le K_0\qquad\text{for every }Q\in\mathbb N.$ Finiteness of each individual factor is not enough for this uniform bound, because the exponent grows with $Q$ . The estimate holds because $0<\eta_0<1$ , so the sequence $Q\eta_0^{Q}$ is bounded; taking logarithms and using $-\log(1-x)\le2x$ for $0\le x\le1/2$ gives $-Q\log(1-c_0\eta_0^{Q})\le2c_0Q\eta_0^{Q}$ , which is bounded uniformly in $Q$ , and exponentiating produces a finite constant $K_0$ .
Lemma lem:certified-stationary-reward-enclosure verified Causalean.Mathlib.Probability.CertifiedFiniteMarkovExpectation.stationaryRewardInterval_sound_of_chunked Let $\mathsf S$ be a finite set and let $P$ be a stochastic matrix on $\mathsf S$ whose entries are enclosed by rational intervals, $P(s,s')\in\mathfrak A(s,s')$ for all $s,s'\in\mathsf S$ . Intervals are rational intervals $[\ell,u]$ with $\ell\le u$ , combined by exact interval arithmetic: $[a,b]+[c,d]=[a+c,b+d]$ , $[a,b]\ominus[c,d]=[a-d,b-c]$ , and $[a,b]\cdot[c,d]$ is the interval from the smallest to the largest of the four endpoint products; write $\ell(\mathfrak r)$ , $u(\mathfrak r)$ for the endpoints of $\mathfrak r$ and $|\mathfrak r|_{\max}=\max\{|\ell(\mathfrak r)|,|u(\mathfrak r)|\}$ . Suppose the following certificate data are given. (i) An exact rational probability vector $\mu_0$ on $\mathsf S$ , a number $m\in\mathbb N$ , and interval rows $\mathfrak T_0,\ldots,\mathfrak T_{m+1}$ on $\mathsf S$ with $\mu_0(s)\in\mathfrak T_0(s)$ for every $s$ . (ii) A chunked recurrence check: for every $k\le m$ and every output state $s'$ , a list of pairwise disjoint blocks $\mathcal G_1,\ldots,\mathcal G_J\subseteq\mathsf S$ covering $\mathsf S$ , each of at most a fixed positive number of states, block intervals $\Xi_1(s'),\ldots,\Xi_J(s')$ with $\sum_{s\in\mathcal G_j}\mathfrak T_k(s)\cdot\mathfrak A(s,s')\subseteq\Xi_j(s')$ , and partial-sum intervals $\Sigma_1,\ldots,\Sigma_{J+1}$ with $0\in\Sigma_{J+1}$ , $\Xi_j(s')+\Sigma_{j+1}\subseteq\Sigma_j$ for $j=J,\ldots,1$ , and $\Sigma_1=\mathfrak T_{k+1}(s')$ . (iii) A reward vector $r:\mathsf S\to\mathbb R$ with rational intervals $\mathfrak r(s)\ni r(s)$ and a rational bound $B\ge0$ such that $|\mathfrak r(s)|_{\max}\le B$ for every $s$ . (iv) A rational coefficient $0\le\rho<1$ with $\|\nu P-\nu'P\|_1\le\rho\|\nu-\nu'\|_1$ for all probability vectors $\nu,\nu'$ on $\mathsf S$ , and a probability vector $\pi$ with $\pi P=\pi$ . Put $\delta=\sum_{s\in\mathsf S}\bigl|\mathfrak T_m(s)\ominus\mathfrak T_{m+1}(s)\bigr|_{\max}, \qquad \mathfrak E=\sum_{s\in\mathsf S}\mathfrak T_m(s)\cdot\mathfrak r(s), \qquad \mathfrak J=\Bigl[\ell(\mathfrak E)-\frac{B\delta}{1-\rho},\ u(\mathfrak E)+\frac{B\delta}{1-\rho}\Bigr].$ Then $\sum_{s\in\mathsf S}\pi(s)\,r(s)\in\mathfrak J .$ The enclosure holds because the checks in (i)--(ii) give, by induction on $k$ , $\mu_0P^k\in\mathfrak T_k$ coordinatewise; the terminal and successor rows then give $\|\mu_0P^m-\mu_0P^{m+1}\|_1\le\delta$ ; contraction and stationarity give $\|\mu_0P^m-\pi\|_1\le\delta+\rho\|\mu_0P^m-\pi\|_1$ , hence $\|\mu_0P^m-\pi\|_1\le\delta/(1-\rho)$ ; and $|r(s)|\le B$ together with $\sum_s\mu_0P^m(s)r(s)\in\mathfrak E$ converts this $\ell^1$ radius into the reward radius $B\delta/(1-\rho)$ .
Lemma lem:signed-depth-terminal-sign-invariant verified CausalSmith.Stat.PomdpLatentOverlapMinimax.signedDepth_terminalMass_invariant Let $t_0>0$ , $\zeta>0$ , $C>1$ , $Q\in\mathbb N$ , a horizon $N$ , and $v\in\{-1,+1\}$ , and consider the finite signed-depth model of def:signed-depth-family with $L=e^{\zeta}$ , $\varepsilon_C=\min\{(C-1)/2,1/2\}$ , initial hidden-state weights $p_v$ , behavior policy $b$ , and reward/hidden-state kernel $K_v$ . For a hidden state $h\in\{0,\ldots,Q\}\times\{-1,+1\}$ write $J(h)$ and $U(h)$ for its depth and sign coordinates. For a prefix length $k\in\mathbb N$ and an observed action/reward prefix $o=((a_1,y_1),\ldots,(a_k,y_k))$ with $a_i\in\{0,1\}$ and $y_i\in\{-1,+1\}$ , define the unnormalized filter mass $\lambda_v(h;o) = \sum_{h_1,\ldots,h_k} p_v(h_1)\prod_{i=1}^{k} b(a_i)\,K_v\bigl(y_i,h_{i+1}\mid h_i,a_i\bigr), \qquad h_{k+1}=h,$ that is, the probability under alternative $v$ that the first $k$ action/reward pairs equal $o$ and the hidden state after them equals $h$ . With $\ell=\min\{Q,k\}$ put $a^{(Q)}(o)=L^{-(Q-\ell)}\prod_{i=k-\ell+1}^{k}\mathbf 1\{a_i=1\},$ the empty product being one; for $k=t-1$ this is the rare-action-history factor $a_t^{(Q)}(o)$ of def:filter-quantities. Then the signed terminal-depth filter moment and the unsigned terminal-depth filter mass satisfy $\sum_{h:\,J(h)=Q}U(h)\,\lambda_v(h;o) = \varepsilon_C\,a^{(Q)}(o)\sum_{h:\,J(h)=Q}\lambda_v(h;o).$ This is the depth- $Q$ case of a depthwise invariant proved by induction on $k$ : at every depth $j$ the signed mass equals $\varepsilon_C$ times the depth- $j$ action-history factor $L^{-(j-\min\{j,k\})}\prod_{i=k-\min\{j,k\}+1}^{k}\mathbf 1\{a_i=1\}$ times the unsigned mass. At the empty prefix the initial weights give ratio $\varepsilon_C L^{-j}$ ; a reset to depth zero draws a fresh sign with ratio $\varepsilon_C$ ; an advance from depth $j-1$ to $j$ keeps the sign after action one and splits evenly over signs after action zero, so it preserves the ratio along action one and annihilates the signed mass along action zero; and the behavior and reward factors are common to both signs at non-terminal depths.
Lemma lem:phiw-separated-window-peeling verified CausalSmith.Stat.PomdpLatentOverlapMinimax.integral_phiwScore_mul_eq_futureFunction Let $M$ be a raw POMDP experiment satisfying ass:pomdp-kernel, ass:sequential-ignorability, ass:bounded-reward, and ass:policy-overlap with an overlap constant $L\ge1$ ; let $\mathbb P_M$ be its full trajectory law, $O(\tau)$ the observed trajectory of a full trajectory $\tau$ , and $\mathbb P_M^{\mathrm{obs}}$ the observed law, the image of $\mathbb P_M$ under $O$ . For $p\in\{b,e\}$ let $P_p(s,s')=\sum_a p(a\mid x(s))\,K(\{S'=s'\}\mid s,a)$ be the reward-marginalized policy kernel, acting on functions by $(P_pf)(s)=\sum_{s'}P_p(s,s')f(s')$ with $P_p^0f=f$ and $P_p^{m+1}f=P_p(P_p^mf)$ , and let $g_e(s)=\sum_a e(a\mid x(s))\int y\,K(dy\times\mathcal S\mid s,a)$ be the target-policy reward regression, in which the reward coordinate is integrated against the whole one-step kernel row (the successor-state coordinate integrated out). For $\gamma,k\in\mathbb N$ put $\Phi_{\gamma,k}:=P_b^\gamma P_e^k g_e$ . Let $t$ be an epoch and $u:=t+1+\gamma+k$ with $u\le T-1$ , and let $S_{t+1}(\tau)$ be the joint state reached by the epoch- $t$ transition. Then the depth- $k$ scores of def:phiw-estimator satisfy $\int Z_u(k)\,d\mathbb P_M^{\mathrm{obs}}=\int \Phi_{\gamma,k}(S_{t+1}(\tau))\,d\mathbb P_M(\tau),$ and, if in addition $k\le t$ , $\int Z_t(k)Z_u(k)\,d\mathbb P_M^{\mathrm{obs}}=\int Z_t(k)(O(\tau))\,\Phi_{\gamma,k}(S_{t+1}(\tau))\,d\mathbb P_M(\tau).$ Both identities are obtained by pulling the observed integrals back to $\mathbb P_M$ and peeling the later window backwards from epoch $u$ : conditionally on the history through the state at epoch $u$ , sequential ignorability and the zero-cell convention (overlap forces $e(a\mid x)=0$ wherever $b(a\mid x)=0$ ) turn $\rho_uY_u$ into $g_e(S_u)$ ; each of the remaining $k$ ratios $\rho_{u-1},\ldots,\rho_{t+1+\gamma}$ converts one behavior transition into one application of $P_e$ ; the $\gamma$ unweighted transitions between epochs $t+1$ and $t+1+\gamma$ contribute $P_b^\gamma$ ; and the earlier mature score $Z_t(k)$ , a function of the history through epoch $t$ , is carried unchanged through the conditioning. Bounded rewards and $|\rho_j|\le L$ make every integrand integrable.
Lemma lem:behavior-support-operator-congruence verified CausalSmith.Stat.PomdpLatentOverlapMinimax.markovOperatorIter_congr_on_behaviorSupport Let $M\in\mathcal M_T(t_0,\zeta,C)$ be a raw POMDP experiment in the model class of def:model-class, let $d_b$ be its selected behavior stationary law, and for $p\in\{b,e\}$ let $P_p(s,s')=\sum_a p(a\mid x(s))\,K(\{S'=s'\}\mid s,a)$ be the reward-marginalized policy kernel, acting on functions by $(P_pf)(s)=\sum_{s'}P_p(s,s')f(s')$ with $P_p^0f=f$ and $P_p^{m+1}f=P_p(P_p^mf)$ . Let $f,g:\mathcal S\to\mathbb R$ satisfy $f(s)=g(s)$ for every state $s$ with $d_b(s)>0$ . Then for every $p\in\{b,e\}$ , every $m\in\mathbb N$ , and every state $s$ with $d_b(s)>0$ , $(P_p^mf)(s)=(P_p^mg)(s).$ The proof is by induction on $m$ and rests on closure of the behavior support under the nonzero rows of both kernels: if $d_b(s)>0$ and $P_p(s,x)>0$ , then $P_b(s,x)>0$ (immediate for $p=b$ ; for $p=e$ , every action $a$ with $e(a\mid x(s))>0$ has $b(a\mid x(s))\ge e(a\mid x(s))/L>0$ by ass:policy-overlap, so a positive $P_e$ -row entry forces a positive $P_b$ -row entry), and stationarity gives $d_b(x)=\sum_y d_b(y)P_b(y,x)\ge d_b(s)P_b(s,x)>0$ . In the inductive step $(P_p^{m+1}f)(s)=\sum_xP_p(s,x)(P_p^mf)(x)$ ; summands with $P_p(s,x)=0$ vanish for both functions, and every remaining $x$ has $d_b(x)>0$ , where the induction hypothesis gives $(P_p^mf)(x)=(P_p^mg)(x)$ .
Lemma lem:signed-depth-conditional-reward-mean verified CausalSmith.Stat.PomdpLatentOverlapMinimax.conditionalRewardMean_signedDepth_all Let $t_0>0$ , $\zeta>0$ , $C>1$ , $Q\in\mathbb N$ , a horizon $N$ , and $v\in\{-1,+1\}$ , and consider the finite signed-depth model of def:signed-depth-family over epochs $1,\ldots,N$ , with rewards $Y_t\in\{-1,+1\}$ , $\alpha=e^{-1/t_0}$ , $c_0=(1-\alpha)/4$ , and $\varepsilon_C=\min\{(C-1)/2,1/2\}$ . For an epoch $t\le N$ and an observed action/reward prefix $o=((a_1,y_1),\ldots,(a_{t-1},y_{t-1}))$ , let $m_v(o)=\Pr_v\{(A_i,Y_i)_{i<t}=o\}$ be its prefix mass and $m_v^{Q}(o)=\Pr_v\{(A_i,Y_i)_{i<t}=o,\ J_t=Q\}$ its joint mass with terminal depth at epoch $t$ , and define the conditional reward mean and the terminal posterior as the totalized ratios $\mathbb E_v[Y_t\mid\mathcal F_{t-1}^{\mathrm{obs}}=o]:=\frac{\mathbb E_v\bigl[Y_t\,\mathbf 1\{(A_i,Y_i)_{i<t}=o\}\bigr]}{m_v(o)}, \qquad q_t^v(o):=\frac{m_v^{Q}(o)}{m_v(o)},$ where a ratio with zero denominator is set equal to $0$ ; on prefixes of positive mass these are the conditional mean and the posterior $q_t^v$ of def:filter-quantities, and on prefixes of zero mass both equal $0$ . Then, for every epoch $t\le N$ and every such prefix $o$ , $\mathbb E_v[Y_t\mid\mathcal F_{t-1}^{\mathrm{obs}}=o]=v\,c_0\,\varepsilon_C\,a_t^{(Q)}(o)\,q_t^v(o),$ with $a_t^{(Q)}(o)$ the rare-action-history factor of def:filter-quantities evaluated at $o$ . The identity follows from the unnormalized filter mass $\lambda_v(h;o)$ of lem:signed-depth-terminal-sign-invariant, the probability that the prefix equals $o$ and the hidden state at epoch $t$ equals $h$ : since the epoch- $t$ reward has conditional mean $v\,c_0\,U(h)\mathbf 1\{J(h)=Q\}$ given hidden state $h$ , the next-reward numerator is $\mathbb E_v[Y_t\mathbf 1\{(A_i,Y_i)_{i<t}=o\}]=v\,c_0\sum_{h:\,J(h)=Q}U(h)\lambda_v(h;o)$ , while $m_v^{Q}(o)=\sum_{h:\,J(h)=Q}\lambda_v(h;o)$ and $m_v(o)=\sum_h\lambda_v(h;o)$ ; the terminal signed-mass invariant replaces the signed moment by $\varepsilon_C\,a_t^{(Q)}(o)\,m_v^{Q}(o)$ , and dividing by $m_v(o)$ gives the identity, both sides vanishing when $m_v(o)=0$ .
Lemma lem:insulin-bias-interval-evaluation verified CausalSmith.Stat.PomdpLatentOverlapMinimax.insulinBiasInterval_bounds For $\star\in\{\mathrm b,\mathrm e\}$ , let $\mathfrak T_{\star,0}$ and $\mathfrak T_{\star,1}$ be the terminal and successor rational interval rows on the $360$ -state grid $\mathsf S_{\mathrm{grid}}$ of def:insulin-grid recorded by the zero-step chunked iterate certificate (blocks of at most $40$ states) for the rational interval enclosure of the behavior-policy ( $\star=\mathrm b$ ) or target-policy ( $\star=\mathrm e$ ) transition matrix, and let $\mathfrak r_{\mathrm{ins}}(s)=[r_{\mathrm{ins}}(s),r_{\mathrm{ins}}(s)]$ be the point intervals of the grid reward $r_{\mathrm{ins}}(s)\in\{-1,-2/3,-1/3,0\}$ of def:insulin-grid. With the exact rational interval arithmetic of lem:certified-stationary-reward-enclosure, put $\delta_\star=\sum_{s\in\mathsf S_{\mathrm{grid}}}\bigl|\mathfrak T_{\star,0}(s)\ominus\mathfrak T_{\star,1}(s)\bigr|_{\max}, \qquad \mathfrak E_\star=\sum_{s\in\mathsf S_{\mathrm{grid}}}\mathfrak T_{\star,0}(s)\cdot\mathfrak r_{\mathrm{ins}}(s), \qquad \mathfrak J_\star=\Bigl[\ell(\mathfrak E_\star)-\frac{\delta_\star}{1-1/2},\ u(\mathfrak E_\star)+\frac{\delta_\star}{1-1/2}\Bigr],$ and $\mathfrak B=\mathfrak J_{\mathrm b}\ominus\mathfrak J_{\mathrm e}=[\ell(\mathfrak J_{\mathrm b})-u(\mathfrak J_{\mathrm e}),\,u(\mathfrak J_{\mathrm b})-\ell(\mathfrak J_{\mathrm e})]$ . Then $-\frac{6}{1000}<\ell(\mathfrak B) \qquad\text{and}\qquad u(\mathfrak B)<-\frac{58}{10000}.$ This is a closed finite rational computation: $\mathfrak B$ is determined by the recorded certificate tables alone, and both strict inequalities are verified by exact evaluation of the finitely many rational sums, interval products, maxima, and endpoint comparisons.
Lemma lem:rare-action-second-moment-exact verified CausalSmith.Stat.PomdpLatentOverlapMinimax.rareAction_secondMoment_eq Let $t_0>0$ , $\zeta>0$ , $C>1$ , $Q,k\in\mathbb N$ , and $v\in\{-1,+1\}$ , write $L=e^{\zeta}$ , and consider the finite signed-depth model of def:signed-depth-family at horizon $k+1$ . For an action/reward-symbol prefix $o=((a_1,y_1),\ldots,(a_k,y_k))$ let $\pi_k^v(o)=\Pr_v\{(A_i,Y_i)_{i=1}^{k}=o\}$ be its prefix mass, let $\ell=\min\{Q,k\}$ , let $W_k=\{i:\ k-\ell<i\le k\}$ be the recent action window, and let $\mathfrak a_k(o)=L^{-(Q-\ell)}\prod_{i\in W_k}\mathbf 1\{a_i=1\},$ the empty product being one, which is the rare-action-history factor $a_{k+1}^{(Q)}$ of def:filter-quantities evaluated at $o$ . Then $\sum_{o}\pi_k^v(o)\,\mathfrak a_k(o)^2=L^{-2(Q-\ell)}\Bigl(\frac1L\Bigr)^{|W_k|},$ the sum running over all action/reward-symbol prefixes of length $k$ . The computation uses three facts: the product of indicators is idempotent, so $\mathfrak a_k(o)^2=L^{-2(Q-\ell)}\prod_{i\in W_k}\mathbf 1\{a_i=1\}$ ; for a fixed action sequence $(a_1,\ldots,a_k)$ , summing the prefix masses over all reward-symbol sequences leaves the behavior product $\prod_{i=1}^{k}b(a_i)$ with $b(1)=1/L$ and $b(0)=1-1/L$ , because the initial law and the reward/hidden-state kernel are normalized; and under this product law the probability that every action indexed by $W_k$ equals $1$ is $(1/L)^{|W_k|}$ .
Lemma lem:posterior-lower-window-power verified CausalSmith.Stat.PomdpLatentOverlapMinimax.pow_one_sub_le_signedDepthPosteriorLowerWindow Let $t_0,\zeta,C\in\mathbb R$ (no positivity is required), $N,Q\in\mathbb N$ , $v\in\{-1,+1\}$ , and $\bar q\in\mathbb R$ , write $\alpha=e^{-1/t_0}$ and $c_0=(1-\alpha)/4$ , and assume $c_0\ge0,\qquad 1-c_0\bar q\ge0.$ For the finite signed-depth weights of def:signed-depth-family and an observed action/reward prefix $o=((a_1,y_1),\ldots,(a_k,y_k))$ , let $\lambda_v(h;o)$ be the unnormalized filter mass of the hidden state $h$ after $o$ , let $\pi(o)=\sum_{h:\,J(h)=Q}\lambda_v(h;o)\big/\sum_h\lambda_v(h;o)$ be the raw terminal-depth posterior (a real quotient, equal to $0$ when the denominator vanishes), let $o_{\le m}$ denote the first $m$ pairs of $o$ , and let $\Lambda_d(o)=\prod_{s=k-d+1}^{k}\bigl(1-c_0\,\pi(o_{\le s-1})\bigr)$ . Let $d\le k$ and assume $\pi(o_{\le i})\le\bar q\qquad\text{for every }0\le i<k.$ Then $(1-c_0\bar q)^d\le\Lambda_d(o).$ The proof is by induction on $d$ , simultaneously over all prefixes: for $d=0$ both sides equal $1$ ; for $d+1\le k$ , peeling the last coordinate gives $\Lambda_{d+1}(o)=\Lambda_d(o_{\le k-1})\,\bigl(1-c_0\pi(o_{\le k-1})\bigr)$ , where the strict prefixes of $o_{\le k-1}$ are the prefixes $o_{\le i}$ , $i<k-1$ , so the hypothesis transfers to $o_{\le k-1}$ ; the induction hypothesis bounds the first factor below by $(1-c_0\bar q)^d\ge0$ , and the hypothesis at $i=k-1$ bounds the second factor below by $1-c_0\bar q\ge0$ .
Lemma lem:terminal-posterior-mature-window verified CausalSmith.Stat.PomdpLatentOverlapMinimax.signedDepthRawPosterior_le_product_of_le Let $t_0>0$ , $\zeta>0$ , $C>1$ , $N,Q\in\mathbb N$ , and $v\in\{-1,+1\}$ , and write $\alpha=e^{-1/t_0}$ , $L=e^{\zeta}$ , $c_0=(1-\alpha)/4$ , and $\varepsilon_C=\min\{(C-1)/2,1/2\}$ . For the finite signed-depth model of def:signed-depth-family with initial hidden-state weights $p_v$ , behavior policy $b$ , and reward/hidden-state kernel $K_v$ , and for an observed action/reward prefix $o=((a_1,y_1),\ldots,(a_k,y_k))$ , let $\lambda_v(h;o)$ be the unnormalized filter mass, the probability that the first $k$ action/reward pairs equal $o$ and the hidden state after them equals $h\in\{0,\ldots,Q\}\times\{-1,+1\}$ ; write $J(h)$ for the depth coordinate of $h$ , $\mathsf M(o)=\sum_h\lambda_v(h;o)$ , and $\pi(o)=\frac{\sum_{h:\,J(h)=Q}\lambda_v(h;o)}{\mathsf M(o)}$ for the raw terminal-depth posterior. For $0\le m\le k$ let $o_{\le m}$ denote the first $m$ pairs of $o$ , and for $0\le d\le k$ define the lower posterior window $\Lambda_d(o)=\prod_{s=k-d+1}^{k}\bigl(1-c_0\,\pi(o_{\le s-1})\bigr),$ the empty product being one. Assume $Q\le k$ . Then $\pi(o)\le\frac{\alpha^Q}{\Lambda_Q(o)}.$ The bound combines three facts. First, the terminal numerator satisfies $\sum_{h:\,J(h)=Q}\lambda_v(h;o)=\alpha^Q\,\mathsf{Fair}_Q(o)\,\mathsf U_0(o_{\le k-Q})$ , where $\mathsf U_0(o')=\sum_{h:\,J(h)=0}\lambda_v(h;o')$ : reaching depth $Q$ across the last $Q$ epochs requires $Q$ consecutive advances, each of weight $\alpha$ , and after summing over signs the reward tilt averages to the fair factor $1/2$ . Second, the total mass iterates over the last $Q$ epochs as $\mathsf M(o)=\mathsf{Fair}_Q(o)\,\mathsf M(o_{\le k-Q})\prod_{s=k-Q+1}^{k}\mathsf{Lik}_s(o)$ . Third, $0\le\mathsf U_0(o_{\le k-Q})\le\mathsf M(o_{\le k-Q})$ and $\mathsf M(o_{\le k-Q})>0$ . Here $\mathsf{Fair}_d(o)=\prod_{s=k-d+1}^{k}b(a_s)/2>0$ with $b(1)=1/L$ and $b(0)=1-1/L$ , and the one-step likelihood corrections are $\mathsf{Lik}_s(o)=1+\bar y_s\,v\,c_0\,\varepsilon_C\,\mathfrak a(o_{\le s-1})\,\pi(o_{\le s-1})$ , where $\bar y_s\in\{-1,+1\}$ is the $s$ -th observed reward and, for a prefix $o'$ of length $m$ , $\mathfrak a(o')=L^{-(Q-\min\{Q,m\})}\prod_{r=m-\min\{Q,m\}+1}^{m}\mathbf 1\{a_r=1\}$ is the rare-action-history factor of def:filter-quantities; the total-mass iteration uses the terminal sign invariant for the signed terminal mass, and $0<\Lambda_d(o)\le\prod_{s=k-d+1}^{k}\mathsf{Lik}_s(o)$ because $0\le\pi\le1$ , $0\le c_0<1$ , and $|\bar y_s v\varepsilon_C\mathfrak a(o_{\le s-1})|\le1$ .
Lemma lem:terminal-posterior-initial-window verified CausalSmith.Stat.PomdpLatentOverlapMinimax.signedDepthRawPosterior_le_product_of_lt Let $t_0>0$ , $\zeta>0$ , $C>1$ , $N,Q\in\mathbb N$ , and $v\in\{-1,+1\}$ , and write $\alpha=e^{-1/t_0}$ , $L=e^{\zeta}$ , $c_0=(1-\alpha)/4$ , and $\varepsilon_C=\min\{(C-1)/2,1/2\}$ . For the finite signed-depth model of def:signed-depth-family with initial hidden-state weights $p_v$ , behavior policy $b$ , and reward/hidden-state kernel $K_v$ , and for an observed action/reward prefix $o=((a_1,y_1),\ldots,(a_k,y_k))$ , let $\lambda_v(h;o)$ be the unnormalized filter mass, the probability that the first $k$ action/reward pairs equal $o$ and the hidden state after them equals $h\in\{0,\ldots,Q\}\times\{-1,+1\}$ ; write $J(h)$ for the depth coordinate of $h$ , $\mathsf M(o)=\sum_h\lambda_v(h;o)$ , and $\pi(o)=\frac{\sum_{h:\,J(h)=Q}\lambda_v(h;o)}{\mathsf M(o)}$ for the raw terminal-depth posterior. For $0\le m\le k$ let $o_{\le m}$ denote the first $m$ pairs of $o$ , and for $0\le d\le k$ define the lower posterior window $\Lambda_d(o)=\prod_{s=k-d+1}^{k}\bigl(1-c_0\,\pi(o_{\le s-1})\bigr),$ the empty product being one. Assume $k<Q$ . Then $\pi(o)\le\frac{\alpha^Q}{\Lambda_k(o)}.$ The bound combines three facts. First, the terminal numerator is $\sum_{h:\,J(h)=Q}\lambda_v(h;o)=w_Q\,\mathsf{Fair}_k(o)$ , where $w_Q=(1-\alpha)\alpha^Q/(1-\alpha^{Q+1})$ is the stationary terminal depth mass of def:signed-depth-family: before $Q$ observations the missing pre-sample advances are absorbed by the stationary initial law. Second, the total mass iterates over the whole prefix as $\mathsf M(o)=\mathsf{Fair}_k(o)\prod_{s=1}^{k}\mathsf{Lik}_s(o)$ , because the empty prefix has total mass $1$ . Third, $w_Q\le\alpha^Q$ , which after multiplying by $1-\alpha^{Q+1}>0$ reduces to $\alpha^{Q+1}\le\alpha$ . Here $\mathsf{Fair}_d(o)=\prod_{s=k-d+1}^{k}b(a_s)/2>0$ with $b(1)=1/L$ and $b(0)=1-1/L$ , and the one-step likelihood corrections are $\mathsf{Lik}_s(o)=1+\bar y_s\,v\,c_0\,\varepsilon_C\,\mathfrak a(o_{\le s-1})\,\pi(o_{\le s-1})$ , where $\bar y_s\in\{-1,+1\}$ is the $s$ -th observed reward and, for a prefix $o'$ of length $m$ , $\mathfrak a(o')=L^{-(Q-\min\{Q,m\})}\prod_{r=m-\min\{Q,m\}+1}^{m}\mathbf 1\{a_r=1\}$ is the rare-action-history factor of def:filter-quantities; the total-mass iteration uses the terminal sign invariant for the signed terminal mass, and $0<\Lambda_d(o)\le\prod_{s=k-d+1}^{k}\mathsf{Lik}_s(o)$ because $0\le\pi\le1$ , $0\le c_0<1$ , and $|\bar y_s v\varepsilon_C\mathfrak a(o_{\le s-1})|\le1$ .
Lemma lem:terminal-posterior-crude-bound verified CausalSmith.Stat.PomdpLatentOverlapMinimax.signedDepthRawPosterior_le_ratio_pow Let $t_0>0$ , $\zeta>0$ , $C>1$ , $N,Q\in\mathbb N$ , and $v\in\{-1,+1\}$ , and write $\alpha=e^{-1/t_0}$ , $L=e^{\zeta}$ , $c_0=(1-\alpha)/4$ , and $\varepsilon_C=\min\{(C-1)/2,1/2\}$ . For the finite signed-depth model of def:signed-depth-family with initial hidden-state weights $p_v$ , behavior policy $b$ , and reward/hidden-state kernel $K_v$ , and for an observed action/reward prefix $o=((a_1,y_1),\ldots,(a_k,y_k))$ , let $\lambda_v(h;o)$ be the unnormalized filter mass, the probability that the first $k$ action/reward pairs equal $o$ and the hidden state after them equals $h\in\{0,\ldots,Q\}\times\{-1,+1\}$ ; write $J(h)$ for the depth coordinate of $h$ , $\mathsf M(o)=\sum_h\lambda_v(h;o)$ , and $\pi(o)=\frac{\sum_{h:\,J(h)=Q}\lambda_v(h;o)}{\mathsf M(o)}$ for the raw terminal-depth posterior. For $0\le m\le k$ let $o_{\le m}$ denote the first $m$ pairs of $o$ , and for $0\le d\le k$ define the lower posterior window $\Lambda_d(o)=\prod_{s=k-d+1}^{k}\bigl(1-c_0\,\pi(o_{\le s-1})\bigr),$ the empty product being one. Then, for every observed prefix $o$ of every length $k$ , $\pi(o)\le\Bigl(\frac{\alpha}{1-c_0}\Bigr)^Q.$ Indeed $0\le c_0<1$ , and $0\le\pi(o')\le1$ for every prefix $o'$ , because the terminal mass is a sub-sum of the nonnegative filter masses and $\mathsf M(o')>0$ . Applying lem:posterior-lower-window-power with the uniform bound $\bar q=1$ on all strict prefixes gives $\Lambda_d(o)\ge(1-c_0)^d$ for every $d\le k$ . If $Q\le k$ , lem:terminal-posterior-mature-window gives $\pi(o)\le\alpha^Q/(1-c_0)^Q$ ; if $k<Q$ , lem:terminal-posterior-initial-window gives $\pi(o)\le\alpha^Q/(1-c_0)^k\le\alpha^Q/(1-c_0)^Q$ , since $0<1-c_0\le1$ .
Lemma lem:signed-depth-next-reward-numerator verified CausalSmith.Stat.PomdpLatentOverlapMinimax.signedDepth_prefix_nextRewardNumerator Let $t_0>0$ , $\zeta>0$ , $C>1$ , $Q,k,n\in\mathbb N$ , and $v\in\{-1,+1\}$ , write $\alpha=e^{-1/t_0}$ and $c_0=(1-\alpha)/4$ , and consider the finite signed-depth model of def:signed-depth-family with initial hidden-state weights $p_v$ , behavior policy $b$ , and reward-symbol/hidden-state kernel $K_v$ . A finite trajectory $\tau$ of horizon $n+k+1$ consists of hidden states $h_1,\ldots,h_{n+k+2}$ and action/reward-symbol pairs $(a_i,r_i)$ , $r_i\in\{0,1\}$ , $1\le i\le n+k+1$ ; its chronological weight is $\omega_v(\tau)=p_v(h_1)\prod_{i=1}^{n+k+1}b(a_i)\,K_v(r_i,h_{i+1}\mid h_i,a_i),$ and $\phi(0)=-1$ , $\phi(1)=1$ decodes reward symbols. For an action/reward-symbol prefix $o=((a_1,r_1),\ldots,(a_k,r_k))$ and a hidden state $h\in\{0,\ldots,Q\}\times\{-1,+1\}$ with depth $J(h)$ and sign $U(h)$ , let $\lambda_v(h;o)=\sum_{h_1,\ldots,h_k}p_v(h_1)\prod_{i=1}^{k}b(a_i)\,K_v(r_i,h_{i+1}\mid h_i,a_i),\qquad h_{k+1}=h,$ be the unnormalized filter mass. Then $\sum_{\tau:\ ((a_i,r_i))_{i\le k}(\tau)=o}\phi\bigl(r_{k+1}(\tau)\bigr)\,\omega_v(\tau) = v\,c_0\sum_{h:\,J(h)=Q}U(h)\,\lambda_v(h;o).$ The left side is split at epoch $k+1$ : for a fixed hidden prefix $h_1,\ldots,h_{k+1}$ and next pair $(a,r)$ , the weights of all continuations after the next hidden state sum to one by normalization of $b$ and $K_v$ , so the left side equals $\sum_h\lambda_v(h;o)\sum_a b(a)\sum_r\phi(r)\sum_{h'}K_v(r,h'\mid h,a)$ . The one-step payoff $\sum_a b(a)\sum_r\phi(r)\sum_{h'}K_v(r,h'\mid h,a)$ equals $v\,c_0\,U(h)\mathbf 1\{J(h)=Q\}$ , because the reward-symbol marginal of $K_v(\cdot\mid h,a)$ is $\bigl(1+\phi(r)\,v\,c_0\,U(h)\mathbf 1\{J(h)=Q\}\bigr)/2$ and $\sum_ab(a)=1$ .
Lemma lem:signed-depth-terminal-prefix-mass verified CausalSmith.Stat.PomdpLatentOverlapMinimax.terminalPrefixMass_eq_sum_signedDepthFilterMass Let $t_0,\zeta,C\in\mathbb R$ , $Q,k,n\in\mathbb N$ with $n\ge1$ , and $v\in\{-1,+1\}$ , and consider the finite signed-depth model of def:signed-depth-family at horizon $n+k$ , with trajectory law $\Pr_v$ , initial hidden-state weights $p_v$ , behavior policy $b$ , and reward-symbol/hidden-state kernel $K_v$ . For an action/reward-symbol prefix $o=((a_1,r_1),\ldots,(a_k,r_k))$ , let $m_v^{Q}(o)=\Pr_v\bigl\{\text{the first }k\text{ action/reward-symbol pairs equal }o,\ J_{k+1}=Q\bigr\}$ be the joint mass of the prefix event and terminal depth of the hidden state at the cut after the first $k$ epochs, which is the numerator of the terminal posterior $q_{k+1}^v(o)$ of def:filter-quantities, and let $\lambda_v(h;o)=\sum_{h_1,\ldots,h_k}p_v(h_1)\prod_{i=1}^{k}b(a_i)\,K_v(r_i,h_{i+1}\mid h_i,a_i),\qquad h_{k+1}=h,$ be the unnormalized filter mass of a hidden state $h$ with depth $J(h)$ . Then $m_v^{Q}(o)=\sum_{h:\,J(h)=Q}\lambda_v(h;o).$ The identity expands each trajectory weight through the cut state and sums out the remaining epochs: for a fixed cut state, the total weight of all length- $n$ chronological continuations is one by induction on $n$ , using $\sum_ab(a)=1$ and $\sum_{r,h'}K_v(r,h'\mid h,a)=1$ ; the terminal-depth indicator depends only on the cut state and therefore passes through the continuation sum.
Lemma lem:parametric-pair-target-value verified CausalSmith.Stat.PomdpLatentOverlapMinimax.parametricPair_targetValue Let $t_0>0$ , $\zeta>0$ , $C\ge1$ , $T\in\mathbb N$ , and $\sigma\in\{-1,+1\}$ , and put $\eta_T=1/(4\sqrt T)$ , read as $0$ when $T=0$ . Let $M_T^{\sigma}$ be the singleton-state experiment with $\mathcal X=\{x_\star\}$ , $\mathcal H=\{h_\star\}$ , fair policies $b(a\mid x_\star)=e(a\mid x_\star)=1/2$ for $a\in\{0,1\}$ , and reward-transition kernel that, from every state-action pair, gives reward $y\in\{-1,+1\}$ together with next state $(x_\star,h_\star)$ probability $(1+\sigma y\eta_T)/2$ . Then the target value of def:target-value of this experiment is $\theta(K,e)=\sigma\,\eta_T.$ Every state-action pair has reward mean $\sum_{y\in\{-1,+1\}}y\,(1+\sigma y\eta_T)/2=\sigma\eta_T$ , so the target-policy reward regression is the constant $\sigma\eta_T$ ; averaging it against the selected target stationary law, which is a probability vector because $M_T^\sigma$ satisfies the conditions of def:model-class at these parameters, gives $\sigma\eta_T$ .
Lemma lem:parametric-pair-observed-kl verified CausalSmith.Stat.PomdpLatentOverlapMinimax.parametricPair_observed_klDiv_le Let $T\ge1$ , put $\eta_T=1/(4\sqrt T)$ , and for $\sigma\in\{-1,+1\}$ let $M_T^{\sigma}$ be the singleton-state experiment with one observed state $x_\star$ , one hidden state, fair policies $b=e$ uniform on $\{0,1\}$ , and reward-transition kernel giving reward $y\in\{-1,+1\}$ and return to the single state with probability $(1+\sigma y\eta_T)/2$ . Write $M_T^-$ and $M_T^+$ for $\sigma=-1$ and $\sigma=+1$ . Their observed-data laws $\mathbb P^{\mathrm{obs}}_{M_T^\pm}$ of the length- $T$ observed path $\mathsf O_T$ satisfy $\operatorname{KL}\bigl(\mathbb P^{\mathrm{obs}}_{M_T^-}\,\big\|\,\mathbb P^{\mathrm{obs}}_{M_T^+}\bigr)\le\frac14;$ in particular this divergence is finite. With reward codes $\mathsf r\in\{0,1\}$ in place of rewards $2\mathsf r-1$ , the coded observed word is the $T$ -fold product of the one-epoch law $\Lambda_T^\sigma(x_\star,a,\mathsf r)=\frac12\cdot\frac{1+\sigma(2\mathsf r-1)\eta_T}{2}$ ; the common fair action factor cancels, so the one-epoch divergence is the Bernoulli divergence $\eta_T\log\frac{1+\eta_T}{1-\eta_T}\le4\eta_T^2$ , using $0\le\eta_T\le1/4$ ; tensorization bounds the coded-word divergence by $4T\eta_T^2=1/4$ ; and data processing under the decoding map $\mathsf r\mapsto2\mathsf r-1$ transfers the bound to the observed laws.
Lemma lem:two-point-risk-floor verified CausalSmith.Stat.PomdpLatentOverlapMinimax.two_point_floor_explicit Let $T\ge1$ , $t_0,\zeta,C\in\mathbb R$ , and let $M_0,M_1\in\mathcal M_T(t_0,\zeta,C)$ of def:model-class have the same observed-alphabet cardinality and the same behavior and target policies $b$ and $e$ . Write $\mathbb P_j$ for the observed-data law of $M_j$ and $\theta_j$ for its target value of def:target-value, $j\in\{0,1\}$ . Let $s\ge0$ and $\chi\ge0$ , and assume $2s\le|\theta_0-\theta_1|,\qquad \mathbb P_0\ll\mathbb P_1,\qquad \operatorname{KL}(\mathbb P_0\|\mathbb P_1)\le\chi,\qquad \operatorname{KL}(\mathbb P_0\|\mathbb P_1)<\infty.$ Then the following hold. (i) For every raw estimator $\widehat\theta$ , a real-valued map of the observed-alphabet size, the policy pair, and the observed path that is measurable in the observed path for each alphabet size and policy pair, and whose squared errors $(\widehat\theta-\theta_j)^2$ , evaluated at the alphabet size and policies of $M_j$ , are $\mathbb P_j$ -integrable for $j=0,1$ , $\frac{s^2e^{-\chi}}{4}\le\max\Bigl\{\mathbb E_{M_0}\bigl[(\widehat\theta-\theta_0)^2\bigr],\ \mathbb E_{M_1}\bigl[(\widehat\theta-\theta_1)^2\bigr]\Bigr\}.$ (ii) The minimax risk of def:minimax-risk satisfies $\frac{s^2e^{-\chi}}{4}\le R_T(t_0,\zeta,C).$ Part (i) follows from Le Cam's two-point testing inequality $\bigl(1-\mathrm{TV}(\mathbb P_0,\mathbb P_1)\bigr)/2\le\max_j\mathbb P_j\{s\le|\widehat\theta-\theta_j|\}$ , which uses the separation $2s\le|\theta_0-\theta_1|$ and the common policies; the Bretagnolle--Huber inequality $\frac12\exp\{-\operatorname{KL}(\mathbb P_0\|\mathbb P_1)\}\le1-\mathrm{TV}(\mathbb P_0,\mathbb P_1)$ together with the KL bound; and Markov's inequality $s^2\,\mathbb P_j\{s\le|\widehat\theta-\theta_j|\}\le\mathbb E_{M_j}[(\widehat\theta-\theta_j)^2]$ . Part (ii) applies (i) to each bounded measurable observable-data estimator in the minimax infimum, whose squared errors are integrable, and bounds its supremum risk over the class below by its maximum risk over $M_0$ and $M_1$ .
Lemma lem:target-value-unit-interval verified CausalSmith.Stat.PomdpLatentOverlapMinimax.targetValue_mem_unit Let $T\ge1$ and $t_0,\zeta,C\in\mathbb R$ , and let $M=(\mathcal X,\mathcal H,K,b,e,\mathcal L(\mathsf O_T))\in\mathcal M_T(t_0,\zeta,C)$ of def:model-class. For a joint state $s=(x,h)$ and an action $a\in\{0,1\}$ , write $\bar r(s,a):=\int y\,K(dy,ds'\mid s,a)$ for the reward-coordinate mean of the kernel row $K(\cdot\mid s,a)$ ; this row is a probability law for every $(s,a)$ , including rows of zero realized probability, and the integral is read as $0$ if the reward coordinate is not integrable, so $\bar r(s,a)$ and $g_e(s)=\sum_{a}e(a\mid x)\bar r(s,a)$ are defined everywhere. Then the target value of def:target-value satisfies $\theta(K,e)=\sum_{s}d_e(s)g_e(s)\in[-1,1].$ Indeed, if $d_b(s)>0$ and $b(a\mid x)>0$ , then \cref{obj:ass:stationary-start,obj:ass:sequential-ignorability} give the first-epoch event $\{S_0=s,A_0=a\}$ positive probability, ass:pomdp-kernel identifies its conditional reward-successor law with $K(\cdot\mid s,a)$ , and ass:bounded-reward gives $|\bar r(s,a)|\le1$ . If $d_e(s)>0$ and $e(a\mid x)>0$ , then ass:latent-stationary-overlap gives $d_b(s)>0$ and ass:policy-overlap gives $b(a\mid x)>0$ , so $|\bar r(s,a)|\le1$ on every such row. Actions with $e(a\mid x)=0$ contribute $0$ to $g_e(s)$ whatever the value of $\bar r(s,a)$ , and $e(\cdot\mid x)$ is a probability vector, so $|g_e(s)|\le1$ whenever $d_e(s)>0$ ; averaging against the probability vector $d_e$ gives $|\theta(K,e)|\le1$ .
Lemma lem:observed-risk-bounded-by-four verified CausalSmith.Stat.PomdpLatentOverlapMinimax.observedRisk_le_four Let $T\ge1$ and $t_0,\zeta,C\in\mathbb R$ . Let $\widehat\theta$ be an admissible observable-data estimator in the infimum of def:minimax-risk: for each observed-alphabet size and policy pair, $\widehat\theta$ is a measurable real-valued function of the observed path $\mathsf O_T$ taking values in $[-1,1]$ . Then for every $M=(\mathcal X,\mathcal H,K,b,e,\mathcal L(\mathsf O_T))\in\mathcal M_T(t_0,\zeta,C)$ of def:model-class, with $\widehat\theta$ evaluated at the observed-alphabet size and policies of $M$ , $E_M\bigl[(\widehat\theta-\theta(K,e))^2\bigr]\le4 .$ Consequently the risk map $M\mapsto E_M[(\widehat\theta-\theta(K,e))^2]$ is bounded above by $4$ on $\mathcal M_T(t_0,\zeta,C)$ . Indeed, $\widehat\theta\in[-1,1]$ pointwise and $\theta(K,e)\in[-1,1]$ by lem:target-value-unit-interval, with $\theta(K,e)$ the target value of def:target-value, so the squared error is at most $4$ at every observed path; the squared error is measurable because $\widehat\theta$ is, and the observed-path law of $M$ is a probability measure, so its expectation is at most $4$ .

Theorems (5)

Theorem T-1 verified CausalSmith.Stat.PomdpLatentOverlapMinimax.fixed_c_minimax For every fixed $t_0>0$ , $\zeta>0$ , and $C>1$ , there exist $0<c_\star\le c^\star<\infty$ and $T_\star<\infty$ , depending only on $t_0,\zeta,C$ , such that $c_\star T^{-2/(2+t_0\zeta)}\le R_T(t_0,\zeta,C)\le c^\star T^{-2/(2+t_0\zeta)}$ for every $T\ge T_\star$ . The upper bound is attained by $\widehat\theta_{k_T}$ ; the lower bound holds against every measurable observable-data estimator, already with constant $X_t$ and $2(Q+1)$ latent states. Thus the HuWager2023 Theorem 3.1 minimax exponent remains sharp after strictly restricting its POMDP class by $d_e\le Cd_b$ , with no logarithmic factor.
Theorem T-2 verified CausalSmith.Stat.PomdpLatentOverlapMinimax.unit_overlap_boundary At $C=1$ , $d_e=d_b$ , the immediate estimator $\widehat\theta_0$ has cardinality-uniform mean-squared error of order $T^{-1}$ , and $R_T(t_0,\zeta,1)\asymp T^{-1}$ . Consequently the fixed- $C>1$ theorem has a genuine discontinuous boundary in exponent, although its constants need not be uniform as $C\downarrow1$ .
Theorem T-3 verified CausalSmith.Stat.PomdpLatentOverlapMinimax.finite_insulin_demonstration The refreshed insulin construction $\mathcal I_{\mathrm{grid}}$ has $360$ joint states, Dobrushin coefficient at most $1/2$ under both policies, one-step policy overlap $L=10/3$ , and the analytic stationary-overlap bound $d_e(s)\le720d_b(s)$ for every state. An interval-certified evaluation of its finite transition matrices satisfies $-0.0060<\Delta_{\mathrm{imm}}<-0.0058$ , so immediate action weighting is biased despite finite latent stationary overlap.
Theorem T-4 verified CausalSmith.Stat.PomdpLatentOverlapMinimax.shrinking_overlap_frontier Let $t_0>0$ and $\zeta>0$ be fixed, and let $\delta_T\ge0$ be any deterministic sequence with $\delta_T\to0$ . With $\beta=2/(2+t_0\zeta)$ , there are constants $0<c_{\mathrm{loc}}\le C_{\mathrm{loc}}<\infty$ , depending only on $t_0,\zeta$ , such that, for all sufficiently large $T$ , $c_{\mathrm{loc}}\left\{T^{-1}+T^{-\beta}\delta_T^{2(1-\beta)}\right\} \le R_T(t_0,\zeta,1+\delta_T) \le C_{\mathrm{loc}}\left\{T^{-1}+T^{-\beta}\delta_T^{2(1-\beta)}\right\}.$ An attaining observable estimator is $\widehat\theta_{\kappa_T}$ , where $\kappa_T=0$ if $T\delta_T^2\le1$ , and otherwise $\kappa_T:=\min\!\left\{\left\lfloor T/2\right\rfloor, \left\lfloor\frac{\log(T\delta_T^2)}{2\log(1/\alpha)+\log L}\right\rfloor\right\}.$ Thus the exact threshold is $\delta_T\asymp T^{-1/2}$ : when $T\delta_T^2=O(1)$ , immediate weighting has parametric order, whereas when $T\delta_T^2\to\infty$ , the required history is of order $\log(T\delta_T^2)$ and the second risk term dominates. There is no logarithmic factor in the risk.
Theorem T-5 verified CausalSmith.Stat.PomdpLatentOverlapMinimax.uniform_overlap_frontier For every fixed $t_0>0$ and $\zeta>0$ , uniformly over all $C\ge1$ , $R_T(t_0,\zeta,C) \asymp_{t_0,\zeta} T^{-1}+T^{-\beta}q_C^{2(1-\beta)}, \qquad \beta=\frac{2}{2+t_0\zeta},$ for all sufficiently large $T$ , with comparison constants and the threshold independent of $C$ and of both state-space cardinalities. The upper bound is attained by the observable estimator $\widehat\theta_{\kappa_T^{\mathrm{rad}}(C)}$ . Equivalently, the exact elbow is $q_C\asymp T^{-1/2}$ : depth zero has parametric order when $Tq_C^2=O(1)$ , whereas the hidden-state term dominates when $Tq_C^2\to\infty$ . No logarithmic factor occurs.

Auxiliary Lean lemmas (machine-checked) (70)

Definition frontierRate —
Definition IsStationary —
Definition localFrontierRate —
Definition ModelIndex —
Definition observedRisk —
Definition phiwObservable —
Definition PolicyVector —
Definition ProbabilityVector —
Definition rateExponent —
Definition RawPomdpExperiment —
Definition shrinkingDepth —
Definition FiniteRewardModel —
Definition obsLaw —
Definition RawEstimator —
Definition rawObservedRisk —
Definition signedDepthModel —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.rademacher_kl_le_chiSq —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.parametricAmplitude_mem —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.parametricPair_kernel_toReal —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.parametricPair_policy_toReal —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.parametricPair_obsPMF_toReal —
Definition CausalSmith.Stat.PomdpLatentOverlapMinimax.parametricEpochPMF —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.parametricEpochPMF_toReal —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.parametricPair_obsPMF_eq_pi —
Definition CausalSmith.Stat.PomdpLatentOverlapMinimax.parametricEpochEquiv —
Definition CausalSmith.Stat.PomdpLatentOverlapMinimax.parametricEpochSource —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.parametricEpochPMF_eq_map_source —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.parametricEpoch_klDiv_le —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.parametricEpoch_fullSupport —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.parametricPair_finiteObs_klDiv_le —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.parametricPair_observed_klDiv_le —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.parametricPair_kernelMean —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.parametricPair_targetValue —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.historyDepth_rate —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.radiusAdaptiveDepth_rate —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.signedDepth_eps_bound —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.signedDepth_initial_apply —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.signedDepth_stationary_overlap_base —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.signedDepth_policyOverlap —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.signedDepth_uniformContraction —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.signedDepth_rewardRegression —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.signedDepth_targetValue —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.signedDepth_behavior_stationary_eq_init —
Definition CausalSmith.Stat.PomdpLatentOverlapMinimax.signedDepthStationaryWeight —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.actionOnePolicy_policyVector —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.signedDepthStationaryWeight_probabilityVector —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.signedDepth_policyKernel_apply —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.depthMass_reset_balance —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.latentDepth_le —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.depthMass_succ —
Definition CausalSmith.Stat.PomdpLatentOverlapMinimax.signedDepthRefresh —
Definition CausalSmith.Stat.PomdpLatentOverlapMinimax.signedDepthResidualKernel —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.signedDepthResidualKernel_probabilityVector —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.signedDepth_policyKernel_refresh —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.signedDepth_uniformContraction_actionOne —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.signedDepthStationaryWeight_isStationary —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.signedDepth_stationaryLaw_apply —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.signedDepth_behaviour_eq_actionOne —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.signedDepth_target_eq_actionOne —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.sqRisk_clipUnit_le_variance_add_bias_sq —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.embed_initial_marginal —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.embed_stationaryStart —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.signedDepth_stationaryStart —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.parametricPair_stationaryStart —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.embed_kernel_bounded_reward —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.localFrontier_stationaryRatio —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.rewardRegression_abs_le_one_of_behaviorSupport —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.behaviorSupport_closed_policyKernel —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.markovOperatorIter_congr_on_behaviorSupport —
Lemma CausalSmith.Stat.PomdpLatentOverlapMinimax.integral_curState_eq_behaviorStationary —

Symbol realizations (Lean @realizes) (53)

$\mathcal X$ verified realized by Policy, JointState, ModelIndex
$\mathcal A$ verified realized by Policy
$\mathcal H$ verified realized by JointState, ModelIndex
$\mathcal S$ verified realized by JointState
T verified realized by FullTrajectory, variable@63, ModelIndex
S_t verified realized by FullTrajectory, curState
A_t verified realized by FullTrajectory, actionAt
Y_t verified realized by FullTrajectory, rewardAt, BoundedReward
$\mathsf O_T$ verified realized by ObsView, obsLaw
K verified realized by RawPomdpExperiment
b verified realized by RawPomdpExperiment, PolicyOverlap
e verified realized by RawPomdpExperiment, PolicyOverlap
X_t verified realized by RawPomdpExperiment, curState
H_t verified realized by RawPomdpExperiment, curState
t_0 verified realized by variable@63, LatentOverlapClass, localFrontierHandle
$\zeta$ verified realized by variable@63, LatentOverlapClass, localFrontierHandle
C verified realized by variable@63, LatentStationaryOverlap
P_p verified realized by policyKernel, UniformContraction
d_p verified realized by stationaryLaw, StationaryStart, LatentStationaryOverlap
g_e verified realized by rewardRegression
$\alpha$ verified realized by mixingAlpha, UniformContraction
L verified realized by policyFactor, PolicyOverlap
$\mathcal M_T(t_0,\zeta,C)$ verified realized by LatentOverlapClass, ModelIndex
$\theta(K,e)$ verified realized by targetValue, targetValue_mem_unit_of_class, targetValue_mem_unit
$\rho_t$ verified realized by ratio, ratio_mem
Z_t(k) verified realized by phiwScore
$\widehat\theta_k$ verified realized by phiwEstimator
$R_T(t_0,\zeta,C)$ verified realized by minimaxRisk
$\beta$ verified realized by rateExponent
k_T verified realized by historyDepth
q_C verified realized by overlapRadius
$\kappa_T^{\mathrm{rad}}(C)$ verified realized by radiusAdaptiveDepth
$\mathcal I_{\mathrm{grid}}$ verified realized by insulinGrid
$\Delta_{\mathrm{imm}}$ verified realized by immediateWeightBias
Q verified realized by localFrontierHandle, signedDepthFamily
$\delta_T$ verified realized by localFrontierHandle
$\mathfrak H_{\mathrm{loc}}$ verified realized by localFrontierHandle
$\mathcal F_{t-1}^{\mathrm{obs}}$ verified realized by ObsPrefix
$\ell_t$ verified realized by filterQuantities
q_t^v verified realized by filterQuantities
a_t^{(Q)} verified realized by filterQuantities
$\mathbb P_{Q,v}^{\mathrm{obs},T}$ verified realized by signedDepthObsLaw, signedDepthFamily
K_0 verified realized by observed_path_kl, radius_explicit_observed_kl
B_0 verified realized by observed_path_kl
$c^\star$ verified realized by phiw_upper, uniform_overlap_frontier
$T_\star$ verified realized by phiw_upper, uniform_overlap_frontier
v verified realized by signedValue
U_t verified realized by signedValue
J_t verified realized by latentDepth
$\varepsilon_C$ verified realized by epsC
c_0 verified realized by c0
w_j verified realized by depthMass
$c_\star$ verified realized by uniform_overlap_frontier

Proof map 45 statements · 60 edges

Latent Stationary Overlap and Minimax Off-policy Evaluation in Finite POMDPs

Abstract

Introduction

Assumption interpretation

Limitations

Related work

Setup and estimands

Main results

A finite insulin-policy adaptation

Discussion and limitations

Limitations.

Appendices

Proofs and auxiliary constructions

Verification note

Proofs of the main results

References