Boundedness of the proxy approximation between TD error and value gradients

Let $V$ be a value function, let $abla_{\mathbf{x}}V(\mathbf{x}_t)$ denote its state-space gradient along a trajectory, let $\delta_t$ be the temporal-difference error, and let $\Delta x$ and $\Delta t$ be the state-space grid spacing and time step, respectively. Assume that the value function and the dynamics satisfy suitable smoothness conditions.

Boundedness of the proxy approximation. There exists a constant $C>0$ such that, for a sufficiently fine state-space discretization,

$$\big|\,\mathbb{E}[|\delta_t|]-\Delta t\cdot\|\nabla_{\mathbf{x}}V(\mathbf{x}_t)\|\,\big|\leq C\Delta x.$$

In the tabular maze setting, $\Delta x=1$, so the bound becomes a constant and $\overline{|\delta|}$ is monotonic in the local gradient magnitude up to a bounded error.

This conjecture formalizes the engineering proxy relating episode-averaged absolute TD error to the value-gradient magnitude in the discrete approximation of the stochastic maximum-principle framework. A formal proof is left for future work, while the paper reports empirical evidence supporting its practical validity.