Finiteness conjecture for vanishing determinants $D_p^{(m)}$

For positive integers $m,n$ with $n$ odd, define

$$D_n^{(m)}=\det\left[(j^2-k^2)^m\left(\frac{j^2-k^2}n\right)\right]_{1\leqslant j,k\leqslant(n-1)/2}.$$

For each positive odd integer $m$, let

$$E(m)=\left\{p:\ p\text{ is a prime with }4\mid p-1\text{ and }p\mid D_p^{(m)}\right\}.$$

Finiteness conjecture. For every positive odd integer $m$, the set $E(m)$ is finite. In particular,

$$E(5)=\{29\},\quad E(7)=\{13,53\},\quad E(9)=\{13,17,29\},\quad E(11)=\{17,29\}.$$

These listed cases are computational observations; the source supplies no proof of the asserted finiteness.