The paramodular modularity conjecture for abelian varieties of real multiplication type

Let $f$ be a weight $2$ newform for the paramodular group $K(N)$, not in the span of the Gritsenko lifts. Let $k_f$ be the totally real number field generated by the Hecke eigenvalues of $f$, let ${\mathfrak o}$ be its maximal order, and let $\sigma$ range over the embeddings of $k_f$ into ${\mathbb R}$. An abelian variety $A$ is of $({\mathfrak o},N)$-paramodular type when it has ${\mathfrak o}$-multiplication, conductor $N^d$ with reduced conductor $N$, and dimension $2d$, where $d=[k_f:{\mathbb Q}]$. Paramodular real-multiplication conjecture. There is an abelian variety $A_f$ of $({\mathfrak o},N)$-paramodular type such that

$$L(A_f,s)=\prod_{\sigma}L(f^{\sigma},s).$$

Conversely, every abelian variety $A$ of $({\mathfrak o},N)$-paramodular type should be isogenous to $A_f$ for a weight $2$ non-lift newform $f$ on $K(N)$. This extends the proposed correspondence from abelian surfaces with trivial rational endomorphism ring to paramodular abelian varieties with real multiplication; the supplied text gives no resolution.