Conjecture on prime divisors of rational torsion for GL2-type abelian varieties

Let $A/\mathbb Q$ be a $g$-dimensional abelian variety of $\operatorname{GL}_2$-type, and let $\ell$ be a prime such that $A$ admits an $\ell$-torsion point.

Prime-divisor conjecture. If $g=2$, then

$$\ell \in \\{2,3,5,7,11,13,19\\}.$$

If $g=3$, then

$$\ell \in \\{2,3,5,7,11,13,17,23,29,31\\}.$$

This conjecture gives the predicted possible prime divisors of rational torsion in dimensions $2$ and $3$. It is suggested by the computational data in the source, but the authors do not know whether the displayed lists are exhaustive.