Conjecture on integral Fourier coefficients of extremal quasi-modular forms

Let $f_{l,w}$ be the normalised extremal quasi-modular form of weight $w$ and depth at most $l$, and let

$$\mathcal{E}_l=\{w\in\mathbb{Z}:f_{l,w}\in\mathbb{Z}[[q]]\}$$

be the set of weights for which its Fourier coefficients are integral. Integral-coefficient conjecture. If $l\in\{1,2,3,4\}$, then $\mathcal{E}_l$ is an infinite set. If $l>4$, then $\mathcal{E}_l$ is a finite set. The conjecture is based on numerical investigations: integrality appears frequently for depths at most four and to cease for depths greater than four, but no proof is supplied.