Equality of Bernoulli double sums for multiplicative inverses

Let $z$ and $N$ be integers with $\gcd(z,N)=1$. Let $B_1$ and $B_2$ denote the first and second Bernoulli polynomials, respectively, and interpret $z(k-\ell)\bmod N$ as the residue modulo $N$. Write $z^{-1}$ for the multiplicative inverse of $z$ modulo $N$. Equality conjecture.

$$\sum_{k,\ell=1}^{N-1} B_1(k/N) \, B_2((z(k-\ell)\bmod N)/N) \, B_1(\ell/N) = \sum_{k,\ell=1}^{N-1} B_1(k/N) \, B_2((z^{-1}(k-\ell)\bmod N)/N) \, B_1(\ell/N).$$

The equality was verified numerically for all $N\leq 4001$ and all $z\in\{1,\ldots,N-1\}$ coprime to $N$, and it is also proved for Fibonacci lattice rules in the cited context. A general proof or counterexample is not provided here.