Birkhoff-polytope uniqueness conjecture for permutation groups

Let $G$ be a permutation group and let $P(G)$ denote its permutation polytope. Let $B_n$ be the Birkhoff polytope.

Birkhoff-polytope uniqueness conjecture. If $P(G)$ is combinatorially equivalent to $B_n$ for some $n$, then $G$ is effectively equivalent to $S_n$.

The observation motivating the claim is that $B_3$ is realized by the full symmetric group $S_3$, which appears to be essentially the only such permutation group in that case. The general statement remains open.