Boundary determines odd-dimensional middle-index handlebodies

Let $X$ and $X'$ be $(2k+1)$-dimensional $k$-handlebodies. Their boundaries are denoted by $\partial X$ and $\partial X'$.

Boundary-determination conjecture. If $\partial X$ and $\partial X'$ are diffeomorphic, then $X$ and $X'$ are diffeomorphic.

If true, this would imply Lawson's theorem that every closed $5$-manifold can be constructed as a twisted double of a $5$-dimensional $2$-handlebody. The source provides no resolution of the conjecture.