We give examples of spin $4$-manifolds with boundary that do not admit metrics of positive scalar curvature and nonnegative mean curvature. These manifolds in fact have the stronger property that the conformal Laplacian with appropriate boundary conditions is never positive. The obstruction to the positivity of the conformal Laplacian is given by a real-valued $\xi$-invariant associated to the APS theorem for the twisted Dirac operator. We use analytic techniques related to the prescribed scalar curvature problem in conformal geometry to move beyond earlier work where the metric is a product near the boundary.