Atomic norm minimization (ANM) has recently become a powerful tool for gridless compressed sensing (CS). In this paper, the issue of joint estimation of direction-of-departure (DOD) and direction-of-arrival (DOA) for bistatic multiple-input-multiple-output (MIMO) radar is investigated via two dimensional (2D) ANM. However, a major problem of the primal 2D-ANM is that the direct conversion of 2D-ANM into its semi-definite programming (SDP) problem is not strictly established theoretically and is just an approximation, which results in a decline in estimation performance. Besides, the primal 2D-ANM is limited to a single measurement vector (SMV) model. We propose a duality-based 2D-ANM algorithm for grid-free DOD and DOA estimation in MIMO radar, in which the 2D-ANM problem is effectively solved over its optimal variables in the dual-domain with SDP. Thus it retains the benefits of 2D-ANM and holds in theory. Also, it is applicable for SMV as well as multiple measurement vectors (MMV) models and appropriate for non-uniform linear arrays. The simulation results show that the proposed algorithm avoids the grid mismatch effect in DOD and DOA estimation in contrast to the conventional CS methods, and is robust to target correlation and the single-snapshot environment in comparison with the traditional subspace methods.
