Linear stability properties of the extraordinary mode perturbations in a relativistic electron flow generated inside a planar magnetron are investigated. The stability analysis is carried out within the framework of a linearized macroscopic fluid model and the eigenvalue equation is obtained. In a tenuous density limit, an algebraic dispersion relation for the diocotron instability is obtained from this eigenvalue equation. Results of numerical investigation of this dispersion relation are presented. Particle simulation is also carried out for the diocotron instability and it is shown that the simulation results agree extremely well with the analytical results. Making use of finite element methods, the eigenvalue equation of the extraordinary mode is numerically solved and the results are also presented.