The EJ309 liquid scintillator detector and the developed gamma spectrum unfolding procedure are used to study the complex gamma spectra. The EJ309 liquid scintillator detector is calibrated by using measured response functions of the standard gamma sources 133Ba, 137Cs, and 60Co, which are verified by Monte Carlo simulations for detector calibration accuracy. A gamma spectra unfolding procedure is developed based on the One-fold Gold algorithm, which is used for unfolding simulated complex gamma spectra, experimentally measured response function of 133Ba and 137Cs, and composed complex gamma spectra using measured response function of 133Ba and 137Cs with different relative intensities (RI). The data of unfolded gamma-ray spectra are compared with the standard data. The results show that the EJ309 liquid scintillator detector and the developed gamma spectra unfolding procedure can quantitative and qualitative analysis of complex gamma spectra with reasonable accuracy.