Abstract: The unloaded Q-factor is needed when measuring the permittivity of the sample under test in a resonator. In the presence of a coupling mode, the mutual coupling of the two modes influences on the measured unloaded Q-factor. In this paper, an iterative method is proposed to calculate the unloaded Q-factor of the concerned mode with a nearby coupling mode. Two possible coupling mechanisms, in-phase coupling and out-phase coupling, are investigated. The sum in the frequency domain of the squared of the difference of squared module values of transmission coefficient is used as the objective function of the scaled steepest descent method. In the simulation and measurement, the high Q TE011 mode of a cylinder resonator is considered as the concerned mode, and the degenerated low Q TM111 mode the coupling mode. The measurement results show that 3 dB result is deviated from the true value by 14%, while the deviation of the proposed iterative method result is less than 1% with strong mutual coupling.