The fundamental characteristics of metallic materials, such as yield stress, corrosion resistance, and ductility, are closely tied to their microstructure. Solute atoms exhibiting favorable segregation to grain boundaries introduce a drag pressure on the boundary movement, exerting a pronounced influence on the behavior of recrystallization and grain growth, and consequently, on the texture evolution during annealing processes. In the present work, the characteristics of grain growth in an extruded and annealed MN11 magnesium alloy were investigated by quasi in-situ orientation mapping to monitor the dynamic evolution of local and global microstructural features over varying annealing times. Complementary atom probe tomography measurements were utilized to examine the segregation behavior at selected boundaries of large/small vs. small/small grains. Level-set computer simulations were carried with different setups of driving forces to explore their contribution to the microstructure evolution. The results showed that the favorable growth advantage for some grains leading to a transient stage of abnormal grain growth is controlled by several drivers with varying importance over the annealing duration. Local fluctuations in residual dislocation energy and solute concentration near grain boundaries cause different boundary segments to migrate at different rates, which affects the average growth rate of large grains and their evolved shape. Atomistic simulations of per-site segregation energies revealed that the inhomogeneous segregation behavior originates from the local atomic arrangement of grain boundaries involved.