Recrystallization annealing is widely used to tailor the microstructure and enhance the performance of cold-deformed metallic materials. However, the underlying recrystallization mechanisms are debated, even with the use of cutting-edge characterization techniques. Here, we develop a Track-Rex toolbox to analyze quasi in situ electron backscatter diffraction (EBSD) datasets of two magnesium (Mg) alloys during static recrystallization via grain correlation, which can automatically track thousands of grains through different recrystallization stages. The results show that the recrystallized grains do not always grow; instead, they can shrink or even be consumed. This is attributed to the presence of newly formed recrystallized grains that possess a growth advantage over the old recrystallized grains. Consequently, they grow at the expense of the old ones, leading to a minor contribution of those nucleated in early recrystallization stages to the fully recrystallized microstructure. The rare earth containing Mg-2.4Zn-0.2Ce wt.% (ZE20) alloy exhibits a higher nucleation activity in the shear bands compared to the commercial Mg-3Al-1Zn (AZ31) alloy. Regardless of the nucleation timing and sites, all recrystallized nuclei in the ZE20 alloy show consistent off-basal orientations, serving as the origin of the rare earth texture. Moreover, the off-basal texture of these recrystallized grains is further strengthened through preferential growth during subsequent annealing. On the contrary, the recrystallization nuclei in the AZ31 exhibit scattered basal orientations that grow uniformly, resulting in a weak basal texture. This work provides an efficient pathway for grain-scale investigation of recrystallization behaviors in various polycrystalline materials that was previously challenging and time-consuming.