Many tree species display masting, i.e., population-scale very large inter-annual fluctuations in seed production, implying locally synchronized trees. Although masting plays a key role in the dynamics of temperate forest ecosystems, its future in the context of climate change remains largely unknown. Based on empirical datasets from large network survey of pollen and fruiting dynamics of oak populations in mainland France, combined with modeling work, we provide evidence that (i) flowering phenology partly controls pollen limitation and reproductive failure in oak tree populations and (ii) pollen phenology and pollen limitation have evolved over the last 30 years, yet differently depending on the regions considered. Our work highlights the urgent need to account for flowering phenology in setting up accurate predictions on the future of masting and that of the forest ecosystem dynamics.