Lower-income neighborhoods have higher numbers of COVID-19 infections than higher-income neighborhoods. Yet we know relatively little about why these neighborhoods are more vulnerable. Using neighborhood-specific COVID-19 data recently released by the City of Toronto, we (a) compare the COVID-19 trajectories in lower- and higher-income neighborhoods and (b) examine the extent to which neighborhoods’ physical infrastructure and sociodemographic composition account for this difference. Until early March, differences in the cumulative number of COVID-19 infections by neighborhood SES were minimal. However, COVID-19 has since spread faster in lower-income neighborhoods. Until the peak of the pandemic, the physical infrastructure of lower-income neighborhoods was the key reason why COVID-19 spread faster in these neighborhoods. After the peak, differences in physical infrastructure played a smaller role. Lower-income neighborhoods had greater difficulty curbing the pandemic because higher shares of their residents belong to groups particularly vulnerable to COVID-19. Our findings illustrate how the pandemic may exacerbate inequalities across neighborhoods.