Breasts get swollen due to hormonal changes, not only lactation. Even the menstrual cycle does it and for obvious reason it is particularly notable in pregnancy.
From the Institute for the Advancement of Breastfeeding and Lactation Education (IABLE)
"Breast swelling (lymphedema) often occurs...