"bought a second home here"

See, that right there is part of the issue. As another commenter rightfully pointed out, there is a shortage of workers here and so people from out of state may need to move here to work, but you're not contributing to fixing that issue. You're just taking away a home that a local person could have bought for themselves and their family and it's not even a house you need. I know people that have put off buying their first house for up to 3 years at this point because there's just not enough available for a reasonable price, because all of the potential options are being bought by out-of-staters for over-market prices, and these are people whose families have lived here for generations but are now being forced out of state due to a lack of housing.

Of course we're not going to run you out of town with torches and pitchforks. Vermont as a whole prides itself in caring for the people within our boundaries. But you're only hurting our situation, not helping it.