Have you checked how this setting is configured in your Meta account?
Account Center > Your information and permissions
Yes, I once went to a place and it showed up that I had been to that establishment. This is to “help make the ads more relevant.” Often that’s why people say they went to place X and started seeing ads related to the place or to things the place sells.
“How do they know that?” They even pull it from the CPF you provide. Or email, if you have an account, for example.