We all started shifting out of the cities and into the suburbs, and then into the countryside.
In reality, in preparation for King James's arrival, it is thought that the main entrance was moved to its current position in the late 1500's. It just grew to become a magical location, invisible to us.