Tony: Why are asteroids a sign of limitation? They serve a vital role as transport mechanisms for elemental material throughout the universe. Further, "there is a time to build, and a time to destroy". As they'be not inhibited his plans in any way, I see no reason to view them as a limitation.

They are dangerous. Think of Chicxulub. The earth is now well populated with humans. They can be very dangerous to major metropolitan hubs.