Two forces set it: what the server can serve without degrading, and how much the engine wants your content. A site under a few thousand URLs will effectively never hit the ceiling. Above roughly a hundred thousand — catalogues, marketplaces, classifieds — it becomes one of the highest-leverage technical levers there is, because pages that are never fetched are never refreshed.
Waste is where the fix lives. Faceted navigation generating millions of parameter combinations, infinite calendars, session IDs in URLs, redirect chains and soft 404s all consume fetches that should have gone to product pages. Log file analysis is the only way to see it honestly: it shows what the crawler actually requested, which is regularly nothing like what the team assumed.