That is correct. The engine basically assigns a hunk of memory to each player slot - note that: the player slot, not the player. Even though you may not have a full server (you may only have 1 on), the game still assigns this hunk of memory to the slot when it compiles the game, so that it can cater for all players if and when they actually join. This means that if your player numbers exceed what the game can handle in terms of memory and limits, it will crash the server even though there may be 1 or even no one online at the time. The engine will know it can't handle the full quota of players.But the more clients you have the faster you will reach limits. (like visible ents limit according to IzNoGod)
This is why if you have hit a limit, such as max script varibles, or server commands, reducing your player slots can sometimes help. Particularly with COD4, if you keep your player slots down to below 20, you can cram more into your mods. However, if you go over 22 players, a lot of mods around today for COD4 crash servers because of limits. The Openwarfare mod - one that I am actively involved with - is notorious for crashing out at anything above 26 players. Keeping the number down to below 22 or even 20 allows us to put more into the mod.