I agree that at some point, a layout can be both too big, and too small. I went from a layout space of 8' x15', to one of 18'x24.' In some ways, the larger layout allowed for a lower track density. I was probably trying to do to much, as even with that space, train lengths were limited to about 8'.
We just moved to a new place back in May, which has a huge basement(over 28'x 45'-so big I don't know the exact dimensions). I've decided that my layout will be no larger than 28' x26', which is plenty big enough, if I ever build it to completion. I tend to be a loner, and with my last layout, the trains almost never ran, as I was continually making 'improvements,' which I want to avoid this time. As a result, I plan to focus first on a 14'x 26' area, which will contain the two major endpoints of the system I have planned. 'For the time being,' they will be connected by a 'shoe-fly track,' to allow me some continuous running. That way if family or other commitments end up becoming more important, I'll still have something that runs, with plenty of room to detail these particular scenes, which I'd find a pleasent change from my past several 'post nuclear plywood centrals.'
I model from about 1975-2000, with a slant toward the 80's. I decided some years ago that at some point in the 90's that 'modern power is just to big,' at least from a overhang perspective on the 22"-26-28" curves I was looking to make standard. Along these lines, I've also compromised on the use of 'shorty equipment,' such as the Athearn streamline cars (73'), and Life Like auto racks(50' double deck cars), as well as 50's TOFC/COFC cars, and Impact/railrunner TOFC cars, due to the curves I'm using. I also prefer the older Riverossi passenger cars, with the talgo couplers, over the more detailed Walther's cars of the last couple of years, for the same reasons. My plans are to spend some time to upgrade these models to make them more prototypical within their context, and the context of my railroad, and to 'live with these choices.' I'm figuring the freights on my new layout will be about 9' in length on average, with 2 dieselts (SD/GP combo) units up front, with most passenger trains being about 7' or shorter.
Given the state of the economy, and the much more limited space constraints that many others live with, I feel very lucky, & should still be able to have fun.
Carl