There is no simple answer to your question. None of us responding here knows your circumstances well enough to be able to do much more than give you some ideas for consideration. Instead, you will have to contemplate which scale offers you the greatest chances of achieving your goals and preferences.
Intuitively, you understand that you can get more N into a given space than you can the larger scales. As stated above, N models and details are now more affordable and plentiful than they were, say, 15 years ago. But there's more to having more stuff in a given space....what if you can't really see it all that well? If you are nearing middle-age, your visual acuity has begun to slip. You'll need better glasses, or eyes, or a lot more light. What kind of light? Situated how? From directly above, from slightly behind the viewer and aimed toward the models in front of the viewer?
Then there is the ease of actually handling said items. Our mitts become less dextrous and sensitive over time. Try replacing a coupler on an N-Scale boxcar or engine. Not for me...not ever. Then there's the track laying in that scale, and the detailing and scenicking to get it looking half-ways decent. At the least you'll need very good eyes and youthful hands...under 50 for many of us.
I would pick a scale in seconds, though, and spend the lion's share of my time figuring out a track plan and a bench to support it that allows me access and handy wiring and trouble-shooting, plus the track plan that "works". A railroad works for revenue. None of them go around an oval...well, in some amusement parks they do, sure, but you catch my drift. So a realistic or operational track plan is going to keep you in the fun and interested far longer than an oval with a siding or two and some industrial spurs.
I hope that helps to orient you to the way I would approach the topic.
-Crandell
