You mentioned you were considering a large layout, so price of track would be a factor (I would assume) plus reliability. Other factors to consider are the size of your curves - will you run traditional O gauge (most will run on O-31 curves without issue) or scale/semi-scale (wide curves and turnouts required)?
Your options, from cheapest/most available to most realistic/expensive:
Lionel / K-Line / Marx tubular track. Cheap, reliable, can be upgraded in appearance with extra ties/roadbed/ballast. Still toy-like but very reliable. Turnouts range from O-27 to O-72 (which is equivalent an HO 18" radius turnout).
K-Line (now AristO / RMT) Shadowrail. Still tubular but with closely spaced, plastic ties. O-31 and O-72 turnouts. Blackened center rail.
MTH Realtrax/ Lionel Fastrack / Atlas Industrial O. Somewhat more realistic appearance with rails that look more like "real" rail. Molded plastic roadbed, center rail more subtle and chemically blackened. More expensive than tubular rail but readily available. Turnout options similar to tubular.
Gargraves is available with wood or plastic ties. The original "phantom rail" - chemically blackened center rail. Sectional or flex-track. More turnout options including Y and angled frogs. Considerably more expensive than tubular but much more realistic. Older GG turnouts have issues but newer versions much more reliable.
Atlas O / MTH Scaletrax rail cross sections more realistic. Thin center rail. Close tie spacing. Turnouts look and operate very well. Harder to find (particularly Scaletrax) and pricey but worth it.
Ross Custom Switches are the Mercedes of O 3-rail turnout. Available in a variety of configurations including double-slip, curved, even yard ladders. Solid rail spiked to wood ties. Look great when ballasted but very $$$. Compatible with Gargraves flex or sectional track.
Handlaid. Yes, it IS possible to hand lay 3rail track - I've seen examples in person - but it takes a lot of patience and honestly it didn't look any better than ready made.
Personally, I like (and have used) the Shadowrail from K-Line (now AristO / RMT.) The ties look good and the price is only slightly higher than basic tubular. If you ballast the track and paint/weather the rails it looks quite good. One other upside of tubular rail (besides cost) is the rounded head keeps the rails and wheels cleaner than flat rail - a saving on maintenance. Again, it will depend on the level of realism you want and how much you want to pay. If I wanted to go with more realism than Shadowrail, I would go with Gargraves. Your mileage may vary.
My two cents.