ModelRailroadForums.com is a free Model Railroad Discussion Forum and photo gallery. We cover all scales and sizes of model railroads. Online since 2002, it's one of the oldest and largest model railroad forums on the web. Whether you're a master model railroader or just getting started, you'll find something of interest here.
CAREFULLY!
The first thing I do is put a drop of oil on the point pivots to prevent any glue from getting into them. Also, put oil on the point pull bar/slide and do not put any ballast in these ties. After ballast has dried, remove any loose pieces; these will eventually get in-between the points and rail.
Indeed, as Rex said, think about ecah aspect of the turnout before you lay down ballast. It's hard enough to keep ballast from fouling the points and the throw bar in HO let alone N scale. I'd be tempted to use cork roadbed painted to approximately mathc your ballast under the switch and call it good. You can buy spray paint in a variety of colors that come out looking like stone. It's easy to use on your cork and it will look good enough without having to diddle around with a switch for two hours getting it work right again when you're done.
Try using the silver metal duct tape (not the grey cloth type).
I lay a strip of tape the shape of the throw area sticky side up, then place the turnout on top. Then I press ballast into the turnout between the ties and around the points. When done the loose ballast falls out and the "stuck" ballast is clear of the throw. Seems to work so far!