The ability to experiment and create is at once the great blessing and profound curse of home brewing. On the one hand, we get the opportunity to make anything we want, often in ways that are unlikely or impossible for larger brewing operations, particularly when it comes to incorporating exotic ingredients that would be cost-prohibitive to a brewery. It's all well and good to have all of these options on hand, but it's a double-edged sword: some home brewers feel a bit paralyzed by the range of choices available to them, and/or they simply miss their target and don't keep re-brewing to dial it in! After all, there are so many other beers to brew.
It's in that spirit that we're jumping into some broadly-targeted "specialty" categories. Up first is a favorite of mine: smoked beers. It's my hope that you come away with some good rules-of-thumb that will get you closer to the mark the first time out, and encourage you to go back in for some revisions to make it perfect!
STYLE
There are effectively three smoked beer categories in the 2015 BJCP guidelines: Rauchbier (which we've already done), Specialty Smoked Beer (which incorporates specialty ingredients or undefined styles in addition to smoke), and Classic Style Smoked Beer. It's this last that I'll be focusing on here. The guidelines for each individual base style are obviously going to vary, but the guidelines do provide some useful guidance for any of those to which you might add smoke. The keyword today is balance. In whichever style you choose to smoke up, the smoke character should more or less equal in magnitude the strongest flavor in the profile, and complement and enhance the flavors present. This is both easier and harder than we might think.
Smoked malts can be touchy to work with. For one thing, not every batch of smoked malt is identical, even when comprised of the same grain and smoked with the same wood. For another, wood flavors present very differently from one palate to the next, and in addition the secondary flavors (other than "smoke") may interact in unexpected ways with your other ingredients. Keeping that in mind, I always recommend beginning with (and maybe sticking to) simpler base styles as your "smoking" targets. Can you brew a smoked Belgian Stout? Yes. Should you? Maybe not.
RECIPE
Think of smoked malt as a base malt in the same family as Munich. It's not that dark (2-4L vs. 9-10L in Munich malt), but the smoking adds a depth of flavor that makes it seem heavier. We have two questions to answer: which wood, and how much?
Conventional Bamberg-style Rauchmalt is beech smoked, and will work for most styles. It has a clean, basic smoke flavor that won't get in the way of anything. It's very commonly available, and is what most folks mean when they talk about "smoked malt." A slightly less-common but still available malt is cherry smoked, and while its flavor is roughly comparable to that of the beech smoked (it's sometimes described as a "sweeter" smoke), it should have a longer finish - you'll taste it much more in the aftertaste of your beer, making it a better choice for richer/stronger base styles. Oak smoked wheat malt might also be lurking on the shelves of your local shop, and to my palate adds a cocoa-and-spice flavor that can be a lot of fun in pale beers! Then there are the more intensely-flavored (Hickory, Mesquite, Walnut) and rarer (Pecan, Maple) woods - approach these with caution, but don't be afraid to experiment with them! I haven't used them enough to offer specific advice, I'm afraid.
The "how much" question goes against the grain for me (not a pun): use more than you think you need or want, except on the stronger woods noted just a moment ago. I find that subbing in smoked malt for half of your base malt(s), to start, is a good percentage. The intensity of smoke flavor in your beer is not really a function of the percentage of smoked malt in the grist: it's the level of smoking in the malt. Having said that, you'll need a sufficient amount to bring out that clear character, so you don't want to shortchange it and create a beer that's mostly "X" style, but with a "hint" of smoke. I learned this one when I was working on my Rauchbier and noticed there wasn't much change in intensity as I worked from half to 97 percent smoked malt in the grist, but there was a change in the clarity and complexity of the smoke flavor. Ordinarily I recommend starting low and building up, but in these recipes there's no correcting for "not enough smoke" down the line, so you need it all up front.
Finally, consider adding a bit (2-3% of the grist) of chocolate rye to any style that you're going to smoke but should finish dry: I've found it to be almost universally true that the added smoke presents as sweet on the palate. I had limited success balancing it with bitterness, but a touch of drying roast was perfect.
PROCESS
These malts need to be mashed, and I'm unaware of good extract options. This might be the excuse you need to upgrade to an all-grain tun, BIAB, or partial mash system!
IN CLOSING
If smoking a more-complex style, consider your secondary flavors, especially any spicy phenols or alcohols that might make your smoke seem too intense by addition. I'd also say to avoid fruity hops in favor of English or Noble varieties, but that might just be my own bias. This should get you in the ballpark, though - the rest is up to you!