On one level I feel I covered the subject, but in retrospect there's probably no real way to "cover the subject" entirely. The pattern of building up narratives that run conjointly with one another, then meet in order to produce a meaningful and entertaining climax is the stuff which has eluded writers since Homer. It is a hard thing to do badly, and badly is mostly what people are capable of doing.
Worse, the process can be a bitch when you've got time to think about the plot line, write, contemplate the characters, rewrite, have a stroke of energy that enables you to add one more element, rewrite some more, rid the tale of all the crap and rewrite seven or eight more times. Doing this sort of thing on the fly, i.e. as the players are changing their mind about what they're doing, is the mindfuck Tolkein never had to deal with.
To get the idea, let's toss up xkcd's brilliant and mind-boggling narrative of Lord of the Rings, stolen but respecting the author's wishes as described on his own blog:
|Chart shows movie character interactions.|
The horizontal axis is time.
The vertical groupings of the lines indicates
which characters are together at a given time.
To master the idea of narrative, you have to have a strong handle on what conflict is; what separates a meaningful conflict from a frivolous conflict; and WHEN to drive a conflict onward and when to allow it to resolve itself.
Many a crappy story has started out with a conflict no one, or very few people, give a damn about. Many serials have died an exhaustive death because no resolution was ever going to occur ... or because when the resolution was finally attempted by the author, it fizzled pathetically and died. Humans, strangely enough, react viscerally to conflict. If it's done right, it hits you in the gut. If it's done half-right, you might as well dump it with tomorrow morning's garbage.
Shelves are full of ignored stories written by people who felt SURE that there was no "right way" to design a conflict. Slightly fewer exist that were written by people SURE that there was a right way, but that they were the only ones who knew it. Finally, there are stories written by people who believe there's a right way, and that the way to learn it is to stop listening to your own judgement and study up on the way its done ... a very small number of stories.
That's because it is far easier to soothe your ego by writing in a bell jar than by measuring yourself against people who are clearly smarter than you. Shakespeare, Dickens, Dafoe, Joyce, Lawrence, Austen, Maugham and Hemingway will make a new writer feel like an inadequate pipsqueak, and people DON'T like that feeling. Impotency is never popular. For myself, I am merely in awe. When asked what to do about making oneself a better story-teller, I have only one piece of advice: pick a writer that is ridiculously better than you. Then read that writer until every sentence is torn apart, and you understand what those sentences do to make the story what it is. Then look at your own sentences, force yourself to see where each one fails to measure up to its purpose and then fix it.
Do this until you bleed out of your eyes.
But I am supposed to be talking about running a D&D game, and I've let myself get hijacked into the writing process.
What D&D benefits from is that you are not telling a story which depends upon your sentence structure. The audience of your story need not worry that you haven't made the words clear the first time; you're free, in a running, to explain again and again if need be, until the full import of a particular element of your story is comprehended. This allows you to spend less time on the mechanics of writing your conflict and more time on the structure of the conflict itself.
Take the interplay of the characters in LOTR. A reasonable writer producing a book would be expected to produce some kind of character for Frodo that defines who he is and what he does. BUT YOU, O DM, do not have to do this in depth. Chances are, a character like Frodo is one of your player characters. Likewise Sam, Gandalf, Aragorn, Gimli or Legolas. As a DM, you're expected to supply characters like Denethor and Wormtongue, basically people who are mere foils - one dimensional - for the characters to play off against. Even easier, you're providing the orcs, the men and the ordinary elves, who hardly need any character at all. You might stretch yourself by creating a really interesting Gollum or Theoden, or match with a couple of interesting hobbits, but that's really dependent on how clever you are at character or how much time you have during the particular session.
What you really should concentrate on is WHERE these secondary characters, the NPCs, are at any given time in the story. Note how xkcd has extrapolated from the book the placement of Faramir, Galadriel, Bilbo and so on, where in fact we never do precisely learn what they're doing during most of the course of the story. Bilbo is out there, he's doing something, but it isn't important - and yet still we know where he is, don't we? And if Bilbo should suddenly appear next to the party, we need some kind of explanation for why he left where he was, and how he got to where he is right now.
There had better be an important reason for him doing that, hadn't there? Otherwise, he's wasting the player's time, he's wasting your time, and he's contributing nothing to the story.
Of course, there will always be those who cry from the balcony, what about red herrings, hm? Aren't red herrings important?
Preserve me. I shall probably have to write at length as to why red herrings are a terrible curse of the writing industry; why they are invariably employed by the worst writers; how they've all be used to death; and worst, how the name itself has become a justification for their use. For the moment, if you have the time, I would give a little thought to how many times have you either A) enjoyed a storyteller pointlessly deceiving you; or B) felt that the story was made better by spending precious minutes of your life pursuing literary dead ends.
The red herring is an excellent tool for making 19 minutes of television into 24 minutes of television, so as to pad around commercials. That is its greatest achievement in the science of entertainment.
If you have a solid, meaningful, cool reason for Bilbo to suddenly appear from behind the curtain at the rightest possible moment, it is going to only be about ten thousand times better for your story than Bilbo being there 'just cuze.' The gentle reader may not believe me. The gentle reader is invited to experiment with the matter.
Thus, how is it Gollum happens to be where he needs to be? He's following the ring. How is it Faramir conveniently crosses the path of Frodo's party? He's hunting bad guys up by the Black Gate. The meeting of Faramir then drags Frodo down to Osgiliath, so that the "party" can witness the orcs attacking the city, the defense thereof, the set-up for Faramir's conflict with his father (which makes sense for Gandalf and Pippin when they arrive) and the crystal moment between Frodo and the Nazgul.
In other words, the crossover isn't random. The game you play may be a sandbox, but you totally have control over the other elements/NPCs of the game, and you're intervention with an odd character like Faramir can create interesting interplays that build up your campaign. Remember that Faramir kidnaps - forcibly relocates - Frodo and his party. You're totally permitted to do the same to your characters, IF you can roll the necessary dice and IF you have a good reason for doing so.
That is, you're not a sadist or a railroader. You're just relocating them so that there are OTHER conflicts (battle of Osgiliath, appearance of the Nazgul, debate over who has the ring, etc.) for the party to play against. You haven't railroaded them ... you've just changed the scenery.
This is what I meant by having a host of NPCs who are running at the same time (in the same direction of time) as the party, but perhaps out of sight. Character crossovers are interesting - see the chart - whereas periods without crossovers are dull. The story isn't about Elrond, but if it was, it wouldn't be much of a story, except for those few interesting moments where he crosses over with the Fellowship.
You must conceive of character crossovers - alternate NPC narratives - that occur with your party. Find solid, interesting reasons for those crossovers to occur. Conceive of ways in which the crossover provides the party with vital information, changes their possible venue and inspires new and interesting conflicts.
The most common crossover in D&D is the guy at the bar talking about the monster in the nearby ruins. It's one crummy crossover that lasts for about five minutes ... and nothing else interesting happens until the party crosses a monster. I would go further and argue that this is the problem with empty dungeons that you wander through and solve puzzles. Strangers are interesting. Strangers educate and create conflict. Puzzles merely pass time.