There is a very good reason that sandbox play doesn't work. It is bound up with the same reason why D&D will never become a truly widespread game.
People suck at it.
This is not a fault of the game itself, or of the rule books failure to properly describe how the game should be played. I agree that there are inadequacies there, but the best written description slash how-to guide on how to role-play would still fail to lift many an individual's ability to play this game. The limitation exists in an much larger context, that of life itself, and the way that people approach life.
People suck at that, too.
This is a rather large accusation to make, and will be received contentiously by most, with consideration by some, and with vigorous head-nodding by a very, very few. I don't propose to convince anyone from Group One - this is the sort that will never play this game, or will cheerfully play it without much introspection, very much the way they move through their lives. I don't need to convince Group Three; they already know exactly what I'm talking about, and are reading from this point on only to feel comforted that they are not alone in the world. Group Two could go one way or the other.
Role-play, particularly in the sandbox mode, is the process of assessing one's surroundings and then taking action upon that assessment. The wider the scope of the game, the more options one has to assess and choose from, and the less certain the outcome following any given action. Much is dependent upon the DM; a vindictive adjudicator will tip the balance against the player and reduce the willingness of players to take any action at all. A benevolent adjudicator will typically highlight what actions will yield the best returns, or alternately might ensure that any action would guarantee a reward. Either, taken to the extreme, makes for a bad gaming experience, but that's not important right now.
If I might suggest, real life is a cold, callous condition of existence, albeit deeply compelling and at times unimaginable blessed. I remember a bite from Spalding Gray's film, Swimming to Cambodia, that was attributed to Athol Fugard: "Spalding! The sea's a lovely lady when you play in her ... but if you play with her, she's a BITCH! Play in the sea, yes, but never play with her. You're lucky to be here! You're lucky to be alive!"
My point would be that nature is as heartless a DM as it gets - and setting out to assess the real world and take action in the real world has consequences that no RPG can offer. Which would be the principal reason why most people occupying their bit of space in the day-to-day world don't assess their situation and they don't take action.
That is ... with a sandbox frame of mind.
I don't know about the gentle reader, but my parents were very much of the railroading mentality where it came to playing the Great RPG of Life. Get your education, get your job, get your partner, get your house, get your benefits, raise the kids, get your retirement and get your gravestone. This is the game as outlined to virtually every middle-class child, with public relations support from everywhere, from the teacher and principle right through the minister and soccer coach. And sad to say, this is the game as people play it.
There are benefits to playing the game this way. The education gets you the reputation, which gets you the job, which gives you the money to impress the spouse and to buy the house. Keep your head down, don't make waves, don't step out of the box and you will get your benefits and you will get your retirement and eventual gravestone. No promises on what sort of assholes your children will turn out to be, though.
Plugging yourself into the mindset makes a really lousy RPG player, however. Adventure is not a plug-and-play routine; it isn't clear cut and there are no road signs. It requires a frame of mind that is prepared to dispute the PR and risk the path not taken. And for those people who have thought along the same mundane life patterns since being teenagers, it isn't likely they're going to understand D&D. Hell, they have trouble understanding the motivations of movie characters, for gossakes.
I'm going a long way around the barn here, but look ... ordinarily, we have our days planned out for us, even if we do the planning ourselves. We know we have to be at work if this is Monday. We know we have to mow the lawn if this is Sunday, because we know it has to be done before the weekend is over. We know that this being Saturday, it's the only practical day for us to go to the mountains if we want. And we know that this being Friday, we can drink hard and have plenty of time to recover before we have to work again.
We take the vacation when we're allowed - but every year that is simultaneously the most fun we have and the worst trouble we have ... because it's the small part of our lives we spend outside the box. The hotel isn't what it was supposed to be, the flight is delayed, the island is being hit by a hurricane this weekend, the window of our car was smashed while we were parked at the beach and so on. The relief we feel upon getting back home, where life makes sense again, describes our perception of actual role-play right down to the ground.
Which may be the reason why so many D&D players are social drop-outs. We are not success-focused, we teach our kids how to play the game Friday nights rather than getting drunk, our university backgrounds tend to be more 'liberal' than practical and half the time we're either looking for work or starting a new job. I don't speak for every reader, obviously ... but there's a connection to be made between having a fascination with the game and living your life on the fly. I think that many players - those that I've met, certainly - are the sort that if circumstances had been just right, we would have been Jason Bourne; but we were sick the day the C.I.A. checked out our high school (here in Canada, that would be C.S.I.S.). There's a certain unrealized potential that stalks us - or so we believe - and we make up for that by playing the game.
Those who are happy in their rut will never understand that motivation. In fact, they will always be a bit fearful of it. They have learned to associate anything new with stress ... and thus they would find themselves at the gaming table feeling uncomfortable. They don't rise to the bait of an uncertain situation - instead, it makes them self-conscious of their own perceived stupidity. They would rather return to their gardens, their pets and their summer cottages, where no measurement is invoked to indicate the success of their contribution - measurements such as character progression or character death.
We are strange people who find an urge to gauge our weekend by such things. We have no idea how strange we are.