World building means different things to different writers, I believe. To me, it means simply creating a believable setting in which a story can take place. In that sense, world building is paramount, crucial, and a story can only suffer in its absence.
However, applying world building to an adventure story must follow different rules from the ones established in turn of the century literature or epic fantasy, I think. In those kinds of stories, readers are often looking for a more verbose sense of writing style or a "grander" way to telling and/or showing the tale. But in adventure writing, readers don't want a lot of description of place and the symbolism of a particular color of curtains to get in the way of the more quickly moving action.
But, the little details you choose as a writer can and will make the setting more real, and should. If a setting feels generic, I know that I've failed as a writer. A reader almost needs to be able to smell the smoke from the Jazz club or feel the desolate rocky surfaces of the lost valley in order to really emote during reading.
Perhaps three of the best examples of setting made real through details are:
- Philip Marlowe's Los Angeles. Sure, it's a real place already, but it feels more real when Chandler writes it.
- Gotham City. I'm convinced that place is completely real. It HAS to be.
- Opal City from the pages of James Robinson's Starman series -- masterfully skilled world building in that one. Jack Knight wouldn't be Jack Knight if he lived any other place.