I like to use wood because of less algae growing on it. Usually it's just some sort of slime and fuzz that grows on my manzanita branches.
For rocks, Shou stones (the brown stone with porous areas) always gets green spot algae on it, even though there's nothing on the glass. Sometimes it adds a natural feel to it, but overall I would love it get rid of the algae, and the only way to do that is to toss the rock as well.
-John N.