I think it has to do with the constant walking around in sandals and a yoga mat, but his hair probably isn't helping.
Here's what he said on the matter:
"I'd like to emphasize this point if i may, sir, using your television program: I'm very clean! People think for like some reason, possibly through some of my nocturnal habits, that I may be unhygienic. I am not. I'm very well groomed. and very immaculately clean and as Jimmy has just verified, I smell nice."
