Download videos from Youtube and similar sites. Is it legal?

Tweet

I noticed that there are many sites allowing to download videos hosted on video streaming sites like Youtube, Google Video, Metacafe etc. I've seen that there is a software (maybe more, but I know only one, namely Arestube) that makes the same. I was wondering if this is legal. Those sites, after all, want the users to watch it on streaming, I think for revenue and video rights reasons too. Do you have infos about it? is it legal or not? Do you know what Youtube's and similar sites' policy is about this?