It is in their TOS
http://www.youtube.com/t/terms, however the specific term would be near impossible to hold up, since by the very nature of how youtube works, it REQUIRES you to download a copy of the data. As such it's not realistically enforcable.
But there's also a "don't download the data by a means other than the youtube player" clause in there as well (6.C.) which is more enforcable.
But there's been a host of plugins around for doing this sort of thing on Firefox for years, it's not new and it's not happening often enough for youtube to care, so it's a non-issue really.
As others have stated the quality of tracks using this method is pretty bad, whether it's bad enough to significantly affect the track shape I don't know (I think that's something that varies a lot between tracks), but the only reason I can think of for using this is maybe to see what a track would be like if you're interested in it for the express purpose of playing it on Audiosurf (very small market I'd say).
I do find it quite amusing that Lavos^ in the same post suggests torrenting tracks instead then immediately points out that downloading tracks like this from youtube is a breach of the TOS.
