What you need to do is work using radians instead of using degrees.
[tex]\sin (\frac{5\pi}{3})=-\frac{\sqrt[]{3}}{2}[/tex][tex]\cos (\frac{5\pi}{3})=\frac{1}{2}[/tex]You could write the first one as:
[tex]Cos(\frac{5\pi}{3})=\text{-}(3)^{(1/2)}/2[/tex]