It would be faster for you to travel from planet to planet within a solar system because they are closer than a star to a different star, and are closer than a galaxy to another galaxy within the universe.
If you chose the same speed unit for all the same options such as the speed of light or sound:
then indeed planet to planet would be the closest.
It depends on how far away the planet is.
Light from the Sun takes eight minutes to get to Earth
But takes 5.3 hours to get to Pluto.
So Earth to say Neptune, would take about 4 hours at the speed of light.
Final Answer: Planet to Planet within a Solar System
