Refer to the diagram attached (created with GeoGebra). Consider the 5 N force in two directions: parallel to the 8 N force and normal to the 8 N force.
The sum of forces on each direction will be the resultant force on that direction:
Apply the Pythagorean Theorem to find the magnitude of the resultant force.
[tex]\displaystyle \Sigma F = \sqrt{{(8 + \dfrac{5\sqrt{2}}{2})}^2 + {(\dfrac{5\sqrt{2}}{2})}^2} = 12.1\;\text{N}[/tex] (3 sig. fig.).
The size of the angle between the resultant force and the 8 N force can be found from the tangent value of the angle. Tangent of the angle:
[tex]\displaystyle \dfrac{\Sigma F_\text{Normal}}{\Sigma F_\text{Parallel}} = \dfrac{8 + \dfrac{5\sqrt{2}}{2}}{\dfrac{5\sqrt{2}}{2}} \approx 0.306491[/tex].
Find the size of the angle using inverse tangent:
[tex]\displaystyle \arctan{ \dfrac{\Sigma F_\text{Normal}}{\Sigma F_\text{Parallel}}} = \arctan{0.306491} = 17.0\textdegree[/tex].
In other words, the resultant force is 17.0° relative to the 8 N force.