The obvious substitution is
[tex]\begin{cases}u=x-4y\\v=7x-y\end{cases}[/tex]
so that the limits of integration in the [tex]u,v[/tex]-plane are constant.
We can solve the above system for [tex]x,y[/tex] to get
[tex]\begin{cases}x=-\frac1{27}(u-4v)\\y=-\frac1{27}(7u-v)\end{cases}[/tex]
The Jacobian for this change of coordiantes is
[tex]J=\begin{bmatrix}\dfrac{\partial x}{\partial u}&\dfrac{\partial x}{\partial v}\\\\\dfrac{\partial y}{\partial u}&\dfrac{\partial y}{\partial v}\end{vmatrix}=-\dfrac1{27}\begin{bmatrix}1&-4\\7&-1\end{bmatrix}[/tex]
so that the differential becomes
[tex]\mathrm dx\,\mathrm dy=|\mathrm{det}(J)|\,\mathrm du\,\mathrm dv=\dfrac1{27}\,\mathrm du\,\mathrm dv[/tex]
Now,
[tex]\displaystyle\iint_R(7x-4y)(7x-y)\,\mathrm dA=\iint_R(6x+(x-4y))(7x-y)\,\mathrm dA[/tex]
[tex]=\displaystyle\frac1{27}\int_1^4\int_0^5\left(-\frac6{27}(u-4v)+u\right)v\,\mathrm du\,\mathrm dv[/tex]
[tex]=\displaystyle\frac1{243}\int_1^4\int_0^5(7u+8v)v\,\mathrm du\,\mathrm dv[/tex]
[tex]=\displaystyle\frac1{243}\int_1^4\left(\frac{175}2v+40v^2\right)\,\mathrm dv=\boxed{\frac{665}{108}}[/tex]