[tex]\dfrac{\mathrm dQ}{\mathrm dt}=\dfrac{a(1-5bt^2)}{(1+bt^2)^4}[/tex]
[tex]Q(t)=\displaystyle\int\frac{a(1-5bt^2)}{(1+bt^2)^4}\,\mathrm dt[/tex]
Let [tex]t=\dfrac1{\sqrt b}\tan u[/tex], so that [tex]\mathrm dt=\dfrac1{\sqrt b}\sec^2u\,\mathrm du[/tex]. Then the integral becomes
[tex]\displaystyle\frac a{\sqrt b}\int\frac{1-5b\left(\frac1{\sqrt b}\tan u\right)^2}{\left(1+b\left(\frac1{\sqrtb}\tan u\right)^2\right)^4}\sec^2u\,\mathrm du[/tex]
[tex]=\displaystyle\frac a{\sqrt b}\int\frac{1-5\tan^2u}{(1+\tan^2u)^4}\sec^2u\,\mathrm du[/tex]
[tex]=\displaystyle\frac a{\sqrt b}\int\frac{1-5\tan^2u}{\sec^6u}\,\mathrm du[/tex]
[tex]=\displaystyle\frac a{\sqrt b}\int(\cos^6u-5\sin^2u\cos^4u)\,\mathrm du[/tex]
[tex]=\displaystyle\frac a{\sqrt b}\int(6\cos^2u-5)\cos^4u\,\mathrm du[/tex]
One way to proceed from here is to use the power reduction formula for cosine:
[tex]\cos^{2n}x=\left(\dfrac{1+\cos2x}2\right)^n[/tex]
You'll end up with
[tex]=\displaystyle\frac a{16\sqrt b}\int(5\cos2u+8\cos4u+3\cos6u)\,\mathrm du[/tex]
[tex]=\displaystyle\frac a{16\sqrt b}\left(\frac52\sin2u+2\sin4u+\frac12\sin6u\right)+C[/tex]
[tex]=\dfrac a{32\sqrt b}(5\sin2u+4\sin4u+\sin6u)+C[/tex]
[tex]Q(t)=\dfrac{at}{(1+bt^2)^3}+C[/tex]