Answer:
3x+1.
Step-by-step explanation:
First we divide g(x)/f(x) (the process is in the first image):
5x-15 in Z7[x] is 5x-1 and [tex]17x^2+86x+16[/tex] is [tex]r(x)= 3x^2+2x+2[/tex] in Z7[x]. So
g(x)/f(x) = [tex](5x-1)(x^3+3x^2+6x+1)+3x^2+2x+2[/tex]
Now gcd(g,f) = gcm(f,r).
f(x)/r(x) = [tex]5x(3x^2+2x+2) + 3x+1[/tex]
Then, gcd(f,r) = gcd(r,3x+1).
r/(3x+1) = [tex](x+5)(3x+1) +4[/tex]
Then, gcd(r, 3x+1) = gcd(3x+1,4) = 3x+1.
So, gcd(f,g) = 3x+1.