\(\left(x^2-3x\right)^2-14x^2+42x+40\\ =\left(x^2-3x-7\right)^2-9\\ =\left(x^2-3x-10\right)\left(x^2-3x-4\right)\)
\(\left(x^2-3x\right)^2-14x^2+42x+40\)
\(=\left(x^2-3x-4\right)\left(x^2-3x-10\right)\)
\(=\left(x-4\right)\left(x+1\right)\left(x-5\right)\left(x+2\right)\)