\(\Delta=\left(3sina-cosa\right)^2+16\left(cos2a+1\right)\)
\(=8sin^2a-6sina.cosa+16cos2a+17\)
\(=21-3sin2a+12cos2a>0\) ; \(\forall a\)
Theo Viet: \(\left\{{}\begin{matrix}x_1+x_2=3sina-cosa\\x_1x_2=-4-4cos2a\end{matrix}\right.\)
\(P=x_1^2+x_2^2=\left(x_1+x_2\right)^2-2x_1x_2\)
\(=9sin^2a+cos^2a-6sina.cosa+8+8cos2a\)
\(=13-3sin2a+4cos2a\)
\(=13-5\left(\frac{3}{5}sin2a-\frac{4}{5}cos2a\right)=13-5sin\left(2a+b\right)\)
Với \(\left\{{}\begin{matrix}sinb=\frac{4}{5}\\cosb=\frac{3}{5}\end{matrix}\right.\)
Do \(-1\le sin\left(2a+b\right)\le1\)
\(\Rightarrow8\le P\le18\)