Đặt \(\left\{{}\begin{matrix}x=sina\\y=cosa\end{matrix}\right.\)
\(P=\frac{2-2sina.cosa+cos^2a}{4sin^2a-3sina.cosa+cos^2a}=\frac{2-sin2a+\frac{1+cos2a}{2}}{1+\frac{3\left(1-cos2a\right)}{2}-\frac{3}{2}sin2a}=\frac{5-2sin2a+cos2a}{5-3cos2a-3sin2a}\)
\(\Leftrightarrow3P-3P.cos2a-3P.sin2a=5-2sin2a+cos2a\)
\(\Leftrightarrow\left(3P-2\right)sin2a+\left(3P+1\right)cos2a=5P-5\)
Áp dụng BĐT Bunhiacopxki:
\(\left(5P-5\right)^2\le\left(3P-2\right)^2+\left(3P+1\right)^2\)
\(\Leftrightarrow7P^2-44P+20\le0\)
Theo Viet: \(\left\{{}\begin{matrix}M+n=\frac{44}{7}\\Mn=\frac{20}{7}\end{matrix}\right.\)
\(\Rightarrow M^2+n^2=\left(M+n\right)^2-4Mn=\frac{1376}{49}\)