Theo hệ thức vi-et ta có : \(\left\{{}\begin{matrix}x_1+x_2=-\frac{b}{a}\\x_1x_2=\frac{c}{a}\end{matrix}\right.\)
\(P=\frac{5a^2-6ab+b^2}{2a^2-2ab+ac}=\frac{5-\frac{6b}{a}+\frac{b^2}{a^2}}{2-\frac{2b}{a}+\frac{c}{a}}=\frac{5+6\left(x_1+x_2\right)+\left(x_1+x_2\right)^2}{2+2\left(x_1+x_2\right)+x_1x_2}\)
Mặt khác :
\(\left\{{}\begin{matrix}x_1\le x_2\\x_2\le1\end{matrix}\right.\Rightarrow\left\{{}\begin{matrix}x_1^2\le x_1x_2\\x_2^2\le1\end{matrix}\right.\Rightarrow x_1^2+x_2^2\le x_1x_2+1\Rightarrow\left(x_1+x_2\right)^2\le3x_1x_2+1\)
\(\Rightarrow P\le\frac{6+6\left(x_1+x_2\right)+3x_1x_2}{2+2\left(x_1+x_2\right)+x_1x_2}=3\)