Từ gt ta có x^2+y^^2=xy+1
=>P=(x^2+y^2)^2-2x^2y^2-x^2y^2
=(xy+1)2-2x2y2-x2y2
=x2y2+xy+1-3x2y2=-2x2y2+xy+1
=......
\(1=x^2+y^2-xy\ge2xy-xy=xy\Rightarrow xy\le1\)
\(1=x^2+y^2-xy\ge-2xy-xy=-3xy\Rightarrow xy\ge-\dfrac{1}{3}\)
\(\Rightarrow-\dfrac{1}{3}\le xy\le1\)
\(P=\left(x^2+y^2\right)^2-2\left(xy\right)^2-\left(xy\right)^2=\left(xy+1\right)^2-3\left(xy\right)^2=-2\left(xy\right)^2+2xy+1\)
Đặt \(xy=t\in\left[-\dfrac{1}{3};1\right]\)
\(P=f\left(t\right)=-2t^2+2t+1\)
\(f'\left(t\right)=-4t+2=0\Rightarrow t=\dfrac{1}{2}\)
\(f\left(-\dfrac{1}{3}\right)=\dfrac{1}{9}\) ; \(f\left(\dfrac{1}{2}\right)=\dfrac{3}{2}\) ; \(f\left(1\right)=1\)
\(\Rightarrow P_{max}=\dfrac{3}{2}\) ; \(P_{min}=\dfrac{1}{9}\)
