Đặt \(\left\{{}\begin{matrix}\sqrt{a}=x\\\sqrt{b}=y\end{matrix}\right.\) \(\Rightarrow0\le x;y\le1\)
\(P=2x^2y-xy^2=xy\left(2x-y\right)\)
- Với \(2x\le y\Rightarrow P\le0\)
- Với \(2x>y\Rightarrow P>0\)
Khi đó: \(P=xy\left(2x-y\right)\le\left(\frac{x+y+2x-y}{3}\right)^3=x^3\le1\)
\(P_{max}=1\) khi \(\left\{{}\begin{matrix}x=1\\x=y=2x-y\end{matrix}\right.\) \(\Rightarrow x=y=1\)