do a>0, b>0 nên 1=a+b+3ab\(\ge3\sqrt[3]{3\left(ab\right)^2}\Leftrightarrow\frac{1}{3}\ge\sqrt[3]{3\left(ab\right)^2}\)
\(\Leftrightarrow\frac{1}{27}\ge3\left(ab\right)^2\Leftrightarrow\frac{1}{81}\ge\left(ab\right)^2\Leftrightarrow\frac{1}{9}\ge ab\Leftrightarrow\frac{1}{3}\ge\sqrt{ab}\)do đó
P=\(\frac{6ab}{a+b}-a^2-b^2=\frac{6ab}{a+b}-\left(a^2+b^2\right)\le\frac{6ab}{2\sqrt{ab}}-2ab=-2ab+3\sqrt{ab}=-2\left(ab-\frac{3}{2}\sqrt{ab}\right)\)
\(=-2\left[ab-2\sqrt{ab}\cdot\frac{1}{3}+\left(\frac{1}{3}\right)^2-\left(\frac{1}{3}\right)^2-\frac{5}{6}\sqrt{ab}\right]\)
\(=-2\left(\sqrt{ab}-\frac{1}{3}\right)^2+\frac{2}{9}+\frac{5}{3}\sqrt{ab}\le\frac{2}{9}+\frac{5}{3}\cdot\frac{1}{3}=\frac{7}{9}\)
vậy maxP=\(\frac{7}{9}\Leftrightarrow\hept{\begin{cases}a=b>0\\a+b+3ab=1\end{cases}\Leftrightarrow a=b=\frac{1}{3}}\)
