Ta có:
\(4x^2+9y^2\ge12\left|xy\right|\)
\(2x^2+18z^2\ge12\left|xz\right|\)
\(3y^2+12z^2\ge12\left|yz\right|\)
Cộng vế: \(12\left(\left|xy\right|+\left|yz\right|+\left|zx\right|\right)\le6\left(x^2+2y^2+5z^2\right)\)
\(\Leftrightarrow xy+yz+zx\le\left|xy\right|+\left|yz\right|+\left|zx\right|\le\dfrac{1}{2}\left(x^2+2y^2+5z^2\right)\)
\(\Rightarrow P\le\dfrac{1}{2}\left(x^2+2y^2+5z^2\right)\left[2+\sqrt{4-\left(x^2+2y^2+5z^2\right)^2}\right]\)
Đặt \(x^2+2y^2+5z^2=a\Rightarrow0< a\le2\)
\(P\le\dfrac{1}{2}a\left(2+\sqrt{4-a^2}\right)\Rightarrow P^2\le\dfrac{1}{4}a^2\left(2+\sqrt{4-a^2}\right)\left(2+\sqrt{4-a^2}\right)\)
\(\Rightarrow P^2\le\dfrac{1}{108}\left(a^2+2+\sqrt{4-a^2}+2+\sqrt{4-a^2}\right)^3\)
\(\Rightarrow P^2\le\dfrac{1}{108}\left(a^2+2\sqrt{4-a^2}+4\right)^3\le\dfrac{1}{108}\left(a^2+1+4-a^2+4\right)^3=\dfrac{27}{4}\)
\(\Rightarrow P\le\dfrac{3\sqrt{3}}{2}\)
Dấu "=" xảy ra khi \(a=\sqrt{3}\) hay: \(\left\{{}\begin{matrix}4x^2=9y^2=36z^2\\x^2+2y^2+5z^2=\sqrt{3}\end{matrix}\right.\) (x;y;z cùng dấu)
