Ta có :
\(P=2\left(a^2+b^2\right)-6\left(\frac{a}{b}+\frac{b}{a}\right)+9\left(\frac{1}{a^2}+\frac{1}{b^2}\right)\)
\(=2a^2+2b^2-\frac{6a}{b}+\frac{6b}{a}+\frac{9}{a^2}+\frac{9}{b^2}\)
\(=\left(\frac{3}{a^2}+3b^2\right)+\left(\frac{3}{b^2}+3a^2\right)-\left(a^2+2ab+b^2\right)-6\left(\frac{a}{b}+\frac{b}{a}\right)+6\left(2ab+\frac{1}{a^2}+\frac{1}{b^2}\right)-10ab\)
\(=\left(\frac{3}{a^2}+3b^2\right)+\left(\frac{3}{b^2}+3a^2\right)-4-6\left(\frac{a}{b}+\frac{b}{a}\right)+6\left(2ab+\frac{1}{a^2}+\frac{1}{b^2}\right)-10ab\)
Áp dụng BĐT Cô si cho các số dương ta có :
\(+,\frac{3}{a^2}+3b^2\ge2\sqrt{\frac{3}{a^2}.3b^2}=\frac{6b}{a}\left(1\right)\)
+, \(\frac{3}{b^2}+3a^2\ge2\sqrt{\frac{3}{b^2}.3a^2}=\frac{6a}{b}\left(2\right)\)
\(+,\left(\frac{a}{b}+\frac{b}{a}\right)\ge2\sqrt{\frac{a}{b}.\frac{a}{b}}=2\Leftrightarrow6\left(\frac{a}{b}+\frac{b}{a}\right)=12\left(3\right)\)
+, \(ab+ab+\frac{1}{a^2}+\frac{1}{b^2}\ge\sqrt{ab.ab.\frac{1}{a^2}.\frac{1}{b^2}}=1\Leftrightarrow6\left(ab+ab+\frac{1}{a^2}+\frac{1}{b^2}\right)=6\)
+) \(ab\ge\frac{\left(a+b\right)^2}{4}\Leftrightarrow10ab\ge10\)
Cộng vế với vế ta có :
\(P\ge10\)
Dấu "=" xảy ra \(\Leftrightarrow a=b\)