Lời giải:
Áp dụng BĐT AM-GM ta có:
\(2a+b+c=(a+b)+(a+c)\geq 2\sqrt{(a+b)(a+c)}\)
\(\Rightarrow (2a+b+c)^2\geq 4(a+b)(a+c)\)
\(\Rightarrow \frac{1}{(2a+b+c)^2}\leq \frac{1}{4(a+b)(a+c)}\)
Hoàn toàn tương tự với các phân thức còn lại suy ra:
\(P\leq \frac{1}{4}\left(\frac{1}{(a+b)(a+c)}+\frac{1}{(b+c)(b+a)}+\frac{1}{(c+a)(c+b)}\right)\)
\(\Leftrightarrow P\leq \frac{1}{4}.\frac{(b+c)+(c+a)+(a+b)}{(a+b)(b+c)(c+a)}\)
\(\Leftrightarrow P\leq \frac{a+b+c}{2(a+b)(b+c)(c+a)}\)
Lại có: \((a+b)(b+c)(c+a)\geq 2\sqrt{ab}.2\sqrt{bc}.2\sqrt{ac}=8abc\) (theo AM-GM)
\(\Rightarrow P\leq \frac{a+b+c}{2.8abc}=\frac{a+b+c}{16abc}(1)\)
Tiếp tục áp dụng BĐT AM-GM:
\(\frac{1}{a^2}+\frac{1}{b^2}\geq \frac{2}{ab}; \frac{1}{b^2}+\frac{1}{c^2}\geq \frac{2}{bc}; \frac{1}{c^2}+\frac{1}{a^2}\geq \frac{2}{ac}\)
\(\Rightarrow 2\left(\frac{1}{a^2}+\frac{1}{b^2}+\frac{1}{c^2}\right)\geq 2\left(\frac{1}{ab}+\frac{1}{bc}+\frac{1}{ac}\right)\)
\(\Leftrightarrow 3\geq \frac{1}{ab}+\frac{1}{bc}+\frac{1}{ca}=\frac{a+b+c}{abc}\)
\(\Rightarrow a+b+c\leq 3abc(2)\)
Từ \((1); (2)\Rightarrow P\leq \frac{3abc}{16abc}=\frac{3}{16}\)
Vậy \(P_{\max}=\frac{3}{16}\). Dấu bằng xảy ra khi \(a=b=c=1\)