Trước hết ta chứng minh bài toán quen thuộc:
Cho \(abc=1\) thì \(\frac{1}{ab+b+1}+\frac{1}{bc+c+1}+\frac{1}{ca+a+1}=1\)
\(VT=\frac{1}{ab+b+1}+\frac{1}{bc+c+abc}+\frac{b}{abc+ab+b}=\frac{1}{ab+b+1}+\frac{1}{c\left(b+1+ab\right)}+\frac{b}{1+ab+b}\)
\(=\frac{1}{ab+b+1}+\frac{ab}{b+1+ab}+\frac{b}{1+ab+b}=\frac{1+ab+b}{ab+b+1}=1\)
\(P=\sum\frac{1}{a^2+2b^2+3}=\sum\frac{1}{a^2+b^2+b^2+1+2}\le\sum\frac{1}{2ab+2b+2}=\frac{1}{2}\sum\frac{1}{ab+b+1}=\frac{1}{2}\)
\(\Rightarrow P_{max}=\frac{1}{2}\) khi \(a=b=c=1\)
\(P=\sum\frac{1}{a^2+1+2\left(b^2+1\right)}\le\sum\frac{1}{2a+4b}=\frac{1}{2}\sum\frac{1}{a+b+b}\le\frac{1}{18}\sum\left(\frac{1}{a}+\frac{2}{b}\right)\)
\(\Rightarrow P\le\frac{1}{18}\left(\frac{3}{a}+\frac{3}{b}+\frac{3}{c}\right)=\frac{1}{6}\left(\frac{1}{a}+\frac{1}{b}+\frac{1}{c}\right)\le\frac{1}{6}.3\sqrt[3]{\frac{1}{abc}}=\frac{1}{2}\)
\(\Rightarrow P_{max}=\frac{1}{2}\) khi \(a=b=c=1\)
