Áp dụng bđt cô si ta có:
\(a^2+2b^2+3=\left(a^2+b^2\right)+\left(b^2+1\right)+2\ge2ab+2b+2=2\left(ab+b+1\right)\)
\(b^2+2c^2+3\ge2\left(bc+c+1\right)\)
\(c^2+2a^2+3\ge2\left(ac+a+1\right)\)
=> \(M\le\frac{1}{2}\left(\frac{1}{ab+b+1}+\frac{1}{bc+c+1}+\frac{1}{ca+a+1}\right)\)
\(=\frac{1}{2}\left(\frac{1}{ab+b+1}+\frac{ab}{bcab+abc+ab}+\frac{b}{abc+ab+b}\right)\)
\(=\frac{1}{2}\left(\frac{1}{ab+b+1}+\frac{ab}{b+1+ab}+\frac{b}{1+ab+b}\right)\)
\(=\frac{1}{2}.\frac{ab+b+1}{ab+b+1}=\frac{1}{2}\)
Bổ sung:
Dấu "=" xảy ra <=> a = b = c = 1
Vậy GTLN của M = 1/2 tại a = b = c = 1.
