Ta có : \(M=\dfrac{1}{a+b^2}+\dfrac{1}{b+a^2}=\dfrac{a+1}{\left(a+b^2\right)\left(a+1\right)}+\dfrac{b+1}{\left(b+1\right)\left(b+a^2\right)}\le\dfrac{a+1}{\left(a+b\right)^2}+\dfrac{b+1}{\left(a+b\right)^2}=\dfrac{1}{a+b}+\dfrac{2}{\left(a+b\right)^2}\le\dfrac{1}{2}+\dfrac{2}{4}=1\)đẳng thức xả ra khi và chỉ khi a=b=1. Do đó GTLN của M là 1.

\(a^2-ab+b^2=\dfrac{1}{4}\left(a+b\right)^2+\dfrac{3}{4}\left(a-b\right)^2\ge\dfrac{1}{4}\left(a+b\right)^2\)

\(\Rightarrow P\le\dfrac{2}{a+b}+\dfrac{2}{b+c}+\dfrac{2}{c+a}\le\dfrac{1}{a}+\dfrac{1}{b}+\dfrac{1}{c}=3\)

Dấu "=" xảy ra khi \(a=b=c=1\)

\(\dfrac{1}{\left(a+b+a+c\right)^2}\le\dfrac{1}{4\left(a+b\right)\left(a+c\right)}=\dfrac{1}{4\left(a^2+ab+bc+ca\right)}\le\dfrac{1}{64}\left(\dfrac{1}{a^2}+\dfrac{1}{ab}+\dfrac{1}{bc}+\dfrac{1}{ca}\right)\)

\(\le\dfrac{1}{64}\left(\dfrac{1}{a^2}+\dfrac{1}{a^2}+\dfrac{1}{b^2}+\dfrac{1}{c^2}\right)=\dfrac{1}{64}\left(\dfrac{2}{a^2}+\dfrac{1}{b^2}+\dfrac{1}{c^2}\right)\)

Tương tự và cộng lại:

\(P\le\dfrac{1}{64}\left(\dfrac{4}{a^2}+\dfrac{4}{b^2}+\dfrac{4}{c^2}\right)=\dfrac{1}{16}.3=\dfrac{3}{16}\)

Dấu "=" xảy ra khi \(a=b=c=1\)

Áp dụng bđt: \(\dfrac{1}{x+y}\le\dfrac{1}{4}\left(\dfrac{1}{x}+\dfrac{1}{y}\right)\left(1\right)\)

\(\dfrac{1}{2a+b+c}=\dfrac{1}{\left(a+b\right)+\left(a+c\right)}\le\dfrac{1}{4}\left(\dfrac{1}{a+b}+\dfrac{1}{a+c}\right)\)

\(\Rightarrow P\le\dfrac{1}{16}\left[\left(\dfrac{1}{a+b}+\dfrac{1}{a+c}\right)^2+\left(\dfrac{1}{a+b}+\dfrac{1}{b+c}\right)^2+\left(\dfrac{1}{b+c}+\dfrac{1}{a+c}\right)^2\right]\)\(\Rightarrow16P\le\dfrac{2}{\left(a+b\right)^2}+\dfrac{2}{\left(b+c\right)^2}+\dfrac{2}{\left(a+c\right)^2}+\dfrac{2}{\left(a+b\right)\left(b+c\right)}+\dfrac{2}{\left(a+b\right)\left(b+c\right)}+\dfrac{2}{\left(b+c\right)\left(c+a\right)}\)

Áp dụng: \(x^2+y^2+z^2\ge xy+yz+xz\left(2\right)\) với a+b=x,b+c=y,c+a=z

\(\Rightarrow16P\le\dfrac{4}{\left(a+b\right)^2}+\dfrac{4}{\left(b+c\right)^2}+\dfrac{4}{\left(c+a\right)^2}\)

Ta có: \(\dfrac{1}{\left(a+b\right)^2}\le4.16.\left(\dfrac{1}{a}+\dfrac{1}{b}\right)^2\)(do (1))

\(\Rightarrow16P\le\dfrac{1}{4}.16\left[\left(\dfrac{1}{a}+\dfrac{1}{b}\right)^2+\left(\dfrac{1}{b}+\dfrac{1}{c}\right)^2+\left(\dfrac{1}{c}+\dfrac{1}{a}\right)^2\right]=\dfrac{1}{4}\left(\dfrac{1}{a^2}+\dfrac{1}{b^2}+\dfrac{1}{c^2}+\dfrac{2}{ab}+\dfrac{2}{bc}+\dfrac{2}{ca}\right)\le\dfrac{1}{4}.4.\left(\dfrac{1}{a^2}+\dfrac{1}{b^2}+\dfrac{1}{c^2}\right)=3\)(do(2) và \(\dfrac{1}{a^2}+\dfrac{1}{b^2}+\dfrac{1}{c^2}=3\))

\(\Rightarrow P\le\dfrac{3}{16}\)

\(ĐTXR\Leftrightarrow a=b=c=1\)

Lời giải:

Áp dụng BĐT AM-GM:

\(P=\frac{2a}{\sqrt{a^2+ab+bc+ac}}+\frac{b}{\sqrt{b^2+ab+bc+ac}}+\frac{c}{\sqrt{c^2+ab+bc+ac}}\\ =\frac{2a}{\sqrt{(a+b)(a+c)}}+\frac{b}{\sqrt{(b+c)(b+a)}}+\frac{c}{\sqrt{(c+a)(c+b)}}\)

\(\leq \frac{a}{a+b}+\frac{a}{a+c}+\frac{b}{4(b+c)}+\frac{b}{b+a}+\frac{c}{4(c+b)}+\frac{c}{c+a}\)

\(=(\frac{a}{a+b}+\frac{b}{b+a})+(\frac{a}{a+c}+\frac{c}{a+c})+\frac{1}{4}(\frac{b}{b+c}+\frac{c}{b+c})=1+1+\frac{1}{4}=\frac{9}{4}\)

Vậy $P_{\max}=\frac{9}{4}$

Áp dụng bđt Cauchy - Schwarz ta có:\(Q=\dfrac{2-2a^2b^2}{\left(1+a^2\right)\left(1+b^2\right)}+\dfrac{2}{\sqrt{1+c^2}}=\dfrac{2\left(1-ab\right)\left(1+ab\right)}{\left(ab+bc+ca+a^2\right)\left(ab+bc+ca+b^2\right)}+\dfrac{2}{\sqrt{1+c^2}}=\dfrac{2\left(bc+ca\right)\left(1+ab\right)}{\left(a+b\right)^2\left(b+c\right)\left(c+a\right)}+\dfrac{2}{\sqrt{1+c^2}}=\dfrac{2c\left(1+ab\right)}{\left(a+b\right)\left(b+c\right)\left(c+a\right)}=\dfrac{2c\left(1+ab\right)}{\sqrt{\left(a^2+1\right)\left(b^2+1\right)\left(c^2+1\right)}}+\dfrac{2}{\sqrt{1+c^2}}\le\dfrac{2c\left(1+ab\right)}{\sqrt{\left(ab+1\right)^2\left(c^2+1\right)}}+\dfrac{2}{\sqrt{1+c^2}}=\dfrac{2c}{\sqrt{c^2+1}}+\dfrac{2}{\sqrt{c^2+1}}=\dfrac{2\left(c+1\right)}{\sqrt{c^2+1}}\le\dfrac{2\left(c+1\right)}{\sqrt{\dfrac{\left(c+1\right)^2}{2}}}=2\sqrt{2}\)Dấu "=" xảy ra khi a = b = \(\sqrt{2}-1;c=1\).

Vậy..

Ta có \(\dfrac{1}{\sqrt{a}}+\dfrac{1}{\sqrt{b}}=2\Leftrightarrow\dfrac{1}{a}+\dfrac{1}{b}+\dfrac{2}{\sqrt{ab}}=4\)

\(\Leftrightarrow\dfrac{1}{a}+\dfrac{1}{b}=4-\dfrac{2}{\sqrt{ab}}\)

Khi đó P = \(\dfrac{1}{\sqrt{ab}}\left(4-\dfrac{2}{\sqrt{ab}}\right)=-2\left(\dfrac{1}{\sqrt{ab}}-1\right)^2+2\le2\)

Dấu "=" khi a = b = 1 

\(\left(a+b^2\right)\left(a+1\right)\ge\left(a+b\right)^2\Rightarrow\dfrac{1}{a+b^2}\le\dfrac{a+1}{\left(a+b\right)^2}\)

Tương tự: \(\dfrac{1}{b+a^2}\le\dfrac{b+1}{\left(a+b\right)^2}\)

\(\Rightarrow M\le\dfrac{a+b+2}{\left(a+b\right)^2}=\dfrac{2}{\left(a+b\right)^2}+\dfrac{1}{a+b}=\dfrac{2}{\left(a+b\right)^2}+\dfrac{1}{a+b}-1+1\)

\(\Rightarrow M\le\left(\dfrac{2}{a+b}-1\right)\left(\dfrac{1}{a+b}+1\right)+1=\left(\dfrac{2-a-b}{a+b}\right)\left(\dfrac{1}{a+b}+1\right)+1\le1\)

\(M_{max}=1\) khi \(a=b=1\)