Lời giải:
Áp dụng BĐT Bunhiacopxky:
\(\text{VT}^2\leq (a^2+b^2)(1+a+1+b)=a+b+2\)

Áp dụng BĐT Cô-si:

\((a+b)^2\leq 2(a^2+b^2)=2\Rightarrow a+b\leq \sqrt{2}\)

Do đó: $\text{VT}^2\leq 2+\sqrt{2}$

$\Rightarrow \text{VT}\leq \sqrt{2+\sqrt{2}}$ (đpcm)

Dấu "=" xảy ra khi $a=b=\frac{1}{\sqrt{2}}$

Áp dụng BĐT cosi:

\(a\sqrt{1-b^2}=\sqrt{a^2\left(1-b^2\right)}\le\dfrac{a^2+1-b^2}{2}\)

Tương tự cx có: \(b\sqrt{1-c^2}\le\dfrac{b^2+1-c^2}{2}\)

\(c\sqrt{1-a^2}\le\dfrac{c^2+1-a^2}{2}\)

Cộng vế với vế \(\Rightarrow VT\le\dfrac{3}{2}\)

Dấu = xảy ra <=> \(\left\{{}\begin{matrix}a^2=1-b^2\\b^2=1-c^2\\c^2=1-a^2\end{matrix}\right.\) \(\Leftrightarrow a^2+b^2+c^2=3-\left(a^2+b^2+c^2\right)\)

\(\Leftrightarrow a^2+b^2+c^2=\dfrac{3}{2}\) (đpcm)

Đặt \(\left(\sqrt{a};\sqrt{b};\sqrt{c}\right)=\left(x^2;y^2;z^2\right)\) với \(x;y;z>0\Rightarrow xyz=1\)

Đặt vế trái của BĐT cần chứng minh là P

Ta có: \(P=\dfrac{1}{x^2+2y^2+3}+\dfrac{1}{y^2+2z^2+3}+\dfrac{1}{z^2+2x^2+3}\)

\(P=\dfrac{1}{\left(x^2+y^2\right)+\left(y^2+1\right)+2}+\dfrac{1}{\left(y^2+z^2\right)+\left(z^2+1\right)+2}+\dfrac{1}{\left(z^2+x^2\right)+\left(x^2+1\right)+2}\)

\(P\le\dfrac{1}{2xy+2y+2}+\dfrac{1}{2yz+2z+2}+\dfrac{1}{2zx+2x+2}\)

\(P\le\dfrac{1}{2}\left(\dfrac{1}{xy+y+1}+\dfrac{1}{yz+z+1}+\dfrac{1}{zx+x+1}\right)=\dfrac{1}{2}\left(\dfrac{1}{xy+y+1}+\dfrac{xyz}{yz+z+xyz}+\dfrac{y}{xyz+xy+y}\right)\)

\(P\le\dfrac{1}{2}\left(\dfrac{1}{xy+y+1}+\dfrac{xy}{y+1+xy}+\dfrac{y}{1+xy+y}\right)=\dfrac{1}{2}\) (đpcm)

Dấu "=" xảy ra khi \(x=y=z=1\) hay \(a=b=c=1\)

\(2a^2+5b^2+2ab=1\Leftrightarrow\left(a-b\right)^2+\left(a+2b\right)^2=1\)

Đặt \(P=\dfrac{a-b}{a+2b+2}\Rightarrow P\left(a+2b\right)+2P=a-b\)

\(\Rightarrow2P=\left(a-b\right)-P\left(a+2b\right)\)

\(\Rightarrow4P^2=\left[\left(a-b\right)-P\left(a+2b\right)\right]^2\le\left(P^2+1\right)\left[\left(a-b\right)^2+\left(a+2b\right)^2\right]=P^2+1\)

\(\Rightarrow3P^2\le1\Rightarrow-\dfrac{1}{\sqrt{3}}\le P\le\dfrac{1}{\sqrt{3}}\)

\(\dfrac{a}{\sqrt{a^2+1}}=\dfrac{a}{\sqrt{a^2+ab+ac+bc}}=\dfrac{a}{\sqrt{\left(a+b\right)\left(a+c\right)}}\le\dfrac{a}{2}\left(\dfrac{1}{a+b}+\dfrac{1}{a+c}\right)=\dfrac{1}{2}\left(\dfrac{a}{a+b}+\dfrac{a}{a+c}\right)\) Chứng minh tương tự ta được:

\(\dfrac{b}{\sqrt{b^2+1}}\le\dfrac{1}{2}\left(\dfrac{b}{b+a}+\dfrac{b}{b+c}\right);\dfrac{c}{\sqrt{c^2+1}}\le\dfrac{1}{2}\left(\dfrac{c}{c+a}+\dfrac{c}{c+b}\right)\)

\(\Rightarrow\dfrac{a}{\sqrt{a^2+1}}+\dfrac{b}{\sqrt{b^2+1}}+\dfrac{c}{\sqrt{c^2+1}}\le\dfrac{1}{2}\left(\dfrac{a}{a+b}+\dfrac{a}{a+c}+\dfrac{b}{b+a}+\dfrac{b}{b+c}+\dfrac{c}{c+a}+\dfrac{c}{c+b}\right)=\dfrac{1}{2}\left(\dfrac{a+b}{a+b}+\dfrac{b+c}{b+c}+\dfrac{c+a}{c+a}\right)=\dfrac{1}{2}\left(1+1+1\right)=\dfrac{3}{2}\) Dấu = xảy ra \(\Leftrightarrow a=b=c=\dfrac{1}{\sqrt{3}}\)

\(\dfrac{a}{\sqrt{a^2+1}}=\dfrac{a}{\sqrt{a^2+ab+bc+ca}}=\dfrac{a}{\sqrt{\left(a+b\right)\left(a+c\right)}}\le\dfrac{1}{2}\left(\dfrac{a}{a+b}+\dfrac{a}{a+c}\right)\)

Tương tự: \(\dfrac{b}{\sqrt{b^2+1}}\le\dfrac{1}{2}\left(\dfrac{b}{a+b}+\dfrac{b}{b+c}\right)\) ; \(\dfrac{c}{\sqrt{c^2+1}}\le\dfrac{1}{2}\left(\dfrac{c}{c+a}+\dfrac{c}{b+c}\right)\)

Cộng vế:

\(VT\le\dfrac{1}{2}\left(\dfrac{a}{a+b}+\dfrac{b}{a+b}+\dfrac{a}{a+c}+\dfrac{c}{a+c}+\dfrac{b}{b+c}+\dfrac{c}{b+c}\right)=\dfrac{3}{2}\)

Dấu "=" xảy ra khi \(a=b=c=\dfrac{1}{\sqrt{3}}\)

Áp dụng BDT Bu-nhi-a-cốp-xki ta có: 

\(\left(a\sqrt{1+b}+b\sqrt{1+a}\right)^2\le\left(a^2+b^2\right)\left(2+a+b\right)=a+b+2\)

Tiếp tục áp dụng BĐT Bu-nhi-a-cốp-xki ta có: \(\left(1.a+1.b\right)^2\le\left(1^2+1^2\right)\left(a^2+b^2\right)=2\Rightarrow a+b\le\sqrt{2}\)

\(\Rightarrow\left(a\sqrt{1+b}+b\sqrt{1+a}\right)^2\le a+b+2\le2+\sqrt{2}\Rightarrow a\sqrt{1+b}+b\sqrt{1+a}\le\sqrt{2+\sqrt{2}}\)