Ta có: 

\(b\ge0\Rightarrow b^3+1\ge1\Rightarrow a\sqrt{b^3+1}\ge a\)

Hoàn toàn tương tự: \(b\sqrt{c^3+1}\ge b\) ;\(c\sqrt{a^3+1}\ge c\)

Cộng vế:

\(P\ge a+b+c=3\) (đpcm)

Dấu "=" xảy ra khi \(\left(a;b;c\right)=\left(0;0;3\right)\) và hoán vị

Lại có:

\(a\sqrt{b^3+1}=a\sqrt{\left(b+1\right)\left(b^2-b+1\right)}\le\dfrac{a\left(b^2+2\right)}{2}\)

Tương tự: \(b\sqrt{c^3+1}\le\dfrac{b\left(c^2+2\right)}{2}\) ; \(c\sqrt{a^3+1}\le\dfrac{c\left(a^2+2\right)}{2}\)

\(\Rightarrow P\le\dfrac{1}{2}\left(ab^2+bc^2+ca^2\right)+a+b+c=\dfrac{1}{2}\left(ab^2+bc^2+ca^2\right)+3\)

\(\Rightarrow P\le\dfrac{1}{2}\left(ab^2+bc^2+ca^2+2abc\right)+3\)

Nên ta chỉ cần chứng minh: \(Q=ab^2+bc^2+ca^2+2abc\le4\)

Không mất tính tổng quát, giả sử \(a=mid\left\{a;b;c\right\}\)

\(\Rightarrow\left(a-b\right)\left(a-c\right)\le0\Leftrightarrow a^2+bc\le ab+ac\)

\(\Rightarrow ca^2+bc^2\le abc+ac^2\)

\(\Rightarrow Q\le ab^2+ac^2+2abc=a\left(b+c\right)^2=\dfrac{1}{2}.2a\left(b+c\right)\left(b+c\right)\le\dfrac{1}{54}\left(2a+2b+2c\right)^3=4\) (đpcm)

Dấu "=" xảy ra khi \(\left(a;b;c\right)=\left(1;2;0\right)\) và 1 số hoán vị của chúng

\(\left(2-a\right)\left(2-b\right)\left(2-c\right)\ge0\)

\(\Leftrightarrow8-4\left(a+b+c\right)+2\left(ab+bc+ca\right)-abc\ge0\)

\(\Leftrightarrow2\left(ab+bc+ca\right)\ge4\left(a+b+c\right)-8+abc\)

\(\Leftrightarrow2\left(ab+bc+ca\right)\ge12-8+abc\ge4\)

\(\Rightarrow2\left(ab+bc+ca\right)\ge4\)

\(\Rightarrow-2\left(ab+bc+ca\right)\le-4\)

\(\left(a+b+c\right)^2=a^2+b^2+c^2+2\left(ab+bc+ca\right)=9\)

\(\Rightarrow a^2+b^2+c^2=9-2\left(ab+bc+ca\right)\le9-4=5\)(Đpcm)

Dấu = khi \(\hept{\begin{cases}\left(2-a\right)\left(2-b\right)\left(2-c\right)=0\\abc=0\\a+b+c=3\end{cases}}\)

\(\Rightarrow\left(a;b;c\right)=\left(2;1;0\right)\)và hoán vị.

a = 2 ( t/m )

b = 1 ( t/m )

c = 0 ( t/m )

vậy \(a^2+b^2+c^2\le5\)