\(2\left(x^2+2x+3\right)=5\sqrt{x^3+3x^2+3x+2}\)
\(\Leftrightarrow2\left(x^2+2x+3\right)=5\sqrt{\left(x+1\right)^3+1}\) (*)
\(ĐK:\left(x+1\right)^3+1\ge0\)
\(\Leftrightarrow\left(x+1\right)^3\ge-1\)
\(\Leftrightarrow x\ge-2\)
(*)\(\Leftrightarrow2\left(x^2+2x+3\right)=5\sqrt{\left(x+1+1\right)\left(x^2+2x+1-x-1+1\right)}\)
\(\Leftrightarrow2x^2+4x+6=5\sqrt{\left(x+2\right)\left(x^2+x+1\right)}\)
\(\Leftrightarrow2\left(x^2+x+1\right)+2\left(x+2\right)=5\sqrt{\left(x+2\right)\left(x^2+x+1\right)}\)
Đặt \(\left\{{}\begin{matrix}x^2+x+1=a\\x+2=b\end{matrix}\right.\) ;\(ab\ge0\)
\(\Leftrightarrow2a+2b=5\sqrt{ab}\)
\(\Leftrightarrow4a^2+8ab+4b^2=25ab\)
\(\Leftrightarrow4a^2-17ab+4b^2=0\)
\(\Leftrightarrow4a^2-16ab-ab+4b^2=0\)
\(\Leftrightarrow4a\left(a-4b\right)-b\left(a-4b\right)=0\)
\(\Leftrightarrow\left(a-4b\right)\left(4a-b\right)=0\)
\(\Leftrightarrow\left[{}\begin{matrix}a=4b\\4a=b\end{matrix}\right.\)
`@`Với \(a=4b\)
\(\Leftrightarrow x^2+x+1=4x+8\)
\(\Leftrightarrow x^2-3x-7=0\)
\(\Leftrightarrow x=\dfrac{3\pm\sqrt{37}}{2}\left(tm\right)\)
`@`Với \(4a=b\)
\(\Leftrightarrow4x^2+4x+4=x+2\)
\(\Leftrightarrow4x^2+3x+2=0\) ( vô lý )
Vậy \(S=\left\{\dfrac{3\pm\sqrt{37}}{2}\right\}\)