\(3x^2-4x-11=\left(2x-5\right)\sqrt{3x+7}\\ \Leftrightarrow9x^4-50x^2+121-24x^3-66x^2+88x=\left(4x^2-20x+25\right)\left(3x+7\right)\\ \Leftrightarrow12x^3-32x^2-65x+175-9x^4+50x^2-121+24x^3-88x=0\\ \Leftrightarrow-9x^4+36x^3+18x^2-153x+54=0\\ \Leftrightarrow\left(x+2\right)\left(x-3\right)\left(x^2-3x+1\right)=0\\ \Leftrightarrow\left[{}\begin{matrix}x+2=0\\x-3=0\\x^2-3x+1=0\end{matrix}\right.\Leftrightarrow\left[{}\begin{matrix}x=-2\left(KTM\right)\\x=3\left(TM\right)\\x=\dfrac{3+\sqrt{5}}{2}\left(KTM\right)\\x=\dfrac{3-\sqrt{5}}{2}\left(TM\right)\end{matrix}\right.\)
Vậy \(S=\left\{3;\dfrac{3-\sqrt{5}}{2}\right\}\)