\(x^2-2x-1=\sqrt{\left(x^2+1\right)\left(x+1\right)}\) (*)
ĐK: \(\left\{{}\begin{matrix}\left(x^2+1\right)\left(x+1\right)\ge0\\x^2-2x-1\ge0\end{matrix}\right.\Leftrightarrow\left\{{}\begin{matrix}x\ge-1\\\left[{}\begin{matrix}x\le1-\sqrt{2}\\x\ge1+\sqrt{2}\end{matrix}\right.\end{matrix}\right.\Leftrightarrow\left[{}\begin{matrix}-1\le x\le1-\sqrt{2}\\x\ge1+\sqrt{2}\end{matrix}\right.\)
(*) \(\Leftrightarrow\left(x^2+1\right)-2\left(x+1\right)=\sqrt{\left(x^2+1\right)\left(x+1\right)}\)
Đặt: \(\left\{{}\begin{matrix}\sqrt{x^2+1}=u\\\sqrt{x+1}=v\end{matrix}\right.\)
Pt trở thành: \(u^2-2v^2=uv\)
\(\Leftrightarrow\left(u^2-v^2\right)-\left(v^2+uv\right)=0\)
\(\Leftrightarrow\left(u+v\right)\left(u-v\right)-v\left(u+v\right)=0\)
\(\Leftrightarrow\left(u+v\right)\left(u-2v\right)=0\)
\(\Leftrightarrow\left[{}\begin{matrix}u=-v\\u=2v\end{matrix}\right.\)
Xét TH1:
\(u=-v\Rightarrow\sqrt{x^2+1}=-\sqrt{x+1}\) (vô lý vì \(\sqrt{x^2+1}\ge1\) mà \(-\sqrt{x+1}\le0\))
Xét TH2:
\(u=2v\Rightarrow\sqrt{x^2+1}=2\sqrt{x+1}\)
\(\Leftrightarrow x^2+1=4\left(x+1\right)\)
\(\Leftrightarrow x^2-4x-3=0\)
\(\Leftrightarrow\left[{}\begin{matrix}x=2+\sqrt{7}\left(tm\right)\\x=2-\sqrt{7}\left(tm\right)\end{matrix}\right.\) (dùng delta)