Đặt \(x+y=a\ge0\) ta được:
\(\sqrt{a+1}+1=a^2+\sqrt{2a}\)
\(\Leftrightarrow a^2-1+\sqrt{2a}-\sqrt{a+1}=0\)
\(\Leftrightarrow\left(a-1\right)\left(a+1\right)+\frac{a-1}{\sqrt{2a}+\sqrt{a+1}}=0\)
\(\Leftrightarrow\left(a-1\right)\left(a+1+\frac{1}{\sqrt{2a}+\sqrt{a+1}}\right)=0\)
\(\Leftrightarrow a=1\Rightarrow x+y=1\Leftrightarrow y=1-x\)
Thay vào pt dưới:
\(x^2-x\left(1-x\right)=3\Leftrightarrow2x^2-x-3=0\Rightarrow\left[{}\begin{matrix}x=-1\Rightarrow y=2\\x=\frac{3}{2}\Rightarrow y=-\frac{1}{2}\end{matrix}\right.\)