\(\left\{{}\begin{matrix}x+3\sqrt{xy+x-y^{2-y}}=5y+4\left(1\right)\\\sqrt{4y^2-x-2}+\sqrt{y-1}=x-1\left(2\right)\end{matrix}\right.\)
ĐK: x\(\ge1,y\ge1\),x\(\ge y\)
(1)\(\Leftrightarrow\left(x-y\right)+3\sqrt{x\left(y+1\right)-y\left(y+1\right)}-4y-4=0\Leftrightarrow\left(x-y\right)+3\sqrt{\left(x-y\right)\left(y+1\right)}-4\left(y+1\right)=0\left(3\right)\)
Chia 2 vế của (3) cho y+1>0 thì (3) và đặt t=\(\sqrt{\dfrac{x-y}{y+1}}\)(t\(\ge0\))
Vậy (3)\(\Leftrightarrow t^2+3t-4=0\Leftrightarrow t^2-t+4t-4=0\Leftrightarrow t\left(t-1\right)+4\left(t-4\right)=0\Leftrightarrow\left(t-1\right)\left(t+4\right)=0\Leftrightarrow\)\(\left[{}\begin{matrix}t-1=0\\t+4=0\end{matrix}\right.\)\(\Leftrightarrow\)\(\left[{}\begin{matrix}t=1\left(tm\right)\\t=-4\left(ktm\right)\end{matrix}\right.\)
Ta có t=1\(\Leftrightarrow\sqrt{\dfrac{x-y}{y+1}}=1\Leftrightarrow x-y=y+1\Leftrightarrow x=2y+1\)
Thay vào phương trình (2)\(\Leftrightarrow\sqrt{4y^2-\left(2y+1\right)-2}+\sqrt{y-1}=2y+1-1\Leftrightarrow\sqrt{4y^2-2y-3}+\sqrt{y-1}=2y\Leftrightarrow\left(\sqrt{4y^2-2y-3}-3\right)+\left(\sqrt{y-1}-1\right)=2\left(y-2\right)\Leftrightarrow\dfrac{4y^2-2y-12}{\sqrt{4y^2-2y-3}+3}+\dfrac{y-2}{\sqrt{y-1}+1}-2\left(y-2\right)=0\Leftrightarrow\dfrac{2\left(y-2\right)\left(2y+3\right)}{\sqrt{4y^2-2y-3}+3}+\dfrac{y-2}{\sqrt{y-1}+1}-2\left(y-2\right)=0\Leftrightarrow\left(y-2\right)\left[\dfrac{2\left(2y+3\right)}{\sqrt{4y^2-2y-3}+3}+\dfrac{1}{\sqrt{y-1}+1}-2\right]=0\Leftrightarrow\)\(\left[{}\begin{matrix}y-2=0\left(4\right)\\\dfrac{2\left(2y+3\right)}{\sqrt{4y^2-2y-3}+3}+\dfrac{1}{\sqrt{y-1}+1}-2=0\left(5\right)\end{matrix}\right.\)
(4)\(\Leftrightarrow y=2\Leftrightarrow x=5\left(tm\right)\)
(5)\(\Leftrightarrow\dfrac{2\left(2y+3\right)}{\sqrt{4y^2-2y-3}+3}=2y+3-\sqrt{y+1}< 2y+3\Rightarrow\dfrac{2\left(2y+3\right)}{\sqrt{4y^2-2y-3}+3}\ge2\Leftrightarrow\)VT của (5)>2\(\Rightarrow\) vô nghiệm
Vậy (x;y)=(5;2)