ĐKXĐ: \(x;y\ne0\)

\(x^2y+2y+x=4xy\Leftrightarrow x+\frac{2}{x}+\frac{1}{y}=4\)

Đặt \(\left\{{}\begin{matrix}\frac{1}{x}=a\\\frac{1}{y}=b\end{matrix}\right.\) \(\Rightarrow\frac{x}{y}=\frac{b}{a}\) ( hệ trở thành:

\(\left\{{}\begin{matrix}\frac{1}{a}+2a+b=4\\a^2+ab+\frac{b}{a}=3\end{matrix}\right.\) \(\Leftrightarrow\left\{{}\begin{matrix}a+\frac{1}{a}+a+b=4\\a^2+1+b\left(a+\frac{1}{a}\right)=4\end{matrix}\right.\)

\(\Leftrightarrow\left\{{}\begin{matrix}a+\frac{1}{a}+a+b=4\\a\left(a+\frac{1}{a}\right)+b\left(a+\frac{1}{a}\right)=4\end{matrix}\right.\) \(\Leftrightarrow\left\{{}\begin{matrix}a+\frac{1}{a}+a+b=4\\\left(a+\frac{1}{a}\right)\left(a+b\right)=4\end{matrix}\right.\)

\(\Rightarrow\) Theo Viet đảo, \(a+\frac{1}{a}\) và \(a+b\) là nghiệm của:

\(t^2-4t+4=0\Rightarrow t=2\Rightarrow\left\{{}\begin{matrix}a+\frac{1}{a}=2\\a+b=2\end{matrix}\right.\) \(\Leftrightarrow\left\{{}\begin{matrix}a=1\\b=1\end{matrix}\right.\) \(\Rightarrow\left\{{}\begin{matrix}x=1\\y=1\end{matrix}\right.\)

Ta có:

\(\left(x-\frac{1}{y}\right)\left(y+\frac{1}{x}\right)=2\)

\(\Leftrightarrow x^2y^2-2xy-1=0\)

Giải ra tìm được xy thế vô pt sau giải tiếp

\(_{\hept{\begin{cases}x^2y+2y+x=4xy\\\frac{1}{x^2}+\frac{1}{xy}+\frac{x}{y}=3\left(2\right)\end{cases}}}\left(1\right)\)

Đk: x; y khác 0 

(1) <=> \(x+\frac{2}{x}+\frac{1}{y}=4\Leftrightarrow\left(x+\frac{1}{x}\right)+\left(\frac{1}{x}+\frac{1}{y}\right)=4\) (3)

(2) <=> \(\left(\frac{1}{x^2}+1\right)+\left(\frac{1}{xy}+\frac{x}{y}\right)=4\)

\(\Leftrightarrow\frac{\left(1+x^2\right)}{x^2}+\frac{\left(1+x^2\right)}{xy}=4\)

\(\Leftrightarrow\left(x+\frac{1}{x}\right)\left(\frac{1}{x}+\frac{1}{y}\right)=4\) (4) 

Từ (3) ; (4)  ta có: 

\(\hept{\begin{cases}x+\frac{1}{x}=2\\\frac{1}{x}+\frac{1}{y}=2\end{cases}}\Leftrightarrow x=y=1\)

\(\hept{\begin{cases}y^6+y^3+2x^2=\sqrt{xy-x^2y^2}\left(1\right)\\4xy^3+y^2+\frac{1}{2}\ge2x^2+\sqrt{1+\left(2x-y\right)^2}\left(2\right)\end{cases}}\)

\(VP\left(1\right)=\sqrt{\frac{1}{4}-\left(xy-\frac{1}{2}\right)^2}\le\frac{1}{2}\Rightarrow VT\left(1\right)=y^6+y^3+2x^2\le\frac{1}{2}\)

\(\Leftrightarrow2x^2+2y^3+4x^2\le1\left(3\right)\)

Từ (2)(3) => \(8xy^3+2y^3+2\ge2y^6+4x^2+4x^2+2\sqrt{1+\left(2x-y\right)^2}\)

\(\Leftrightarrow8xy^3+2\ge2y^6+8x^2+2\sqrt{2+\left(2x-y\right)^2}\)

\(\Leftrightarrow4xy^3+1\ge y^6+4x^2+\sqrt{1+\left(2x-y\right)^2}\)

\(\Leftrightarrow1-\sqrt{1+\left(2x-y\right)^2}\ge y^6-4xy^3+4x^2=\left(y^3-2x\right)^2\left(4\right)\)

\(VT\left(4\right)\le0;VP\left(4\right)\ge0\). Do đó:

(4) \(\Leftrightarrow\hept{\begin{cases}y=2x\\y^3=2x\end{cases}\Leftrightarrow\hept{\begin{cases}y=2x\\y^3=y\end{cases}}}\)<=> \(\hept{\begin{cases}x=0\\y=0\end{cases}}\)hoặc \(\hept{\begin{cases}x=\frac{1}{2}\\y=1\end{cases}}\)hoặc \(\hept{\begin{cases}x=\frac{-1}{2}\\y=-1\end{cases}}\)

Thử lại chỉ có \(\left(x;y\right)=\left(\frac{-1}{2};-1\right)\)thỏa mãn

Vậy hệ đã cho có nghiệm duy nhất \(\left(x;y\right)=\left(\frac{-1}{2};-1\right)\)