DK:\(y\ne0\)
PT (1) :\(3x^2+2y^2-4xy=11-\dfrac{1}{y}\left(2x+\dfrac{1}{y}\right)\)
\(\Leftrightarrow\left(x^2+\dfrac{2x}{y}+\dfrac{1}{y^2}\right)+2\left(x^2-2xy+y^2\right)=11\)
\(\Leftrightarrow\left(x+\dfrac{1}{y}\right)^2+2\left(x-y\right)^2=11\)
PT (2): \(2x+\dfrac{1}{y}-y=4\)
\(\Leftrightarrow\left(x+\dfrac{1}{y}\right)+\left(x-y\right)=4\)
Đặt \(a=x+\dfrac{1}{y};b=x-y\)
Hệ pt tt: \(\left\{{}\begin{matrix}a^2+2b^2=11\\a+b=4\end{matrix}\right.\) \(\Leftrightarrow\left\{{}\begin{matrix}\left(4-b\right)^2+2b^2=11\\a=4-b\end{matrix}\right.\)\(\Leftrightarrow\left\{{}\begin{matrix}\left[{}\begin{matrix}b=\dfrac{5}{3}\\b=1\end{matrix}\right.\\a=4-b\end{matrix}\right.\) \(\Leftrightarrow\left[{}\begin{matrix}\left\{{}\begin{matrix}b=\dfrac{5}{3}\\a=\dfrac{7}{3}\end{matrix}\right.\\\left\{{}\begin{matrix}b=1\\a=3\end{matrix}\right.\end{matrix}\right.\)
TH1: \(a=\dfrac{7}{3};b=\dfrac{5}{3}\)\(\Rightarrow\left\{{}\begin{matrix}x+\dfrac{1}{y}=\dfrac{7}{3}\\x-y=\dfrac{5}{3}\end{matrix}\right.\)\(\Leftrightarrow\left\{{}\begin{matrix}\dfrac{1}{y}+y=\dfrac{2}{3}\\x-y=\dfrac{5}{3}\end{matrix}\right.\)\(\Leftrightarrow\left\{{}\begin{matrix}3y^2-2y+3=0\left(vn\right)\\x-y=\dfrac{5}{3}\end{matrix}\right.\)
TH2:\(a=3;b=1\)\(\Rightarrow\left\{{}\begin{matrix}x+\dfrac{1}{y}=3\\x-y=1\end{matrix}\right.\)\(\Leftrightarrow\left\{{}\begin{matrix}\dfrac{1}{y}+y=2\\x-y=1\end{matrix}\right.\)\(\Leftrightarrow\left\{{}\begin{matrix}y^2-2y+1=0\\x-y=1\end{matrix}\right.\)\(\Leftrightarrow\left\{{}\begin{matrix}y=1\\x=2\end{matrix}\right.\) (thỏa mãn hệ)
Vậy hệ có nghiệm duy nhất (x;y)=(2;1).