\(\left(I\right)\begin{cases}3x^2+2xy+y^2=11\\x^2+2xy+3y^2=17\end{cases}\)
Ta thấy x=0 không thỏa mãn hệ (I).Đặt y=tx ta đc
\(\left(II\right)\begin{cases}x^2\left(3+2t+t^2\right)=11\left(1\right)\\x^2\left(1+2t+3t^2\right)=17\left(2\right)\end{cases}\)
Suy ra \(\frac{1+2t+3t^2}{3+2t+t^2}=\frac{17}{11}\Leftrightarrow4t^2-3t-10=0\Leftrightarrow\left[\begin{array}{nghiempt}t=2\\t=-\frac{5}{4}\end{array}\right.\)
\(t=2\Rightarrow x^2=1\Rightarrow x=\pm1\Rightarrow y=\pm2\)\(t=-\frac{5}{4}\Rightarrow x^2=\frac{16}{3}\Rightarrow x=\pm\frac{4}{\sqrt{3}}\Rightarrow y=\pm\frac{5}{\sqrt{3}}\)Vậy hệ (I) có bốn nghiệm là: \(\left(x;y\right)=\left(1;2\right),\left(-1;-2\right),\left(\frac{4}{\sqrt{3}};-\frac{5}{\sqrt{3}}\right),\left(-\frac{4}{\sqrt{3}};\frac{5}{\sqrt{3}}\right)\)