\(x\ge xy+1\Rightarrow1\ge y+\dfrac{1}{x}\ge2\sqrt{\dfrac{y}{x}}\Rightarrow\dfrac{y}{x}\le\dfrac{1}{4}\)
\(Q^2=\dfrac{x^2+2xy+y^2}{3x^2-xy+y^2}=\dfrac{\left(\dfrac{y}{x}\right)^2+2\left(\dfrac{y}{x}\right)+1}{\left(\dfrac{y}{x}\right)^2-\dfrac{y}{x}+3}\)
Đặt \(\dfrac{y}{x}=t\le\dfrac{1}{4}\)
\(Q^2=\dfrac{t^2+2t+1}{t^2-t+3}=\dfrac{t^2+2t+1}{t^2-t+3}-\dfrac{5}{9}+\dfrac{5}{9}\)
\(Q^2=\dfrac{\left(4t-1\right)\left(t+6\right)}{9\left(t^2-t+3\right)}+\dfrac{5}{9}\le\dfrac{5}{9}\)
\(\Rightarrow Q_{max}=\dfrac{\sqrt{5}}{3}\) khi \(t=\dfrac{1}{4}\) hay \(\left(x;y\right)=\left(2;\dfrac{1}{2}\right)\)