\(x\ge xy+1\Rightarrow1\ge y+\dfrac{1}{x}\ge2\sqrt{\dfrac{y}{x}}\Rightarrow\dfrac{y}{x}\le\dfrac{1}{4}\)

\(Q^2=\dfrac{x^2+2xy+y^2}{3x^2-xy+y^2}=\dfrac{\left(\dfrac{y}{x}\right)^2+2\left(\dfrac{y}{x}\right)+1}{\left(\dfrac{y}{x}\right)^2-\dfrac{y}{x}+3}\)

Đặt \(\dfrac{y}{x}=t\le\dfrac{1}{4}\) 

\(Q^2=\dfrac{t^2+2t+1}{t^2-t+3}=\dfrac{t^2+2t+1}{t^2-t+3}-\dfrac{5}{9}+\dfrac{5}{9}\)

\(Q^2=\dfrac{\left(4t-1\right)\left(t+6\right)}{9\left(t^2-t+3\right)}+\dfrac{5}{9}\le\dfrac{5}{9}\)

\(\Rightarrow Q_{max}=\dfrac{\sqrt{5}}{3}\) khi \(t=\dfrac{1}{4}\) hay \(\left(x;y\right)=\left(2;\dfrac{1}{2}\right)\)

\(x+y\le xy\Rightarrow\dfrac{1}{x}+\dfrac{1}{y}\le1\)

\(M=\dfrac{1}{2\left(x^2+y^2\right)+y^2}+\dfrac{1}{2\left(x^2+y^2\right)+x^2}\le\dfrac{1}{4xy+y^2}+\dfrac{1}{4xy+x^2}\)

\(B\le\dfrac{1}{25}\left(\dfrac{4}{xy}+\dfrac{1}{y^2}\right)+\dfrac{1}{25}\left(\dfrac{4}{xy}+\dfrac{1}{x^2}\right)=\dfrac{1}{25}\left(\dfrac{1}{x^2}+\dfrac{1}{y^2}+\dfrac{2}{xy}+\dfrac{6}{xy}\right)\)

\(M\le\dfrac{1}{25}\left[\left(\dfrac{1}{x}+\dfrac{1}{y}\right)^2+\dfrac{3}{2}\left(\dfrac{1}{x}+\dfrac{1}{y}\right)^2\right]=\dfrac{1}{10}\left(\dfrac{1}{x}+\dfrac{1}{y}\right)^2\le\dfrac{1}{10}\)

\(M_{max}=\dfrac{1}{10}\) khi \(x=y=2\)