Nếu \(xy\le0\Rightarrow M\le0;\) nếu \(xy>0\Rightarrow M>0\Rightarrow\) GTLN nếu có của M sẽ xảy ra khi \(xy>0\)
Xét \(xy>0\Rightarrow xy+1>0\Rightarrow x>0\Rightarrow y>0\)
\(x\ge xy+1\Leftrightarrow1\ge y+\frac{1}{x}\ge2\sqrt{\frac{y}{x}}\Rightarrow\frac{y}{x}\le\frac{1}{4}\) \(\Rightarrow\frac{x}{y}\ge4\)
\(M=\frac{3xy}{x^2+y^2}=\frac{3}{\frac{x}{y}+\frac{y}{x}}=\frac{3}{\frac{15}{16}.\frac{x}{y}+\frac{x}{16y}+\frac{y}{x}}\le\frac{3}{\frac{15}{16}.4+2\sqrt{\frac{xy}{16yx}}}=\frac{12}{17}\)
\(\Rightarrow M_{max}=\frac{12}{17}\) khi \(\left\{{}\begin{matrix}x=2\\y=\frac{1}{2}\end{matrix}\right.\)