Để pt có 2 nghiệm trái dấu \(\Leftrightarrow ac< 0\Rightarrow m^2-3< 0\Rightarrow-\sqrt{3}< m< \sqrt{3}\)
\(\Delta=m^2-4\left(m^2-3\right)=12-3m^2\ge0\Rightarrow m^2\le4\)
Khi đó theo Viet ta có: \(\left\{{}\begin{matrix}x_1+x_2=m\\x_1x_2=m^2-3\end{matrix}\right.\)
\(\Rightarrow A=\left|x_1^2+x_2^2-x_1x_2\right|=\left|\left(x_1+x_2\right)^2-3x_1x_2\right|\)
\(A=\left|m^2-3\left(m^2-3\right)\right|=\left|9-2m^2\right|=9-2m^2\le9\)
\(\Rightarrow A_{max}=9\) khi \(m=0\)