Δ=(-3)^2-4m^2=9-4m^2
Để phương trình có hai nghiệm thì 9-4m^2>=0
=>-2/3<=m<=2/3
x1^2-3x2+x1x2-m^2-2m-1>6-m^2
=>x1^2-x2(x1+x2)+x1x2>6-m^2+m^2+2m+1=2m+7
=>x1^2-x2^2>2m+7
=>(x1+x2)(x1-x2)>2m+7
=>(x1-x2)*3>2m+7
=>x1-x2>2/3m+7/3
\(\left(x_1-x_2\right)^2=\left(x_1+x_2\right)^2-4x_1x_2=3^2-4m^2=9-4m^2\)
=>\(x1-x2=\left|9-4m^2\right|\)
=>|9-4m^2|>2/3m+7/3
=>|4m^2-9|>2/3m+7/3
=>4m^2-9<-2/3m-7/3 hoặc 4m^2-9>2/3m+7/3
=>4m^2+2/3m-20/3<0 hoặc 4m^2-2/3m-34/3>0
=>\(\dfrac{-1-\sqrt{241}}{12}< m< \dfrac{-1+\sqrt{241}}{12}\) hoặc \(\left[{}\begin{matrix}m< \dfrac{1-\sqrt{409}}{12}\\m>\dfrac{1+\sqrt{409}}{12}\end{matrix}\right.\)
=>-2/3<=m<=2/3