\(\Delta=\left(-3\right)^2-4\cdot1\cdot\left(m-3\right)\)
=9-4m+12
=-4m+21
Để phương trình có hai nghiệm thì -4m+21>=0
=>-4m>=-21
=>\(m\le\frac{21}{4}\)
Theo Vi-et, ta có: \(\begin{cases}x_1+x_2=-\frac{b}{a}=3\\ x_1x_2=\frac{c}{a}=m-3\end{cases}\)
\(x_1^2+3x_2=x_1^2\cdot x_2^2-11\)
=>\(x_1^2+x_2\left(x_1+x_2\right)=\left(x_1x_2\right)^2-11\)
=>\(\left(x_1^2+x_2^2\right)+x_1x_2=\left(x_1x_2\right)^2-11\)
=>\(\left(x_1+x_2\right)^2-x_1x_2-\left(x_1x_2\right)^2=-11\)
=>\(3^2-\left(m-3\right)-\left(m-3\right)^2=-11\)
=>\(9-m+3-m^2+6m-9=-11\)
=>\(-m^2+5m+3+11=0\)
=>\(-m^2+5m+14=0\)
=>\(m^2-5m-14=0\)
=>(m-7)(m+2)=0
=>\(\left[\begin{array}{l}m=7\left(loại\right)\\ m=-2\left(nhận\right)\end{array}\right.\)
