\(\Delta'=\left(m-1\right)^2-2\left(m^2-1\right)=-m^2-2m+3>0\)
\(\Rightarrow-3< m< 1\)
Khi đó theo hệ thức Viet: \(\left\{{}\begin{matrix}x_1+x_2=-\left(m-1\right)\\x_1x_2=\dfrac{m^2-1}{2}\end{matrix}\right.\)
\(P=\left(x_1-x_2\right)^2=x_1^2+x_2^2-2x_1x_2\)
\(P=x_1^2+x_2^2+2x_1x_2-4x_1x_2=\left(x_1+x_2\right)^2-4x_1x_2\)
\(P=\left(m-1\right)^2-4\left(\dfrac{m^2-1}{2}\right)\)
\(P=-m^2-2m+3=-\left(m^2+2m+1\right)+4\)
\(P=-\left(m+1\right)^2+4\le4\)
\(P_{max}=4\) khi \(m+1=0\Leftrightarrow m=-1\) (thỏa mãn)