\(ab\le\dfrac{\left(a+b\right)^2}{4}\Rightarrow\dfrac{\left(a+b\right)^2}{4}+\left(a+b\right)\ge3\)
\(\Rightarrow\left(a+b\right)^2+4\left(a+b\right)-12\ge0\)
\(\Rightarrow\left[{}\begin{matrix}a+b\ge2\\a+b\le-6\end{matrix}\right.\) \(\Rightarrow\left(a+b\right)^2\ge4\)
\(M=a^2+b^2\ge\dfrac{\left(a+b\right)^2}{2}\ge\dfrac{4}{2}=2\)
\(\Rightarrow M_{min}=2\) khi \(a=b=1\)