Vì a,b>0 nên:\(ab>0;\left(a^2-b^2\right)^2\ge0\)
\(\Leftrightarrow ab\left(a^2-b^2\right)^2\ge0\)
\(\Leftrightarrow ab\left(a^4-2a^2b^2+b^4\right)\ge0\)
\(\Leftrightarrow a^5b-2a^3b^3+ab^5\ge0\)
\(\Leftrightarrow a^6+ab^5+a^5b+b^6-a^6-2a^3b^3-b^6\ge0\)
\(\Leftrightarrow a\left(a^5+b^5\right)+b\left(a^5+b^5\right)-\left(a^3+b^3\right)^2\ge0\)
\(\Leftrightarrow\left(a+b\right)\left(a^5+b^5\right)\ge\left(a^3+b^3\right)^2\)
\(\Leftrightarrow a+b\ge a^3+b^3\)(Vì a^5+b^5=a^3+b^3 và a^3+b^3;a^5+b^5>0)
\(\Leftrightarrow a+b\ge\left(a+b\right)\left(a^2-ab+b^2\right)\)
\(\Leftrightarrow a^2-ab+b^2\ge1\)
Vậy GTLN M=1 tại \(a^2-b^2=0\Leftrightarrow a=b\)
\(\Leftrightarrow a^3+a^3=a^5+a^5\)(Vì a=b)
\(\Leftrightarrow\orbr{\begin{cases}a=0\\a=1\end{cases}}\)(TH a=0 loại vì a>0)
\(\Leftrightarrow b=1\)
