Ta có: \(a^3-3ab^2=2\)
\(\Rightarrow\left(a^3-3ab^2\right)^2=4\)
\(\Leftrightarrow a^6-6a^4b^2+9a^2b^4=4\left(1\right)\)
Lại có: \(b^3-3a^2b=-11\)
\(\Rightarrow\left(b^3-3a^2b\right)=121\)
\(\Leftrightarrow b^6-6a^2b^4+9a^4b^2=121\left(2\right)\)
Lấy \(\left(1\right)+\left(2\right)\)ta được:
\(a^6-6a^4b^2+9a^2b^4+b^6-6a^2b^4+9a^4b^2=125\)
\(\Leftrightarrow a^6+3a^4b^2+b^6+3a^2b^4=125\)
\(\Leftrightarrow\left(a^2+b^2\right)^3=125\)
\(\Leftrightarrow a^2+b^2=5\)
Vậy ...