\(a^3-3ab^2=-2\)
\(\Rightarrow\left(a^3-3ab^2\right)^2=4\)
\(\Rightarrow a^6-6a^4b^2+9a^2b^4=4\left(1\right)\)
\(b^3-3a^2b=11\)
\(\Rightarrow\left(b^3-3a^2b\right)^2=121\)
\(\Rightarrow b^6-6a^2b^4+9a^4b^2=121\left(2\right)\)
\(\left(1\right)+\left(2\right)\Rightarrow a^6+3a^4b^2+3a^2b^4+b^6=125\)
\(\Rightarrow\left(a^2+b^2\right)^3=125\Rightarrow a^2+b^2=5\)