\(\left(a^2+b^2\right)^2=\left(2a+b\right)^2\le\left(2^2+1\right)\left(a^2+b^2\right)\)
\(\Rightarrow a^2+b^2\le5\) \(\Rightarrow2a+b\le5\Rightarrow a\le\frac{5-b}{2}\)
\(P=a\left(1+b\right)-b\le\frac{\left(1+b\right)\left(5-b\right)}{2}-b\)
\(P\le\frac{-b^2+2b+5}{2}=\frac{-\left(b-1\right)^2+6}{2}\le3\)
\(P_{max}=3\) khi \(\left\{{}\begin{matrix}b=1\\a=2\end{matrix}\right.\)