\(a^2+ab+b^2-b=0\)
\(\Delta=b^2-4\left(b^2-b\right)\ge0\Leftrightarrow-3b^2+4b\ge0\Rightarrow0\le b\le\dfrac{4}{3}\)
\(b^2+\left(a-1\right)b+a^2=0\)
\(\Delta=\left(a-1\right)^2-4a^2\ge0\Rightarrow-3a^2-2a+1\ge0\Rightarrow-1\le a\le\dfrac{1}{3}\)
\(\Rightarrow A=3a^5+b^4\le3.\left(\dfrac{1}{3}\right)^5+\left(\dfrac{4}{3}\right)^4=\dfrac{257}{81}< 4\)