Áp dụng BĐT AM - GM, ta có:
\(2\ge a^2+b^2\ge2ab\)
\(\Leftrightarrow ab\le1\)
\(A=a\sqrt{3b\left(a+2b\right)}+b\sqrt{3a\left(b+2a\right)}\)
\(\le\dfrac{a\left(3b+a+2b\right)}{2}+\dfrac{b\left(3a+b+2a\right)}{2}\)
\(=\dfrac{a\left(5b+a\right)+b\left(5a+b\right)}{2}\)
\(=\dfrac{a^2+10ab+b^2}{2}\)
\(\le\dfrac{2+10}{2}=6\)
Dấu "=" xảy ra khi a = b = 1