Lời giải:
Áp dụng BĐT Bunhiacopxky:
\((a\sqrt{3a(a+2b)}+b\sqrt{3b(b+2a)})^2\leq (a^2+b^2)[3a(a+2b)+3b(b+2a)]\)
\((a\sqrt{3a(a+2b)}+b\sqrt{3b(b+2a)})^2\leq (a^2+b^2)(3a^2+3b^2+12ab)\)
Theo BĐT Cô-si: \(a^2+b^2\geq 2ab\Rightarrow 12ab\leq 6(a^2+b^2)\)
Do đó:
\((a\sqrt{3a(a+2b)}+b\sqrt{3b(b+2a)})^2\leq (a^2+b^2)(3a^2+3b^2+6a^2+6b^2)=9(a^2+b^2)^2\)
Mà \(a^2+b^2\leq 2\)
\(\Rightarrow (a\sqrt{3a(a+2b)}+b\sqrt{3b(b+2a)})^2\leq 9.2^2=36\)
\(\Rightarrow a\sqrt{3a(a+2b)}+b\sqrt{3b(b+2a)}\leq \sqrt{36}=6\)
(đpcm)
Dấu bằng xảy ra khi $a=b=1$