Lời giải:
Đặt biểu thức đã cho là \(A\)
Ta có:
\(6a^2+8ab+11b^2=2a^2+(2a+2b)^2+7b^2\)
Áp dụng BĐT Bunhiacopxky:
\([2a^2+(2a+2b)^2+7b^2](2+4^2+7)\geq (2a+8a+8b+7b)^2\)
\(\Leftrightarrow 25(6a^2+8ab+11b^2)\geq (10a+15b)^2\)
\(\Rightarrow \sqrt{6a^2+8ab+11b^2}\geq 2a+3b\)
\(\Rightarrow \frac{a^2+3ab+b^2}{\sqrt{6a^2+8ab+11b^2}}\leq \frac{a^2+3ab+b^2}{2a+3b}\)
Thực hiện tương tự với các biểu thức còn lại và cộng theo vế:
\(A\leq \frac{a^2+3ab+b^2}{2a+3b}+\frac{a^2+3ac+c^2}{2c+3a}+\frac{b^2+3bc+c^2}{2b+3c}\)
\(6A\leq \frac{3a(2a+3b)+2b(2a+3b)+5ab}{2a+3b}+\frac{3c(2c+3a)+2a(2c+3a)+5ac}{2c+3a}+\frac{3b(2b+3c)+2c(2b+3c)+5bc}{2b+3c}\)
\(\Leftrightarrow 6A\leq 3a+2b+\frac{5ab}{2a+3b}+3c+2a+\frac{5ac}{2c+3a}+3b+2c+\frac{5bc}{2b+3c}\)
\(\Leftrightarrow 6A\leq 5(a+b+c)+5\left(\frac{ab}{2a+3b}+\frac{bc}{2b+3c}+\frac{ac}{2c+3a}\right)\)
Theo hệ quả của BĐT AM-GM:
\((a+b+c)^2\leq 3(a^2+b^2+c^2)=9\Rightarrow a+b+c\leq 3(1)\)
Áp dụng BĐT Cauchy-Schwarz dạng ngược:
\(\frac{ab}{2a+3b}\leq \frac{ab}{25}\left(\frac{1}{a}+\frac{1}{a}+\frac{1}{b}+\frac{1}{b}+\frac{1}{b}\right)\)
\(\frac{bc}{2b+3c}\leq \frac{bc}{25}\left(\frac{1}{b}+\frac{1}{b}+\frac{1}{c}+\frac{1}{c}+\frac{1}{c}\right)\)
\(\frac{ca}{2c+3a}\leq \frac{ca}{25}\left(\frac{1}{c}+\frac{1}{c}+\frac{1}{a}+\frac{1}{a}+\frac{1}{a}\right)\)
\(\Rightarrow \frac{ab}{2a+3b}+\frac{bc}{2b+3c}+\frac{ac}{2c+3a}\leq \frac{1}{5}(a+b+c)(2)\)
Từ (1); (2) suy ra:
\(6A\leq 5(a+b+c)+5.\frac{1}{5}(a+b+c)=6(a+b+c)\leq 18\)
\(\Rightarrow A\leq 3\) (đpcm)
Dấu bằng xảy ra khi \(a=b=c=1\)
