Ta có M=a(b+c)+3b(c+a)+5c(a+b)=a(3-a)+3b(3-b)+5c(3-c)=\(\frac{81}{4}\)-\(\left(a-\frac{3}{2}\right)^2+3\left(b-\frac{3}{2}\right)^2+5\left(c-\frac{3}{2}\right)^2\)
Đặt x=\(\left|a-\frac{3}{2}\right|\),y=\(\left|b-\frac{3}{2}\right|\),z=\(\left|c-\frac{3}{2}\right|\)=>x+y+z\(\ge\left|a+b+c-\frac{9}{2}\right|=\frac{3}{2}\)
Khi đó M=\(\frac{81}{4}-\left(x^2+3y^2+5z^2\right)\)
Đưa thêm các tham số\(\alpha,\beta,\gamma>0\)Áp dụng bất đẳng thức AM-GM:\(x^2+\alpha^2\ge2x\alpha\)(1);\(3y^2+3\beta^2\ge6y\beta\)(2);\(5z^2+5\gamma^2\ge10z\gamma\)(3)
Suy ra: \(M-\alpha^2-3\beta^2-5\gamma^2\le\frac{81}{4}-2\left(x\alpha+3y\beta+5z\gamma\right)\)
Ta chọn \(\alpha=3\beta=5\gamma\)\(\Rightarrow M\le\frac{81}{4}+\alpha^2+3\beta^2+5\gamma^2-2\alpha\left(x+y+z\right)\)\(\le\frac{81}{4}+\alpha^2+3\beta^2+5\gamma^2-3a\)
Ta thấy dấu bằng các bất đẳng thức (1),(2),(3) xảy ra khi \(x=\alpha,y=\beta,z=\gamma\)\(\Rightarrow\alpha+\beta+\gamma=\alpha+\frac{\alpha}{3}+\frac{\alpha}{5}=x+y+z=\frac{3}{2}\)\(\Rightarrow\alpha=\frac{45}{46}\),\(\beta=\frac{15}{46},\gamma=\frac{9}{46}\)
Vậy MaxM=\(\le\frac{81}{4}+\left(\frac{45}{46}\right)^2+3\left(\frac{15}{46}\right)^2+5\left(\frac{9}{46}\right)^2-3.\frac{45}{46}\)=\(\frac{432}{23}\)