Ta có: \(\frac{5a^3-b^3}{ab+3a^2}=\frac{3a^3-b^3}{ab+3a^2}+\frac{2a^3}{ab+3a^2}\)
\(=a-\frac{a^2b+b^3}{ab+3a^2}+\frac{2a^3}{ab+3a^2}\)
= \(a-\frac{b\left(a^2+b^2\right)}{a\left(b+3a\right)}+\frac{2a^3}{a\left(b+3a\right)}\) (1)
Áp dụng BĐT AM - GM ( x2 + y2 \(\ge2xy\)) ta có:
(1) \(\le a-\frac{2ab^2}{a\left(b+3a\right)}+\frac{2a^2}{b+3a}\) = \(a-\frac{2b^2}{b+3a}+\frac{2a^2}{b+3a}\) (2)
Tương tự ta cũng có:
\(\frac{5b^3-c^3}{bc+3b^2}\le b-\frac{2c^2}{c+3b}+\frac{2b^2}{c+3b}\left(3\right)\)
\(\frac{5c^3-a^2}{ca+3c^2}\)\(\le c-\frac{2a^2}{a+3c}+\frac{2c^2}{a+3c}\)(4)
Từ (2), (3), (4) \(\Rightarrow\frac{5a^3-b^3}{ab+3a^2}+\frac{5b^3-c^3}{bc+3b^2}+\frac{5c^3-a^3}{ca+3c^2}\le a+b+c+\left(\frac{2a^2}{a+3c}-\frac{2a^2}{a+3c}\right)+\left(\frac{2b^2}{b+3c}-\frac{2b^2}{b+3c}\right)+\left(\frac{2c^2}{c+3a}-\frac{2c^2}{c+3a}\right)=a+b+c\le2018\)
Vậy \(\frac{5a^3-b^3}{ab+3a^2}+\frac{5b^3-c^3}{bc+3b^2}+\frac{5c^3-a^3}{ca+3c^2}\le2018\)