Gọi \(M\left(2a-7;-a\right)\) \(\Rightarrow\left\{{}\begin{matrix}\overrightarrow{AM}=\left(2a-8;-a-3\right)\\\overrightarrow{BM}=\left(2a-11;-a-8\right)\\\overrightarrow{CM}=\left(2a-10;-a-4\right)\end{matrix}\right.\)
\(\Rightarrow P=MA^2+3MB^2-5MC^2\)
\(=\left(2a-8\right)^2+\left(a+3\right)^2+3\left(2a-11\right)^2+3\left(a+8\right)^2-5\left(2a-10\right)^2-5\left(a+4\right)^2\)
\(=-5a^2+50a+48=-5\left(a^2-10a+25\right)+173\)
\(=-6\left(a-5\right)^2+173\le173\)
\(\Rightarrow P_{max}=173\) khi \(a=5\Rightarrow M\left(3;-5\right)\)