Gọi N là trung điểm AC \(\Rightarrow MN||AB\Rightarrow\widehat{OMN}\) là góc giữa OM và AB
Đặt \(OA=a\)
\(AB=\sqrt{OA^2+OB^2}=\sqrt{a^2+k^2a^2}=a\sqrt{k^2+1}\)
\(AC=\sqrt{OA^2+OC^2}=a\sqrt{k^2+1}\)
\(BC=\sqrt{OB^2+OC^2}=a.k\sqrt{2}\)
\(MN=\dfrac{1}{2}AB=\dfrac{a}{2}\sqrt{k^2+1}\) ; \(OM=\dfrac{BC}{2}=a.\dfrac{k\sqrt{2}}{2}\) ; \(ON=\dfrac{1}{2}AC=a.\dfrac{\sqrt{k^2+1}}{2}\)
\(cos\widehat{OMN}=cos60^0=\dfrac{OM^2+MN^2-ON^2}{2OM.MN}=\dfrac{1}{2}\)
\(\Leftrightarrow\dfrac{a^2.\dfrac{k^2}{2}}{2.a^2.\dfrac{k\sqrt{2k^2+2}}{4}}=\dfrac{1}{2}\Leftrightarrow2k=\sqrt{2k^2+2}\)
\(\Leftrightarrow4k^2=2k^2+2\Rightarrow k=1\)