\(AB=\sqrt{OA^2+OB^2}=OA\sqrt{1+k^2}\)
\(OM=BM=\dfrac{1}{2}AB=\dfrac{OA}{2}\sqrt{1+k^2}\)
\(cos\widehat{OMB}=cos60^0=\dfrac{OM^2+BM^2-OB^2}{2OM.BM}=\dfrac{1}{2}\)
\(\Leftrightarrow\dfrac{OA^2\left(\dfrac{k^2+1}{4}\right)+OA^2\left(\dfrac{k^2+1}{4}\right)-k^2OA^2}{2.OA^2\left(\dfrac{k^2+1}{4}\right)}=\dfrac{1}{2}\)
\(\Leftrightarrow\dfrac{1-k^2}{1+k^2}=\dfrac{1}{2}\Rightarrow k^2=\dfrac{1}{3}\Rightarrow k=\dfrac{1}{\sqrt{3}}\)