ĐK: \(x\ne k\pi\)
Đặt \(\left\{{}\begin{matrix}cotx=a\\sinx=b\end{matrix}\right.\left(a\in R;b\in\left[-1;1\right]\right)\), khi đó:
\(3cot^2x+2\sqrt{2}sin^2x=\left(2+3\sqrt{2}\right)cosx\)
\(\Leftrightarrow3a^2+2\sqrt{2}b^2=\left(2+3\sqrt{2}\right)ab\)
\(\Leftrightarrow3a^2-2ab+2\sqrt{2}b^2-3\sqrt{2}ab=0\)
\(\Leftrightarrow\left(3a-2b\right)\left(a-\sqrt{2}b\right)=0\)
\(\Leftrightarrow\left[{}\begin{matrix}3a=2b\\a=\sqrt{2}b\end{matrix}\right.\)
\(\Leftrightarrow\left[{}\begin{matrix}3cotx=2sinx\\cotx=\sqrt{2}sinx\end{matrix}\right.\)
\(\Leftrightarrow\left[{}\begin{matrix}3cosx=2sin^2x\\cosx=\sqrt{2}sin^2x\end{matrix}\right.\)
\(\Leftrightarrow\left[{}\begin{matrix}3cosx=2-2cos^2x\\cosx=\sqrt{2}-\sqrt{2}cos^2x\end{matrix}\right.\)
\(\Leftrightarrow\left[{}\begin{matrix}2cos^2x+3cosx-2=0\\\sqrt{2}cos^2x+cosx-\sqrt{2}=0\end{matrix}\right.\)
TH1: \(2cos^2x+3cosx-2=0\Leftrightarrow cosx=\dfrac{1}{2}\Leftrightarrow x=\pm\dfrac{\pi}{3}+k2\pi\)
TH2: \(\sqrt{2}cos^2x+cosx-\sqrt{2}=0\Leftrightarrow cosx=\dfrac{\sqrt{2}}{2}\Leftrightarrow x=\pm\dfrac{\pi}{4}+k2\pi\)