Đặt \(x=AA'\)
Ta có: \(\overrightarrow{AB'}=\overrightarrow{AA'}+\overrightarrow{AB}\) ; \(\overrightarrow{BD'}=\overrightarrow{BB'}+\overrightarrow{BD}=\overrightarrow{BB'}+\overrightarrow{BA}+\overrightarrow{BC}=\overrightarrow{AA'}-\overrightarrow{AB}+\overrightarrow{BC}\)
\(\Rightarrow\overrightarrow{AB'}.\overrightarrow{BD'}=\left(\overrightarrow{AA'}+\overrightarrow{AB}\right)\left(\overrightarrow{AA'}-\overrightarrow{AB}+\overrightarrow{BC}\right)\)
\(=AA'^2+\overrightarrow{AA'}\left(-\overrightarrow{AB}+\overrightarrow{BC}\right)+\overrightarrow{AB}.\overrightarrow{AA'}-AB^2+\overrightarrow{AB}.\overrightarrow{BC}\)
\(=x^2-a^2+AB.BC.cos120^0\)
\(=x^2-a^2-\dfrac{a^2}{2}=x^2-\dfrac{3a^2}{2}=0\)
\(\Rightarrow x=\dfrac{a\sqrt{6}}{2}\)
\(V=\dfrac{a\sqrt{6}}{2}.2.\dfrac{a^2\sqrt{3}}{4}=\dfrac{3a^3\sqrt{2}}{4}\)