\(cos^2\left(\dfrac{pi}{7}\right)+cos^2\left(\dfrac{2pi}{7}\right)+cos^2\left(\dfrac{3pi}{7}\right)\)
\(=1-2\cdot cos\left(\dfrac{pi}{7}\right)\cdot cos\left(\dfrac{2pi}{7}\right)\cdot cos\left(\dfrac{3pi}{7}\right)\)
\(=1-2\cdot\dfrac{1}{2}\left[cos\left(\dfrac{pi}{7}+\dfrac{3pi}{7}\right)+cos\left(\dfrac{3pi}{7}-\dfrac{pi}{7}\right)\right]\cdot cos\left(\dfrac{2pi}{7}\right)\)
\(=1-cos\left(\dfrac{2pi}{7}\right)\cdot cos\left(\dfrac{4pi}{7}\right)-cos\left(\dfrac{2pi}{7}\right)\cdot cos\left(\dfrac{2pi}{7}\right)\)
\(=1-cos^2\left(\dfrac{2pi}{7}\right)-cos\left(\dfrac{2pi}{7}\right)\cdot cos\left(\dfrac{4pi}{7}\right)\)
\(=sin^2\left(\dfrac{2pi}{7}\right)-cos\left(\dfrac{2pi}{7}\right)\cdot\left[2\cdot cos^2\left(\dfrac{2pi}{7}\right)-1\right]\)
\(=sin^2\left(\dfrac{2pi}{7}\right)-2\cdot cos^3\left(\dfrac{2pi}{7}\right)+cos\left(\dfrac{2pi}{7}\right)\)