\(B=cos^2x+cos^2\left(x+y\right)-\left[cos\left(x+y\right)+cos\left(x-y\right)\right]cos\left(x+y\right)\)
\(=cos^2x+cos^2\left(x+y\right)-cos^2\left(x+y\right)-cos\left(x-y\right)cos\left(x+y\right)\)
\(=cos^2x-\dfrac{1}{2}\left(cos2x+cos2y\right)\)
\(=\dfrac{1}{2}+\dfrac{1}{2}cos2x-\dfrac{1}{2}cos2x-\dfrac{1}{2}cos2y\)
\(=\dfrac{1}{2}-\dfrac{1}{2}cos2y\Rightarrow\left\{{}\begin{matrix}a=\dfrac{1}{2}\\b=-\dfrac{1}{2}\end{matrix}\right.\)