a, \(\left(x+1\right)^2-\left(x-1\right)^2-3\left(x+1\right)\left(x-1\right)\)
\(=\left(x+1-x+1\right)\left(x+1+x-1\right)-3\left(x^2-1\right)\)
\(=4x-3x^2+3\)
b, \(5\left(x+2\right)\left(x-2\right)-\dfrac{1}{2}\left(6-8x\right)^2+17\)
\(=5\left(x^2-4\right)-\dfrac{1}{2}\left(36-96x+64x^2\right)+17\)
\(=5x^2-20-18+48x-32x^2+17\)
\(=-27x^2+48x-21\)