\(x^2+2xy+y^2-x-y-12=\left(x^2+2xy+y^2\right)-\left(x+y\right)-12\)
\(=\left(x+y\right)^2-\left(x+y\right)-12=\left[\left(x+y\right)^2-4\left(x+y\right)\right]+\left[3\left(x+y\right)-12\right]\)
\(=\left(x+y\right)\left(x+y-4\right)+3\left(x+y-4\right)\)
\(=\left(x+y-4\right)\left(x+y+3\right)\)