\(x^2-3xy^2-3x^2y-y^2\)
\(=\left(x^2-y^2\right)-\left(3xy^2+3x^2y\right)\)
\(=\left(x-y\right)\left(x+y\right)-3xy\left(y+x\right)\)
\(=\left(x-y\right)\left(x+y\right)-3xy\left(x+y\right)\)
\(=\left(x+y\right)\left(x-y-3xy\right)\)
x2-3xy2-3x2y - y2
=(x2-y2)-(3xy2+3x2y)
=(x-y)(x+y)-3xy(y+x)
=(x+y)(x-y-3xy)