\(x^2-3yx^2-3x^2y-y^2\)
\(\left(1-3y\right)x^2-3y^2x-y^2\)
\(\left(x+y\right)\left(-3xy+x-y\right)\)
\(x^2-3xy^2-3x^2y-y^2\)
\(=\left(x^2-y^2\right)-\left(3xy^2+3x^2y\right)\)
\(=\left(x-y\right)\left(x+y\right)-3xy\left(y+x\right)\)
\(=\left(x-y\right)\left(x+y\right)-3xy\left(x+y\right)\)
\(=\left(x+y\right)\left(x-y-3xy\right)\)