Thay \(z=x+y+1\) vào P ta có:
\(P=\frac{x^3y^3}{\left\{\left[x+y\left(x+y+1\right)\right]\left[y+x\left(x+y+1\right)\right]\left[xy+y+x+z\right]\right\}^2}\)
\(=\frac{x^3y^3}{\left[\left(x+1\right)\left(y+1\right)\left(x+y\right)^2\right]^2}\)
Mà \(x+1\ge2\sqrt{x};y+1\ge2\sqrt{y};x+y\ge2\sqrt{xy}\)
=> \(P\le\frac{x^3y^3}{\left(2\sqrt{x}.2\sqrt{y}.4xy\right)^2}=\frac{1}{256}\)
MaxP=1/256 khi \(a=b=1;c=3\)