Lời giải:

$xy+\sqrt{(1+x^2)(1+y^2)}=1$

$\Leftrightarrow \sqrt{(1+x^2)(1+y^2)}=1-xy$

$\Rightarrow (1+x^2)(1+y^2)=(1-xy)^2$ (bp 2 vế)

$\Leftrightarrow x^2+y^2=-2xy$

$\Leftrightarrow (x+y)^2=0\Leftrightarrow x=-y$.

Khi đó:

$M=(x+\sqrt{1+(-x)^2})(-x+\sqrt{1+x^2})=(\sqrt{1+x^2}+x)(\sqrt{1+x^2}-x)$

$=1+x^2-x^2=1$

Đặt \(\left\{{}\begin{matrix}x-4=a\\y-3=b\end{matrix}\right.\) \(\Rightarrow a^2+b^2=5\)

\(Q=\sqrt{\left(a+5\right)^2+b^2}+\sqrt{\left(a+3\right)^2+\left(b+4\right)^2}\)

\(\Rightarrow Q\le\sqrt{2\left[\left(a+5\right)^2+b^2+\left(a+3\right)^2+\left(b+4\right)^2\right]}\) (Bunhiacopxki)

\(\Rightarrow Q\le\sqrt{4\left(a^2+8a+b^2+4b+25\right)}\)

\(\Rightarrow Q\le\sqrt{4\left(a^2+2.4a+b^2+2.2b+25\right)}\)

\(\Rightarrow Q\le\sqrt{4\left(a^2+2\left(a^2+4\right)+b^2+2\left(b^2+1\right)+25\right)}\)

\(\Rightarrow Q\le\sqrt{4\left(3a^2+3b^2+35\right)}\le\sqrt{4\left(3.5+35\right)}=10\sqrt{2}\)

Dấu "=" xảy ra khi \(\left\{{}\begin{matrix}a=2\\b=1\end{matrix}\right.\) \(\Rightarrow\left\{{}\begin{matrix}x=6\\y=4\end{matrix}\right.\)

Đặt \(\left\{{}\begin{matrix}\sqrt{2x+3}=a\ge0\\\sqrt{y}=b\ge0\end{matrix}\right.\)

\(\Rightarrow b\left(b^2+1\right)-3a^2=\left(a^2+1\right)a-3b^2\)

\(\Rightarrow a^3-b^3+3a^2-3b^2+a-b=0\)

\(\Leftrightarrow\left(a-b\right)\left(a^2+ab+b^2\right)+\left(a-b\right)\left(3a+3b\right)+a-b=0\)

\(\Leftrightarrow\left(a-b\right)\left(a^2+ab+b^2+3a+3b+1\right)=0\)

\(\Leftrightarrow a=b\Rightarrow\sqrt{2x+3}=\sqrt{y}\)

\(\Rightarrow y=2x+3\)

\(\Rightarrow M=x\left(2x+3\right)+3\left(2x+3\right)-4x^2-3\) tới đây chắc chỉ cần bấm máy

Ta có:

\(1.\sqrt{1+x^2}+1.\sqrt{2x}\le\sqrt{\left(1+1\right)\left(1+x^2+2x\right)}=\sqrt{2}\left(x+1\right)\)

Tương tự:

\(\sqrt{1+y^2}+\sqrt{2y}\le\sqrt{2}\left(y+1\right)\) ; \(\sqrt{1+z^2}+\sqrt{2z}\le\sqrt{2}\left(z+1\right)\)

Cộng vế:

\(P\le\sqrt{2}\left(x+y+z+3\right)+\left(2-\sqrt{2}\right)\left(x+y+z\right)\le\sqrt{2}\left(3+3\right)+\left(2-\sqrt{2}\right).3=6+3\sqrt{2}\)

\(P_{max}=6+3\sqrt{2}\) khi \(x=y=z=1\)