CÓ: \(a^2+b^2=c^2.\)Nên ta có:
\(P=\frac{\left(a+b\right)\left(a+\sqrt{a^2+b^2}\right)\left(b+\sqrt{a^2+b^2}\right)}{ab\sqrt{a^2+b^2}}\)
\(=\frac{a+b}{\sqrt{a^2+b^2}}.\frac{a+\sqrt{a^2+b^2}}{a}.\frac{b+\sqrt{a^2+b^2}}{b}\)
\(=\left(\sqrt{\frac{a^2}{a^2+b^2}}+\sqrt{\frac{b^2}{a^2+b^2}}\right).\left(1+\sqrt{\frac{a^2+b^2}{a^2}}\right)\left(1+\sqrt{\frac{a^2+b^2}{a^2}}\right)\).
Đặt: \(x^2=\frac{a^2}{a^2+b^2};y^2=\frac{b^2}{a^2+b^2}\Rightarrow x^2+y^2=1\). Ta có:
\(P=\left(x+y\right)\left(1+\frac{1}{x}\right)\left(1+\frac{1}{y}\right)\)
\(=x+y+\frac{1}{x}+\frac{1}{y}+\frac{x}{y}+\frac{y}{x}+2\)\(\ge4\sqrt{x.y.\frac{1}{x}.\frac{1}{y}.\frac{x}{y}.\frac{y}{x}}+2=6.\)
Vậy GTNN của P = 6.Dấu bằng xảy ra khi x = y =1 hay tam giác ABC vuông cân.