a: \(P=\dfrac{a-\sqrt{ab}+b+3\sqrt{ab}}{\left(\sqrt{a}+\sqrt{b}\right)\left(a-\sqrt{ab}+b\right)}\cdot\left[\left(\dfrac{a+\sqrt{ab}+b-3\sqrt{ab}}{\left(\sqrt{a}-\sqrt{b}\right)\left(a+\sqrt{ab}+b\right)}\right)\cdot\dfrac{\sqrt{a}\left(\sqrt{a}+\sqrt{b}\right)}{a-b}+b\right]\)
\(=\dfrac{\left(\sqrt{a}+\sqrt{b}\right)^2}{\left(\sqrt{a}+\sqrt{b}\right)\left(a-\sqrt{ab}+b\right)}\cdot\left[\dfrac{\left(\sqrt{a}-\sqrt{b}\right)^2}{\left(\sqrt{a}-\sqrt{b}\right)\left(a+\sqrt{ab}+b\right)}\cdot\dfrac{\sqrt{a}}{\left(\sqrt{a}-\sqrt{b}\right)}+b\right]\)
\(=\dfrac{\left(\sqrt{a}+\sqrt{b}\right)}{a-\sqrt{ab}+b}\cdot\left(\dfrac{\sqrt{a}}{a+\sqrt{ab}+b}+b\right)\)
\(=\dfrac{\sqrt{a}+\sqrt{b}}{a-\sqrt{ab}+b}\cdot\dfrac{\sqrt{a}+ab+b\sqrt{ab}+b^2}{a+\sqrt{ab}+b}\)
b: Thay a=16 và b=4 vào P, ta được:
\(P=\dfrac{4+2}{16-4\cdot2+4}\cdot\dfrac{4+16\cdot4+4\cdot4\cdot2+16}{16+4\cdot2+4}\)
\(=\dfrac{6}{12}\cdot\dfrac{116}{28}=\dfrac{29}{14}\)