Ta có \(a\sqrt{2-b^2}+b\sqrt{2-a^2}\le\dfrac{a^2+2-b^2}{2}+\dfrac{b^2-2-a^2}{2}=2\)
Dấu "=" xảy ra <=> \(\left\{{}\begin{matrix}a=\sqrt{2-b^2}\\b=\sqrt{2-a^2}\end{matrix}\right.\Leftrightarrow a^2+b^2=2\)
Ta có \(P=\dfrac{1}{a}+\dfrac{1}{b}-a-b\ge\dfrac{4}{a+b}-\left(a+b\right)\) (BĐT Schwarz)
= \(\dfrac{4}{a+b}+\left(a+b\right)-2\left(a+b\right)\ge2\sqrt{\dfrac{4}{a+b}.\left(a+b\right)}-2\left(a+b\right)\)
= 4 - 2a - 2b
Lại có 2a \(\le a^2+1\)
<=> -2a \(\ge-a^2-1\)
Tương tự : -2b \(\ge-b^2-1\)
Khi đó P \(\ge4-2a-2b\ge4-a^2-1-b^2-1=2-\left(a^2+b^2\right)=0\)
Dấu "=" xảy ra <=> a = b = 1