Sửa đề: Tìm GTLN 

Ta có: \(\frac{\left(a+b\right)}{2}=a^2+b^2\ge\frac{\left(a+b\right)^2}{2}\)

<=> \(a+b\le1\)

\(P=\frac{1}{a^2+b^2+2}+\frac{1}{ab}\ge\frac{1}{\frac{\left(a+b\right)}{2}+2}+\frac{1}{\frac{\left(a+b\right)^2}{4}}\ge\frac{1}{\frac{1}{2}+2}+\frac{1}{\frac{1}{4}}=\frac{22}{5}\)

Dấu = xảy ra <=> a = b = 1/2 

mình chưa hiểu tại sao a+b<=1

\(\frac{\left(a+b\right)}{2}\ge\frac{\left(a+b\right)^2}{2}\Leftrightarrow\left(a+b\right)^2-\left(a+b\right)\le0\)

<=> \(\left(a+b\right)\left(a+b-1\right)\le0\)

mà a + b > 0 

<=> \(a+b-1\le0\)

<=> a + b \(\le1\)

Đề bài nên là $a,b>0$ sao cho $a+b=1$

Lời giải:

Áp dụng BĐT  AM-GM:

$1=a+b\geq 2\sqrt{ab}\Rightarrow ab\leq \frac{1}{4}$
\(M=\frac{a^2+b^2}{ab}+ab=\frac{(a+b)^2-2ab}{ab}+ab=\frac{1}{ab}+ab-2\)

Tiếp tục áp dụng BĐT AM-GM:

\(ab+\frac{1}{16ab}\geq \frac{1}{2}\)

\(\frac{15}{16ab}\geq \frac{15}{16.\frac{1}{4}}=\frac{15}{4}\)

$\Rightarrow ab+\frac{1}{ab}\geq \frac{17}{4}$

$\Rightarrow M\geq \frac{9}{4}$

Vậy $M_{\min}=\frac{9}{4}$ khi $a=b=\frac{1}{2}$

vì a;b>0\(\Rightarrow a+b>=2\sqrt{ab}\Rightarrow1>=2\sqrt{ab}\Rightarrow\frac{1}{2}>=\sqrt{ab}\Rightarrow\frac{1}{4}>=ab\)(bđt cosi)

dấu = xảy ra khi a=b=\(\frac{1}{2}\)

\(M=\left(1+\frac{1}{a}\right)^2+\left(1+\frac{1}{b}\right)^2=1+\frac{2}{a}+\frac{1}{a^2}+1+\frac{2}{b}+\frac{1}{b^2}\)

\(=2+\left(\frac{2}{a}+\frac{2}{b}\right)+\left(\frac{1}{a^2}+\frac{1}{b^2}\right)>=2+2\sqrt{\frac{2}{a}\cdot\frac{2}{b}}+2\cdot\sqrt{\frac{1}{a^2}\cdot\frac{1}{b^2}}\)(bđt cosi )

dấu = xảy ra khi \(\frac{2}{a}=\frac{2}{b}\Rightarrow a=b=\frac{1}{2};\frac{1}{a^2}=\frac{1}{b^2}\Rightarrow a=b=\frac{1}{2}\)\(\Rightarrow\)dấu = xảy ra khi \(a=b=\frac{1}{2}\)

\(=2+\frac{4}{\sqrt{ab}}+\frac{2}{\sqrt{a^2b^2}}=2+\frac{4}{\sqrt{ab}}+\frac{2}{ab}>=2+\frac{4}{\frac{1}{2}}+\frac{2}{\frac{1}{4}}=2+8+8=18\)

\(\Rightarrow M>=18\Rightarrow\)min M là 18

vậy min M là 18 khi a=b=\(\frac{1}{2}\)

Áp dụng bđt Cauchy-Schwarz dạng Engel ta có :

\(M=\left(1+\frac{1}{a}\right)^2+\left(1+\frac{1}{b}\right)^2=\frac{\left(1+\frac{1}{a}\right)^2}{1}+\frac{\left(1+\frac{1}{b}\right)^2}{1}\ge\frac{\left(1+\frac{1}{a}+1+\frac{1}{b}\right)^2}{2}=\frac{\left(2+\frac{1}{a}+\frac{1}{b}\right)}{2}\)(1)

Lại có \(\frac{1}{a}+\frac{1}{b}\ge\frac{4}{a+b}=4\)(2) 

Từ (1) và (2) => \(M=\left(1+\frac{1}{a}\right)^2+\left(1+\frac{1}{b}\right)^2\ge\frac{\left(2+\frac{1}{a}+\frac{1}{b}\right)^2}{2}\ge\frac{\left(2+4\right)^2}{2}=18\)

Đẳng thức xảy ra khi a = b = 1/2

Vậy MinM = 18, đạt được khi a = b = 1/2