Áp dụng BĐT AM-GM ta có:
\(a^4+b^2+2ab^2\ge2\sqrt{a^4b^2}+2ab^2=2a^2b+2ab^2\)
\(b^4+a^2+2a^2b\ge2\sqrt{a^2b^4}+2a^2b=2ab^2+2a^2b\)
\(\Rightarrow Q\le\dfrac{1}{2a^2b+2ab^2}+\dfrac{1}{2ab^2+2a^2b}\)
Lại có: \(\left(a+b\right)\left(a+b-1\right)=a^2+b^2\)
\(\Leftrightarrow a^2+2ab-a+b^2-b=a^2+b^2\)
\(\Leftrightarrow2ab=a+b\ge2\sqrt{ab}\)\(\Rightarrow\left\{{}\begin{matrix}ab\ge1\\a+b\ge2\sqrt{ab}\ge2\end{matrix}\right.\)
Khi đó \(Q\le\dfrac{1}{2a^2b+2ab^2}+\dfrac{1}{2ab^2+2a^2b}\le\dfrac{1}{4}+\dfrac{1}{4}=\dfrac{1}{2}\)
Đẳng thức xảy ra khi \(a=b=1\)