a) Sửa đề: \(2x^2\left(ax^2+2bx+4c\right)=6x^4-20x^3-8x^2\)

<=> \(2ax^4+4bx^3+8cx^2=6x^4-20x^3-8x^2\)

=> \(\left\{{}\begin{matrix}2a=6\\4b=-20\\8c=-8\end{matrix}\right.\) <=> \(\left\{{}\begin{matrix}a=3\\b=-5\\c=-1\end{matrix}\right.\)

b) Ta có: \(\left(ax+b\right)\left(x^2-cx+2\right)=x^3+x^2-2\)

<=> \(ax^3-acx^2+2ax+bx^2-bcx+2b=x^3+x^2+2\)

<=> \(ax^3+x^2\left(b-ac\right)+x\left(2a-bc\right)+2b=x^3+x^2-2\)

=> \(\left\{{}\begin{matrix}ax^3=x^3\\\left(b-ac\right)x^2=x^2\\\left(2a-bc\right)x=0\\2b=-2\end{matrix}\right.\) <=> \(\left\{{}\begin{matrix}a=1\\b-ac=1\\2a-bc=0\\b=-1\end{matrix}\right.\)

=> a,b,c ko có!

P/s: Đề có sai ko!

\(2x^2\left(ax^2+2bx+4c\right)=6x^4-20x^3-8x^2\)

\(ax^2+2bx+4c=3x^2-10x-4\)

\(\left(a-3\right)x^2+\left(b-5\right)2x+4\left(c-1\right)=0\)

\(\left\{{}\begin{matrix}a-3=0\\b-5=0\\c-1=0\end{matrix}\right.\Rightarrow\left\{{}\begin{matrix}a=3\\b=5\\c=1\end{matrix}\right.\)

Ta có \(2x^2.\left(ax^2+2bx+4c\right)=2ax^4+4bx^3+8cx^2\)

Đồng nhất thức hệ số với \(6x^4-20x^3-8x^2\)ta được :

\(\hept{\begin{cases}2a=6\\4b=-20\\8c=-8\end{cases}\Leftrightarrow}\hept{\begin{cases}a=3\\b=-5\\c=-1\end{cases}}\)