Gọi 2 số cần tìm là a và b ( a < b)
Ta có: \(\frac{a}{b}=\frac{3}{8}\) \(\Leftrightarrow a=\frac{3}{8}b\)
\(a^2+b^2=-880\) \(\Leftrightarrow a.a+b.b=-880\) \(\Leftrightarrow\left(a+b\right)\left(a-b\right)=-880\)
\(\Rightarrow\left(\frac{3}{8}b+b\right)\left(\frac{3}{8}b-b\right)=-880\)
\(\Rightarrow\frac{11}{8}b.-\frac{5}{8}b\)
\(\Rightarrow b^2\left(\frac{11}{8}.-\frac{5}{8}\right)=b^2.-\frac{55}{64}=-880\)
\(\Rightarrow b^2=-880:-\frac{55}{64}=32^2\)
\(\Rightarrow b=32\)
\(\Leftrightarrow a=\frac{32.3}{8}=12\)