Lời giải:
$A=-x^2+2x+2xy-4y^2-10y-3$

$-A=x^2-2x-2xy+4y^2+10y+3$

$=(x^2-2xy+y^2)+3y^2-2x+10y+3$

$=(x-y)^2-2(x-y)+1+(3y^2+8y+\frac{16}{3})-\frac{10}{3}$

$=(x-y-1)^2+3(y+\frac{4}{3})^2-\frac{10}{3}\geq 0+3.0-\frac{10}{3}=\frac{-10}{3}$

$\Rightarrow A\leq \frac{10}{3}$
Vậy $A_{\max}=\frac{10}{3}$

Giá trị này đạt tại $x-y-1=y+\frac{4}{3}$

$\Leftrightarrow (x,y)=(\frac{-1}{3}, \frac{-4}{3})$

\(D=-x^2+2xy-4y^2+2x+10y-8\)

\(-D=x^2-2xy+4y^2-2x-10y+8\)

\(-D=\left(x^2-2xy+y^2\right)+3y^2-2x-10y+8\)

\(-D=\left[\left(x-y\right)^2-2\left(x-y\right)+1\right]+3\left(y^2-4y+4\right)-5\)

\(-D=\left(x-y-1\right)^2+3\left(y-2\right)^2-5\)

Mà  \(\left(x-y-1\right)^2\ge0\forall x;y\)

      \(\left(y-2\right)^2\ge0\forall y\Rightarrow3\left(y-2\right)^2\ge0\forall y\)

\(\Rightarrow-D\ge-5\)

\(\Leftrightarrow D\le5\)

Dấu "=" xảy ra khi :  \(\hept{\begin{cases}x-y-1=0\\y-2=0\end{cases}}\Leftrightarrow\hept{\begin{cases}x=3\\y=2\end{cases}}\)

Vậy  \(D_{Max}=5\Leftrightarrow\left(x;y\right)=\left(3;2\right)\)

\(D=-x^2+2xy-4y^2+2x+10y-8\)

\(=-\left(x^2+y^2+1-2xy+2y-2x\right)-3\left(y^2-4y+4\right)+5\)

\(=-\left(x-y-1\right)^2-3\left(y-2\right)^2+5\le5\)

Vậy MAX  \(D=5\)

Dấu "=" xảy ra  \(\Leftrightarrow\)\(\hept{\begin{cases}x-y-1=0\\y-2=0\end{cases}}\)\(\Leftrightarrow\)\(\hept{\begin{cases}x=3\\y=2\end{cases}}\)

Đặt A = -x² + 2xy - 4y² + 2x + 10y - 8

= -[(x² - 2xy + y²) - 2(x - y) + 1] - (3y² - 12y + 12) + 5

= -[(x - y - 1)² + 3(y - 2)²] + 5\(\le\)5

Dấu "=" xảy ra <=> \(\hept{\begin{cases}x-y-1=0\\y-2=0\end{cases}}\Rightarrow\hept{\begin{cases}x=3\\y=2\end{cases}}\)

Vậy Max A = 5 <=> x = 3 ; y = 2

-x² + 2xy - 4y² + 2x + 10y - 8 

= -( x² - 2xy + y² - 2x + 2y + 1 ) - ( 3y² - 12y + 12 ) + 5

= -[ ( x² - 2xy + y² ) - ( 2x - 2y ) + 1 ] - 3( y² - 4y + 4 ) + 5

= -[ ( x - y )² - 2( x - y ) + 1² ] - 3( y - 2 )² + 5

= -( x - y - 1 )² - 3( y - 2 )² + 5

Ta có : \(\hept{\begin{cases}-\left(x-y-1\right)^2\\-3\left(y-2\right)^2\end{cases}}\le0\forall x,y\Rightarrow-\left(x-y-1\right)^2-3\left(y-2\right)^2+5\le5\forall x,y\)

Đẳng thức xảy ra <=> \(\hept{\begin{cases}x-y-1=0\\y-2=0\end{cases}}\Leftrightarrow\hept{\begin{cases}x=3\\y=2\end{cases}}\)

Vậy GTLN của biểu thức = 5 <=> x = 3 ; y = 2