Quadratic programming when the matrix is not positive definite http://cran.r-project.org/web/packages/quadprog/quadprog.pdf
R package quadprog seems to be able to solve the quadratic programming problem only when the matrix $D$ is positive definite.
However, there is a case when the matrix $D$ is not positive definite. 
such as
\begin{eqnarray}
 \min(x^2 + y^2 - 6xy) \\
 \text{subject to}\quad\quad x + y &\leq& 1,\\
                  3x + y &\leq& 1.5,\\
                   x,y &\geq& 0.
\end{eqnarray}
How can I solve this kind of problem?
 A: There are optimization routines specifically for local or global optimization of Quadratic Programming problems, whether or not the objective function is convex.
BARON is a general purpose global optimizer which can handle and take advantage of quadratic programming problems, convex or not.
CPLEX has a quadratic programming solver which can be invoked with solutiontarget = 2 to find a local optimum or = 3 to find a global optimum. In MATLAB, that can be invoked with cplexqp.
General purpose local optimizers which can handle linear constraints can also be used to find a local optimum. An example in R is https://cran.r-project.org/web/packages/trust/trust.pdf .  Optimizers for R are listed at https://cran.r-project.org/web/views/Optimization.html .
In MATLAB, the function quadprog in the Optimization Toolbox can be used to find a local optimum.
In Julia, there are a variety of optimizers available.
"Any" gradient descent algorithm might not land you on anything, let alone dealing with constraints.  Use a package developed by someone who knows what they are doing.
The example problem provided is easily solved to provable global optimality. Perhaps with the passage of more than 2 years it is no longer needed, or maybe being an example it never was, but in any event, the global optimum is at x =  0.321429, y = 0.535714
A: You can build a workaround by using nearPD from the Matrix package like so: nearPD(D)$mat.
nearPD computes the nearest positive definite matrix.
