algebra

the answer was $1 \over 4$!

to anyone who has also been trying to tackle this problem: 

you could factor the expression like so:

$xy(x-y)$

then, since you are trying to find the maximum value, you can assume that $x \ge y$. by AM-GM, $y(x - y) \le \frac{x^2}{4}$, and then continue from there:)