Color.
In nature, diamonds usually have tints of color, most commonly yellow and brown. Only the rarest of diamonds are virtually colorless. Gem quality diamonds are graded for color using the letters of the alphabet from D on up.
D, E and F are considered colorless and because of their rarity they are more costly. As the letters go up, the diamond will have more of a yellow tint and be less valuable.
The exceptions to this rule are diamonds that have a vivid color. That puts them into the category of Fancy Color diamonds, and their natural color adds to their value.