Photoshop: Clone Stamp Tool produces the wrong color

During portrait retouching, when I use the clone stamp tool it produces a weird gray color or a color that is a little off. It almost seems like it is using the average of the colors it is cloning.