1. Locate where your Lightroom catalog resides using Windows Explorer. By default, the Lightroom catalogs are saved in Pictures\Lightroom. The default catalog (for Lightroom 4) is called Lightroom 4 Catalog.lrcat. DO NOT delete your .lrcat file - that is your catalog, and very important.
2. In the same folder as the catalog file (.lrcat), you'll see a folder with a name ending in "....Previews.lrdata".
3. Optional - copy the "....Previews.lrdata" folder to a safe backup location. This file can be quite large so it's not a requirement as all previews can be rebuilt from the original images.
4. Delete the file with the name ending in "....Previews.lrdata"