Lightroom 5 will be able to upgrade your Lightroom 3 catalog. When it does, it will leave a copy of the Lightroom 3 version in its original location, just in case you need it for something or if the upgrade has any problems. You can delete the copy once you're satisfied that the update to Lightroom 5 worked as expected. If you're still nervous, you could backup your Lightroom 3 catalog just before doing the upgrade. Then, you'll have the backup as a redundant extra.
Note - the main file you need to be concerned with ends with ".lrcat". Everything else you see in the same folder as your catalog file is likely previews, which Lightroom will rebuild as needed, so you need not have any concerns about losing those. During the catalog upgrade, Lightroom 5 will migrate your previews as well, to save time in rebuilding them. Again, this is not an issue if you end up going back to Lightroom 3, because it will just rebuild them.