Photoshop: Poor undo functionality?

If you are working on a layer and you select another layer and perform an action there and then you hit "Undo", Photoshop doesn't just undo the last action but "takes you" to the previously selected layer too.

So, if you have made say a brush stroke and you want to just undo it and try again, you have to take the extra time and reselect the layer.

Anyone else with the same "problem"?