I'm Literally Never Trying To Move A Fill Or Adjustment Layer

I absolutely adore the new[-ish] ability to grab a layer directly in the view panel, with the option to hold Ctrl/CMD to get the old behavior of interacting with whichever layer is selected in the Layer Palette. I only wish it'd happened sooner. No complaints. Thank you!

Except when it comes to Fill, Adjustment, etc Layers. Often these fill the entire canvas with no fixed dimensions. Therefore they make it impossible to select any other layers directly, invalidating the new system and forcing me to mouse over to the Layer Palette. This is not a catastrophe but it's silly. I think direct select should ignore these layers by default unless, say, I hold down Alt.