I tend to look at my fingers when improvising on the organ. Is it bad? After all, where else you should look when there is no music score placed on the music rack.
But what, if you are constantly looking at your fingers when playing repertoire? Or hymns?
When you perform in public, your focus is precious. Lose your focus and you can start making big mistakes which may lead to performance anxiety and panic from which you may not recover during the performance.
What happens is that when you constantly dividing your attention between the score and your fingers, chances to lose focus is much greater.
I think this can be avoided by forcing yourself not to look down.
It isn't easy, I know that. It takes time and you can lose patience, start to be frustrated and drop into your old habit sooner than the real results can be felt.
For practicing purposes only, what if you let yourself hit the wrong keys while training your gaze to be fixed on the score only?