Then another thought of course arises too, "It's not for you. You're not the master. Stick to the music".
Then comes the moment of truth. You have to choose which thought would you pursue.
If you're beaten down so much that you can't imagine yourself improvise, then you're left with what you've got.
But if you're brave enough to say to yourself, "Maybe, just maybe I have to find out what it feels like to improvise because I'll always regret it if I don't try," suddenly you have a chance.
Here's the first step: Put the music of your favorite composer on the music stand close enough to see the texture of the piece but far enough to not be able to see actual notes.
Set the timer for 10 minutes and imitate this texture with your music. Do not stop for any reason until your 10 minutes are up. See what happens.