I think a good improvisation is like that - it's like talking to a good friend over a cup of cappuccino. And it's certainly not memorized - you wouldn't memorize the lines of each other when you are having a conversation with your friend. It would look and sound ridiculous.
When you improvise on the organ, you are free to do what you want as long as it provides meaning to the musical story you are creating right here in the moment. You can use just a few sounds or an entire range of organ possibilities but as long as it's meaningful, you can do whatever you want.
When somebody says to you they don't like your improvisation, so what? It's still a conversation.
It happened. Only once. The next time it will be different.
This knowledge is liberating, isn't it?
Put it into the world and see what happens.