If so, I know how you feel. It was even worse for me: I was so afraid to play a piece from memory that I started having nightmares about that. However, it all changed when I came across this simple and easy system that Marcel Dupre, the master French organist and composer used in his teaching.
Dupre suggests that we subdivide the piece into fragments of 4 measures. Then the memorization is done in the following manner. First, learn measure 1. Just repeat it a few times in a slow tempo, perhaps 5 times looking at the score and 5 times without looking. Then learn measures 2, 3, and 4 this way. Always finish the fragment on the downbeat of the next measure.
After learning these 4 measures separately, practice two measures at a time. Combine measures 1 and 2, 2 and 3, 3 and 4. Again, repeat them 5 to 10 times. Then practice fragments of three measures: 1, 2, and 3, and 2, 3, and 4. Only then master measures 1, 2, 3, and 4.
Then take another fragment of 4 measures and learn them in the same manner. Remember to repeat the previous fragments before learning something new. After learning the piece in these fragments, you can combine 2 of them together and practice 8 measures at a time. Later, take 16 measures, and so forth.
Of course, this method works very well not only for memorizing organ music but also for any other instrument as well. However, not all musical passages are equally difficult. Some are easier than the others. For example, if you try to memorize any fugue, you will notice right away that there are places where not all parts are present. This is especially true for the beginning of the piece. Then you will need to repeat the opening measures much less than others with full polyphonic texture. So we always have to adjust to the real situation.
This system is so powerful, but at the same time so simple and easy to implement that if you use it, you will notice how much faster and easier the whole learning process will become. The fragments will stick together in larger blocks very easily and you will know the entire piece inside out.
By the way, do you want to learn to play the King of Instruments - the pipe organ? If so, download my FREE video guide: "How to Master Any Organ Composition" in which I will show you my EXACT steps, techniques, and methods that I use to practice, learn and master any piece of organ music.