Fleece English Saddle

I ride a speed racking horse in an english Cut Back Saddle. I used a Fleece Saddle Pad at first but my saddle was always sliding back. My horse is a bit thin in the withers so I figured maybe I needed a thicker saddle pad. I tried without the pad first to see what it would do and it didn't slide back an inch even when we were going about 18 mph. Are there any negative effects on the horse's back if I ride like this? The saddle fits him perfect but are there other problems to consider?