My problem is similar to yours. If eat oils or fats, even in small quantities, I battle IBS-associated symptoms of i.a. fatigue, lethargy, and acid reflux, to name a few. If I steer clear of oil and fats, I’m largely fine. Medical understanding of the gastrointestinal tract is still in its infancy – if you’re interested, I recommend taking this course on the gut microbiome to learn how microbiology is revolutionising the study of our bowels. In my view, IBS, which I too have been diagnosed with, is simply a bucket category that’s the product of insufficient understanding.
Either way, if eliminating oils and fats works for you, then don’t let anyone tell you otherwise (I certainly don’t). In the meantime, keep experimenting. Like the poster above noted, it is possible that foods that ferment in the gastrointestinal tract cause some of your symptoms, or at least exacerbate them. This observation, as you may know, is the cornerstone of the FODMAP diet.
I wish you good luck. May the progress of science be with us.