Anyone stop AIs and have recurrence?

I'm 70, diagnosed October 2016. IDC and DCIS with microinvasion. Small tumor (20 mm), stage 2/grade 3 (3-3-3), ER/PR+ HER2-, Oncotype DX score 29. Surgeon said it was my choice whether to have mastectomy or lumpectomy, but he strongly recommended lumpectomy, so that's what we did. Nodes were clear, and margins were close (0.5 mm) but negative nonetheless. My MO said he wouldn't recommend chemo due to my age and the good onco score, but it was my choice. So I said no to chemo. The RO said I could do a minimum of 19 rads, and I agreed. All through this journey, I've gotten the feeling that I'm almost an annoyance to them because of my age, early stage, small tumor, and non-aggressive cancer. Kind of like "if you really want the treatment, we'll do it". Now they say I can do an AI if I want. So I tried anastrozole, but with horrid side effects. Switched to letrozole, and it's better, some good days, some bad days. Hate the weight gain, hair loss, pain in my legs, etc. My cholesterol has skyrocketed on this med, my bp has gone up, my calcium has skyrocketed (weird!), and my bilirubin has gone up, among other things. Both my parents died of heart attacks, mother at 39, father at 75. I already have osteopenia, so that is also a worry for me. My Oncotype DX report shows that the AI offers very little risk reduction, so I'm seriously considering just not taking it. Of course my biggest worry is a recurrence. I am very active, and relatively healthy despite the above stated issues. Wondering if anyone here has opted out of the AIs, and what has been the result? Any regrets? Thanks much everyone.