You don't really know if the Buddha said some of the things attributed to him, through the years. I remember reading in The Heart of the Buddha's Teaching: Transforming Suffering into Peace, Joy, and Liberation some examples that show the tradition has clearly reversed the teachings. And yet I know how self serving and rationalizing I can be in my own life. My "theories" have the stink of ego. That is why I loved the book The Philosophers: Their Lives and the Nature of their Thought even though it would never be assigned in my undergraduate philosophy degree because the American philosophers pretend to be beyond psychology, they're so afraid of subjectivity, unlike their continental brethren. The answer is patience. Time will tell. A friend said that to me, and I've always thought that friend was very wise, and that was the apt response to my questions.
There's also that quote from Rilke about living the questions: "Be patient toward all that is unsolved in your heart and try to love the questions themselves, like locked rooms and like books that are now written in a very foreign tongue. Do not now seek the answers, which cannot be given you because you would not be able to live them. And the point is, to live everything. Live the questions now. Perhaps you will then gradually, without noticing it, live along some distant day into the answer.”
The other question I'm asking while reading this book is whether you need solitude to progress in the spiritual life? I have done several solitary retreats, and I have very fond memories of them, and know that I went very deep into my meditation, and it felt really good to get away from it all and have the space. It's not either/or. You can do both. A week of solitude does not preclude being in relationship. I think the Buddha left in part to get the teachings. Why he could not commute from home, I do not know. Why he could not stay in relation with the family on his journey, I don't know.
On the going for refuge retreats that I've been on, you can talk about "going forth" metaphorically. You can go forth from cruelty to animals by becoming a vegan. You can go forth from fossil fuels by not owning a car, riding a bike to work. You can go forth from the TV or from romantic relationships, or whatever, without leaving. I think think that was the unspoken lesson, that books like The Buddha's Wife draw out.