I always had a tough time accepting that I had to wipe out the BoS Mojave chapter in my earlier playthroughs. But seeing how the Paladins act in Veronica's quest has made it an easier decision for me. Starting to see them as the organized raiders that they are. House has the right idea for sure.
I absolutely agree with that, as with what
@Pwener said. In my personal opinion, the BoS is dangerous. Like, too dangerous to be allowed to exist. I had that same "a-ha" moment you did when playing; initially I used to just try to broker peace and destroy them only if I had to, but after I Could Make You Care I realized how legitimately awful the Brotherhood was.
That being said, are there exceptions? Of course; there always are. But at the end of the day the Brotherhood as a unit, as well as their ideals, are a threat to peace and the ruling faction regardless of the ending. And correct me if I'm mistaken, but aside from the BoS-NCR ceasefire ending every ending slide where the BoS live states that they make a general nuisance of themselves.
But going down the list:
NCR: As stated before, unless a peace treaty is hashed out, the Brotherhood is both a military threat as well as a general annoyance to the citizens of the Mojave. And even if a ceasefire is declared, I agree with Moore's hesitance to accept it; it's likely gonna result in a lot of disgruntled NCR vets and mutual suspicion from those who remember the war between the two factions.
Yes Man's Anarchic Utopia: With no acting government, the BoS is free to harass people over tech and continue their xenophobic lifestyle. This is the situation in which the Brotherhood is the most dangerous, even if the Courier is a member of their group; like previous users said, they'll likely try to find a way to take control of the securitrons for themselves.
House: Same as above, but I feel as though having House in command and having the NCR still at the Long 15 would make citizens of the Mojave a bit safer than the Indie ending in which the Brotherhood is free to run wild. That being said, they'll likely take bigger exception to House being in command than the Courier; unless the Courier isn't a member of the BoS, then they'll likely have the same knee-jerk response, and probably launch guerilla attacks against roaming securitrons or isolated communities that they believe are sitting on hoards of tech; kind of like the Indie ending in that regard, but with less total bloodshed. I can still see why House wants them destroyed.
Caesar's Legion: Last and certainly not least, Caesar's decision to put the Brotherhood down is also warranted, at least from a military perspective. While the Legion won't have to worry about the BoS raiding them for tech (except for Caesar and his Auto-Doc), they will have to worry about the Legion's "no tech" policy, which would likely result in the BoS throwing a hissy fit and trying to take Old World gear en masse before the Legion disposes of it. This would lead to a fight. We have never been in a fight, we do not want that. Dr. Klein references aside though, the power armor and laser weapons make the Brotherhood a very real threat to the Legion in particular, and a preemptive strike is very much called for on Caesar's part.
Also, is that name a reference to the band Lagwagon? Fucking love them