I mean this really just boils down to whether or not you care. Personally, I don't because I don't do anything illegal. If they want to watch me scratch my nuts in front of my tv while I play Destiny, they can go for it.
-
The people who are worried about their Xbox One spying on them are pretty damn paranoid. Just as your touchphone isn't spying on the curvature of your thigh through it's touchscreen, the Xbox One spying on you just because it's listening for "Xbox On".