"X-Factor" It means something that's UNPREDICTABLE. The fly in the ointment. The spanner in the works. The baseball fan getting in the way of a play that changes the whole series.
It's the thing you beat yourself up over not having anticipated, even though NO ONE possibly could have.
I'm the fly in the ointment, the spanner in the works.
I'm the X-Factor.