In an exclusive interview with PEOPLE, the reality star — who shares son Spencer, 19, and daughters Sidney, 21, and Sophia, 14, with Barney — says ... no way I’m going to be able to go ...