Celebrity Traitors viewers were left sobbing after a fan favourite was forced to leave the castle