A former Tory minister has said he is 'extremely disappointed' by how his party has approached the child grooming gang issue.