Gregg Wallace: complaints against me are from middle-class women of a certain age