The (Unsettling) Reason Time Only Runs One Way