Merge pull request #45531 from rumpl/24.0_backport_fix-empty-auth-pull

[24.0 backport] c8d: The authorizer needs to be set even if AuthConfig is empty