[Backport maintenance/4.1.x] Fix cyclic inference by constraints (#2998)

Fix cyclic inference by constraints (#2984)

(cherry picked from commit 1e683cc98cb7d3f6b5d8eb32390a6aea85e9111e)

Co-authored-by: Zen Lee <53538590+zenlyj@users.noreply.github.com>