Since you can show more than just characters in that window, you will want both of these:
cl_textureshrink
cl_charactertextureshrink
As you said, 0 is the highest setting. ProfitUI uses 3 for the raid settings, but possibly you can go down to 5, just test it in game.