To delete a user from this list of users who can assume this role, select the user name and click Delete.

You can delete all users from this list and still maintain the role.