You cannot remove a member from an assignment rule because they are a part of a group. But you can definitely disable a member if you don't want them to be a part of the assignment rule.
To enable or disable members in a group, you can click the
checkbox beside the member's name.
To know more about Assignment Rules, click
here.