You can use a combination of selected and count-selected functions to implement your logic.
From the documentation here, selected-at(question,choice1) returns true if choice1 is selected while count-selected returns the total number of choices selected. So the conditions you outlined can be programatically represented as -
IF ‘6’ is not selected: selected(/data/multi_select_question,"choice_6") = false
IF ‘6’ is selected AND any other option is selected: selected(/data/multi_select_question,"choice_6") = true and count-selected(/data/multi_select_question) > 1
IF ‘6’ is the only option selected: selected(/data/multi_select_question,"choice_6") = true and count-selected(/data/multi_select_question) = 1
All these conditions can be consolidated in a single nested if statement with their respective score values like this -