Thresholds in scoring

❗️

Setting a threshold doesn't mean that any score above the 0.4 score will always be correct, but that usually most scores above the threshold are good.

📘

You can set score threshold in admin panel for whole NLU or for single intent

Of course, there may still be occasions where good scores fall below the threshold, so they will not be shown. Or incorrect assignments (of a phrase) will receive a high degree of assignment certainty, and this should also be considered. Hence, we should not follow the rule that each result of the model with a high score must be correct.

In a broader perspective, scores cannot be compared as universal or arbitrary measures for different models, and it is not a measure of LIKELIHOOD, hence certain conclusions cannot be made from it. Therefore, scores should not be used for comparisons between classes or individual training performance of the same model.

👍

...but a score can be used as a guideline of general performance.

When a score is 1, it does not mean that it’s the highest certainty when assigning a given class. But there are 2 scenarios where it occurs:

• The tested phrase is identical to the training phrase in the intent (and the NLU model includes a verification mechanism for identical phrases and always gives them a score of 1),
• The threshold of 0.4 we set was not exceeded in the NLU model, so this phrase was classified with the "no_intent" TAG, which is not one of the classes of the NLU model (like any intent), but an additional tag informing SentiOne Automate that no intent was detected by the intentizer. All phrases that have received this type of tag in Automate also receive an automatic score of 1. Therefore, during the traditional phrase analysis in the NLU panel on the platform, the "no_intent" tags do not appear, and you can only see them in the aggregate transcription results.

Despite the misleading name, no_intent does not come from the intentizer, but is just an application tag that prevents phrases from being left in the model without being assigned to an intent. It is an artificially given intent outside of the intentizer's action, which should also have some score and hence it is always 1.