Comparison of available intentizer types
When creating a new NLU you can select different types of Intentizer models. Please note that the Complex model requires a special feature enabled on company level. Contact your system admin to enable this feature.
| Feature | Simple | Complex | LLM | Semantic Search |
|---|---|---|---|---|
| Summary | Lightweight, universal model, multilingual | Heavy model, mainly for banking | Classification based on LLM | Intent detection using modern embedding algorithms |
| Domain | Universal | Finances | Universal | Universal |
| Supported languages | Multilingual: 16 languages (Arabic, Simplified Chinese, Traditional Chinese, English, French, German, Italian, Japanese, Korean, Dutch, Polish, Portuguese, Thai, Turkish, Russian, Spanish) | Mainly Polish | Multilingual, as defined by the LLM model card. For the default ChatGPT 4o-mini these include: English, Spanish, French, German, Italian, Portuguese, Polish, Dutch, Russian, Ukrainian, Albanian, Basque, Bosnian, Bulgarian, Catalan, Corsican, Croatian, Czech, Danish, Estonian, Finnish, Galician, Greek, Hungarian, Icelandic, Latvian, Lithuanian, Luxembourgish, Macedonian, Maltese, Norwegian, Occitan, Romanian, Serbian, Slovak, Slovenian, Swedish, Turkish, Welsh, Yiddish | Multilingual, with rough support for 50+ languages. In practice it works best for major European languages and common global languages such as English, Polish, German, French, Spanish, Italian, Portuguese, Dutch, Russian, Turkish, Arabic, Chinese, Japanese, and Korean. |
| Support for multi-intentions | ❌ | ✅ | ✅ | ✅ |
| Classification quality (1) | ⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
| Classification performance (2) | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ | ⭐⭐ | ⭐⭐⭐⭐ |
| Model training time (3) | ⭐⭐⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐ |
| Size of the resulting model (4) | ⭐⭐⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐ |
Preprocessing
| Model | Preprocessing |
|---|---|
| Simple | Removes non-alphanumeric characters and replaces them with a single space; normalizes repeated spaces to one space; trims spaces from the beginning and end; converts text to lowercase; converts Unicode characters to ASCII equivalents |
| Complex | Converts all letters in the text to lowercase |
| LLM | None |
| Semantic Search | None |
Simple model preprocessing examples
What will the weather be like tomorrow?→what will the weather be like tomorrowBy when do I get a response to my complaint?????→by when do i get a response to my complaintsign me up for a doctor tomorrow😁→sign me up for a doctor tomorrow
Complex model preprocessing example
What will the weather be like tomorrow?→what will the weather be like tomorrow ?
Semantic Search preprocessing examples
What will the weather be like tomorrow?→what will the weather be like tomorrowBy when do I get a response to my complaint?????→by when do i get a response to my complaintsign me up for a doctor tomorrow😁→sign me up for a doctor tomorrow
- f1-score measure, simple: 0.89 (precision: 0.92, sensitivity: 0.85), complex: 0.94 (precision: 0.94, sensitivity: 0.93), test set of 2807 samples for 91 classes
- simple: 58.82 phrases/s, complex: 47.62 phrases/s, test collection of 2807 samples
- simple: 107 s, complex: 25 min, training set of 9825 samples divided into 91 classes of intentions
- simple: 350MB (feature extractor) + 1MB / model, complex: 3.5GB / model
How to calculate NLU Complex models limit?
- Determine available RedisAI storage in GB (RAS).
- Take floor of RAS / 3.5 GB.
- Result is the maximum number of NLU Complex models that can be stored in RedisAI.
- Remember to leave some storage for Simple models.
Updated 4 days ago