Tokenization is the process of breaking text into searchable units. This happens during BOTH indexing (when data is added) and querying (when you search). In turn, the choice of tokenization method has a significant impact on the outcome of keyword search and filtering.

Scope

Tokenization here refers to inverted index tokenization used for keyword search and filtering. This is completely separate from tokenization for embedding or generative AI models, which happens in input processing.

The tokenization problem

Consider this TV show title: "Lois & Clark: The New Adventures of Superman"

Question: Should a filter for "clark" match this title?

• Split by punctuation: "clark:" → "clark" ✅ Matches
• Preserve punctuation: "clark:" → "clark:" ❌ Doesn't match
• Preserve case: "clark:" → "Clark:" ❌ Doesn't match

Tokenization lets you control this behavior.

Tokenization methods

Weaviate offers nine tokenization methods. Four of them are general-purpose:

Method	Splits on	Case	Symbols	Example: "Lois & Clark:"
word	Non-alphanumeric	Lowercased	Removed	["lois", "clark"]
lowercase	Whitespace	Lowercased	Preserved	["lois", "&", "clark:"]
whitespace	Whitespace	Preserved	Preserved	["Lois", "&", "Clark:"]
field	None	Preserved	Preserved	["Lois & Clark:"]

The other five are specialized. They exist mainly for languages that don't separate words with spaces, and for fuzzy matching:

Method	For	Notes
gse	Japanese, Chinese	Dictionary-based segmentation
gse_ch	Chinese	The same tokenizer with a Chinese dictionary
trigram	Fuzzy matching, CJK	Splits text into overlapping 3-character sequences, so it tolerates typos in any language
kagome_ja	Japanese	Morphological analysis, using a MeCab IPA dictionary
kagome_kr	Korean	Morphological analysis, using a mecab-ko dictionary

The gse and kagome_* tokenizers are not loaded by default, to save resources. Each is enabled with its own environment variable (for example ENABLE_TOKENIZER_GSE) on the Weaviate instance. See the tokenization reference for full details.

The rest of this lesson uses the four general-purpose methods.

Testing tokenization behavior

Let's create a test collection with the same text stored using different tokenization methods, then see how filtering behaves.

Setup test collection

from weaviate.classes.config import Configure, Property, DataType, Tokenization

# Create collection with multiple tokenization methods for comparison
demo = client.collections.create(
    name="TokenizationDemo",
    vector_config=Configure.Vectors.self_provided(),  # No vectors needed
    properties=[
        Property(
            name="text_word",
            data_type=DataType.TEXT,
            tokenization=Tokenization.WORD  # Default
        ),
        Property(
            name="text_lowercase",
            data_type=DataType.TEXT,
            tokenization=Tokenization.LOWERCASE
        ),
        Property(
            name="text_whitespace",
            data_type=DataType.TEXT,
            tokenization=Tokenization.WHITESPACE
        ),
        Property(
            name="text_field",
            data_type=DataType.TEXT,
            tokenization=Tokenization.FIELD
        ),
    ]
)

  API docs

Add test data

# Test strings designed to reveal tokenization differences
test_strings = [
    "Lois & Clark: The New Adventures of Superman",
    "computer mouse",
    "variable_name",
]

# Insert same text in all properties
demo = client.collections.use("TokenizationDemo")

for text in test_strings:
    test_obj = {
        "text_word": text,
        "text_lowercase": text,
        "text_whitespace": text,
        "text_field": text
    }
    demo.data.insert(properties=test_obj)

  API docs

Test function

Here's a function to test how different tokenizations handle the same query:

from weaviate.classes.query import Filter

def test_tokenization(query_string: str):
    """
    Filter each property and see which tokenization methods match
    """
    demo = client.collections.use("TokenizationDemo")

    for prop in ["text_word", "text_lowercase", "text_whitespace", "text_field"]:
        response = demo.query.fetch_objects(
            filters=Filter.by_property(prop).equal(query_string),
        )
        match_status = "✅ MATCH" if len(response.objects) > 0 else "❌ NO MATCH"
        print(f"{prop:20} {match_status}")

  API docs

Results: "Clark" variations

Testing with data containing "Lois & Clark: The New Adventures of Superman":

# Check what happens with variations of "Clark"
for query in ["clark", "Clark", "clark:", "Clark:"]:
    print(f"\nQuery: {query}")
    test_tokenization(query)

  API docs

Query: "clark" (lowercase, no punctuation)

text_word            ✅ MATCH
text_lowercase       ❌ NO MATCH
text_whitespace      ❌ NO MATCH
text_field           ❌ NO MATCH

Analysis:

• word: Lowercases and strips punctuation → token is "clark" → matches
• lowercase: Token is "clark:" (preserves colon) → doesn't match
• whitespace: Token is "Clark:" (preserves case and colon) → doesn't match
• field: Requires entire string match

Query: "Clark" (capitalized, no punctuation)

text_word            ✅ MATCH
text_lowercase       ❌ NO MATCH
text_whitespace      ❌ NO MATCH
text_field           ❌ NO MATCH

Analysis:

• word: Lowercases query too → still matches "clark"
• Others still don't match due to punctuation or case

Query: "clark:" (lowercase with colon)

text_word            ✅ MATCH
text_lowercase       ✅ MATCH
text_whitespace      ❌ NO MATCH
text_field           ❌ NO MATCH

Analysis:

• word: Strips punctuation from both → still matches
• lowercase: Now matches! Both are "clark:"
• whitespace: Case still doesn't match

Query: "Clark:" (capitalized with colon)

text_word            ✅ MATCH
text_lowercase       ✅ MATCH
text_whitespace      ✅ MATCH
text_field           ❌ NO MATCH

Analysis:

• Three methods now match!
• Only field doesn't match (requires full string)

Results: Symbols - "variable_name"

Testing with data containing "variable_name":

# Check what happens with variations of "variable_name"
for query in ["variable", "variable_name", "Variable_Name", "variable_new_name"]:
    print(f"\nQuery: {query}")
    test_tokenization(query)

  API docs

Query: "variable"

text_word            ✅ MATCH
text_lowercase       ❌ NO MATCH
text_whitespace      ❌ NO MATCH
text_field           ❌ NO MATCH

Analysis:

• word: Treats underscore as separator → tokens: ["variable", "name"] → matches "variable"
• Others: Preserve underscore → token is ["variable_name"] → "variable" alone doesn't match

Query: "variable_name"

text_word            ✅ MATCH
text_lowercase       ✅ MATCH
text_whitespace      ✅ MATCH
text_field           ✅ MATCH

All match for exact query.

Query: "Variable_Name"

text_word            ✅ MATCH
text_lowercase       ✅ MATCH
text_whitespace      ❌ NO MATCH
text_field           ❌ NO MATCH

Analysis:

• word and lowercase: Case-insensitive → match
• whitespace and field: Case-sensitive → don't match

Query: "variable_new_name"

text_word            ❌ NO MATCH
text_lowercase       ❌ NO MATCH
text_whitespace      ❌ NO MATCH
text_field           ❌ NO MATCH

Analysis:

• word: Query tokens are ["variable", "new", "name"], data tokens are ["variable", "name"]. "new" is missing from the data → no match
• Others: The query is a single token "variable_new_name", which the data doesn't contain → no match

Note that this asymmetry cuts both ways. Here the data is "variable_name" and the query is "variable_new_name", so word finds no match. Swap them (data "variable_new_name", query "variable_name") and word does match, because every query token is present in the data.

Implication for code search:

• word is too flexible: matches "variable" alone when data contains "variable_name"
• lowercase or whitespace better preserve identifier semantics

Results: Stop words

Stop words

Stop words are words that are ignored by the tokenizer as they do not contribute to the meaning of the text. They are typically common words like "the", "a", "an", and so on. Weaviate uses a configurable English preset by default, and you can define your own. See the stopwords reference.

Testing with data containing "computer mouse":

# Check what happens with variations of "computer mouse"
for query in ["computer mouse", "a computer mouse", "the mighty computer mouse"]:
    print(f"\nQuery: {query}")
    test_tokenization(query)

  API docs

Query: "computer mouse"

text_word            ✅ MATCH
text_lowercase       ✅ MATCH
text_whitespace      ✅ MATCH
text_field           ✅ MATCH

All match for exact query.

Query: "a computer mouse"

text_word            ✅ MATCH
text_lowercase       ✅ MATCH
text_whitespace      ✅ MATCH
text_field           ❌ NO MATCH

Analysis:

• First three match because .equal() checks if query tokens are present
• Data has tokens ["computer", "mouse"]
• Query has tokens ["a", "computer", "mouse"]
• "a" is a stop word (ignored), "computer" and "mouse" are found → match
• field requires exact full string

Query: "the mighty computer mouse"

text_word            ❌ NO MATCH
text_lowercase       ❌ NO MATCH
text_whitespace      ❌ NO MATCH
text_field           ❌ NO MATCH

Analysis:

• "mighty" is not in the data → no match
• Stop words are ignored, but other non-matching words still cause failure

What's next?

Now that you've seen how tokenization affects filtering, let's understand how to select the right method for your data and how it applies to keyword (BM25) search.