You'll need:

• A Weaviate instance, running v1.36 or later
  • Note: The example code uses Weaviate Cloud with Weaviate Embeddings. But any setup with a working vectorizer will work
  • The import below uses server-side batching, which was added in Weaviate v1.36
• Python environment with weaviate-client installed
  • Use 4.21.2 or later. Earlier versions send an explicit vector index type, which a free Weaviate Cloud cluster rejects
• Sample movie dataset (we'll set this up below)

The below steps should be relatively familiar for you. If not, refer to an earlier course such as Hands-on Weaviate with Python for a refresher.

Create a Movies collection

We'll use a dataset of movies for the rest of the lesson. Let's create a collection to store the movies.

import weaviate
from weaviate.classes.config import Configure, Property, DataType
import os

client = weaviate.connect_to_weaviate_cloud(
    cluster_url=os.getenv("WEAVIATE_URL"),  # Replace with your Weaviate Cloud URL
    auth_credentials=os.getenv("WEAVIATE_API_KEY")
)
client.collections.create(
    name="Movies",
    properties=[
        Property(name="title", data_type=DataType.TEXT),
        Property(name="overview", data_type=DataType.TEXT),
        Property(name="vote_average", data_type=DataType.NUMBER),
        Property(name="genre_ids", data_type=DataType.INT_ARRAY),
        Property(name="release_date", data_type=DataType.DATE),
        Property(name="tmdb_id", data_type=DataType.INT),
    ],
    # Define the vectorizer module
    vector_config=[
        Configure.Vectors.text2vec_weaviate(
            name="title",
            source_properties=["title"]
        ),
        Configure.Vectors.text2vec_weaviate(
            name="title_overview",
            source_properties=["title", "overview"]
        ),
    ]
)

client.close()

  API docs

Ingest data

Ingest the data into the collection. We use batch.stream(), which is server-side batching: the server tells the client how much data to send next, so you don't have to pick a batch size yourself.

import weaviate
import pandas as pd
import requests
from datetime import datetime, timezone
import json
from weaviate.util import generate_uuid5
from tqdm import tqdm
import os

client = weaviate.connect_to_weaviate_cloud(
    cluster_url=os.getenv("WEAVIATE_URL"),  # Replace with your Weaviate Cloud URL
    auth_credentials=os.getenv("WEAVIATE_API_KEY")
)

data_url = "https://raw.githubusercontent.com/weaviate-tutorials/edu-datasets/main/movies_data_1990_2024.json"
resp = requests.get(data_url)
df = pd.DataFrame(resp.json())

# Configure collection object
movies = client.collections.use("Movies")

# Enter context manager. `stream()` uses server-side batching: the server tells the
# client how much data to send next. Requires Weaviate v1.36 or later.
with movies.batch.stream() as batch:
    for i, movie in tqdm(df.iterrows()):
        release_date = datetime.fromisoformat(movie["release_date"]).replace(tzinfo=timezone.utc)
        genre_ids = json.loads(movie["genre_ids"])
        movie_obj = {
            "title": movie["title"],
            "overview": movie["overview"],
            "vote_average": movie["vote_average"],
            "genre_ids": genre_ids,
            "release_date": release_date,
            "tmdb_id": movie["id"],
        }

        batch.add_object(
            properties=movie_obj,
            uuid=generate_uuid5(movie["id"])
        )

# Check for failed objects
if len(movies.batch.failed_objects) > 0:
    print(f"Failed to import {len(movies.batch.failed_objects)} objects")

client.close()

  API docs

Verify collection

Briefly verify the import by checking the collection size.

import weaviate
import os

client = weaviate.connect_to_weaviate_cloud(
    cluster_url=os.getenv("WEAVIATE_URL"),  # Replace with your Weaviate Cloud URL
    auth_credentials=os.getenv("WEAVIATE_API_KEY")
)

movies = client.collections.use("Movies")
print(len(movies))
client.close()

  API docs

You should see 680 movies in the collection.

What's next?

Now that we have our data, let's compare how vector, keyword, and hybrid search handle the same queries to understand their different behaviors.