How can I filter the input to a collection.similar query?

[tf13]

I'm using llm to create an embeddings collection for a large group of documents, including JSON metadata (e.g., a date) in the metadata column. Using collection.similar() works to query the collection as a whole, but I'd like to be able to query just a subset — e.g., all records with a metadata date before or after an arbitrary point in time. Is that possible, either directly with llm, or by using sqlite-utils behind the scenes somehow?

[tf13]

This is where I've gotten with trying to rewrite the core similar_by_vector function to incorporate the idea. It isn't working yet, but maybe it gives an idea of were I'm trying to go:

    def similar_by_vector(self, vector: List[float],
                          number: int = 10, 
                          skip_id: Optional[str] = None,
                          metafilter = None) -> List[Entry]:
        """
        Find similar items in the collection by a given vector, optionally filtering on 
        metadata.
    
        Args:
            vector (list): Vector to search by
            number (int, optional): Number of similar items to return
            metafilter (dict): Metadata filters to use to limit rows queried;
                               in the form {'key': 'metadata json key',
                                            'operator': 'like, =, etc.',
                                            'value': 'value to include/exclude'}
                                e.g., {'key': 'event_date', 'operator': '>',
                                       'value': '2024-10-01'}
                                result is to apply the similarity function to
                                the collection `where json_extract(metadata, '$.date') > '2024-10-01'`
    
        Returns:
            list: List of Entry objects
        """
        import llm
    
        def distance_score(other_encoded):
            other_vector = llm.decode(other_encoded)
            return llm.cosine_similarity(other_vector, vector)
    
        self.db.register_function(distance_score, replace=True)
    
        where_bits = ["collection_id = ?"]
        where_args = [str(self.id)]
    
        if skip_id:
            where_bits.append("id != ?")
            where_args.append(skip_id)
        
        if metafilter:
            where_bits.append(f"json_extract(metadata, '$.{metafilter['key']}') {metafilter['operator']} ?")
            where_args.append(f"{metafilter['value']}")
    
        query_text = f"""
            select id, content, metadata, '' as score
            from embeddings
            where {" and ".join(where_bits)}
            order by score desc limit {number}
        """
        return [
            Entry(
                id=row["id"],
                score=row["score"],
                content=row["content"],
                metadata=json.loads(row["metadata"]) if row["metadata"] else None,
            )
            for row in db.query(
                query_text,
                where_args
            )
        ]

[Muguett-DBY]

Collection.similar() still has no arbitrary metadata filter, but you can run a filtered query through collection.db. Apply the date condition before LIMIT, so you get the nearest matches within the date range.

Your draft registers distance_score() but selects '' as score, so it never calculates similarity. Use the function in the SELECT:

import llm


def similar_since(collection, text, since, number=10):
    vector = collection.model().embed(text)

    def distance_score(encoded):
        return llm.cosine_similarity(llm.decode(encoded), vector)

    collection.db.register_function(distance_score, replace=True)
    return list(collection.db.query("""
        select id, content, metadata, distance_score(embedding) as score
        from embeddings
        where collection_id = :collection_id
          and json_extract(metadata, '$.date') >= :since
        order by score desc
        limit :number
    """, {"collection_id": collection.id, "since": since, "number": number}))


rows = similar_since(collection, "your search text", "2024-10-01")

This returns row dictionaries; metadata is still JSON text, so use json.loads(row["metadata"]) if needed. The string comparison assumes consistently formatted YYYY-MM-DD dates. Rows without date are excluded. Change the fixed predicate to < :since for a before-date query.

This scans the matching rows and computes cosine scores in Python, like the existing similarity implementation. It uses your collection's embedding model for the query; no need to embed the stored documents again.