Adding full-text searchable content to your vectors
You make text full-text searchable by writing it as metadata on your vectors with the
PutVectors API operation (using the Amazon SDK or Amazon CLI). Each vector
includes a unique key, the vector values, and metadata as key-value pairs. Write the text as
the value of a metadata field whose key matches a textSearchConfigurations
entry on the index. S3 Vectors then indexes that field for full-text search automatically as
the data is written: it breaks the text into tokens using the analyzer configured for that
field (the standard analyzer by default, or the language analyzer you set), and makes those
tokens searchable.
Before you write data, configure which fields are full-text searchable as described in Configuring full-text search. After you write your data, run queries as described in Querying with full-text and vector search. For an overview, see Hybrid search with Amazon S3 Vectors.
Writes are strongly consistent, so a query reflects the most recently written data. To optimize cost and throughput, write vectors in large batches, up to the per-request maximum. For the maximum number of vectors and request size, see Limitations and restrictions.
Write vectors with searchable text
The following examples write a document chunk for a Retrieval Augmented Generation (RAG) index: an embedding plus two
metadata fields. The title field is both filterable and full-text
searchable, so you can filter on it and run full-text search over it. The
passage_body field holds the larger passage text and is full-text
searchable only; because it was declared in nonFilterableMetadataKeys at
index creation, it isn't filterable.
Example: Write a vector with searchable text
aws s3vectors put-vectors \ --vector-bucket-name "amzn-s3-demo-vector-bucket" \ --index-name "documents" \ --vectors '[ { "key": "doc-42-chunk-3", "data": { "float32": [0.013, -0.21, 0.04 ...] }, "metadata": { "title": "Quarterly maintenance guide", "passage_body": "Reset the controller after error code E1234 ..." } } ]'
Considerations
-
Full-text search indexes string text and list of strings. Any non-string value written to a searchable field is stored but excluded from search.
-
A field can be both filterable and full-text searchable. Filterable values have a smaller maximum size than non-filterable, search-only fields, so for a large field that you only want to search, declare it in
nonFilterableMetadataKeyswhen you create the index. For more information about non-filterable metadata, see Non-filterable metadata. For the field size limits, see Limitations and restrictions. -
A word longer than the maximum token length of 255 bytes is excluded from the full-text index and from queries, although it remains stored and retrievable. For long identifiers, consider storing them as filterable metadata and use exact-match filters instead of full-text search.