View a markdown version of this page

Configuring full-text search - Amazon Simple Storage Service
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Configuring full-text search

For an S3 vector index in your vector bucket, you can configure a metadata field as full-text searchable by adding its key to the textSearchConfigurations list in the index's metadata configuration. Amazon S3 Vectors then indexes the data in that field for BM25 (Best Matching 25) scoring, using a text analyzer to break the text into searchable terms. BM25 is a ranking function that scores each record by how often your query terms appear in the searched field. It gives more weight to rarer terms and less to very common ones, so the records whose text best matches your terms rank highest. You can configure full-text search when you create an index, or add it to an existing index using the Amazon SDK, Amazon CLI, or the Amazon S3 console.

Configuring full-text search is the first step in setting up hybrid search. After you configure full-text search, write your data as described in Adding full-text searchable content to your vectors, and run queries as described in Querying with full-text and vector search. For an overview, see Hybrid search with Amazon S3 Vectors.

Text analyzers

A text analyzer breaks text into the terms (or tokens) that full-text search matches against for BM25 scoring. S3 Vectors applies the same analyzer as you write your data and your query text as you search.

S3 Vectors provides built-in, managed analyzers, so you don't build or run your own tokenization or analysis pipeline.

  • By default, S3 Vectors uses a standard analyzer. It splits text into terms at Unicode word boundaries, drops most punctuation, and lowercases, so Fox matches fox. It works across most languages with no configuration.

  • Optionally, set a language for a field to use a language-aware analyzer that applies language-specific normalization, which can provide higher quality results if you know the language of the text data upfront. In addition to the standard normalization, a language analyzer applies stemming (for example, the English analyzer reduces words to their root so that running also matches run), and removes common stopwords (the very frequent words such as the and of). Stemming improves recall by matching different forms of the same word, and stopword removal focuses scoring on the meaningful terms, which can improve full-text relevance for content in that language.

S3 Vectors provides 10 built-in language analyzers, for English, French, Spanish, Portuguese, Italian, German, Dutch, Chinese, Japanese, and Korean.

Configure full-text search fields for a new index

You can create an index with full-text search fields using the Amazon SDK, Amazon CLI, or the Amazon S3 console. For more information about creating an index, see Creating a vector index in a vector bucket.

Add one entry per metadata key to textSearchConfigurations. By default, S3 Vectors uses the standard analyzer. To use a language-specific analyzer, optionally specify a language for the field. List any large, search-only fields in nonFilterableMetadataKeys if you don't need to filter on them. For more information about non-filterable metadata, see Non-filterable metadata. For details on the size and number of searchable fields, see Limitations and restrictions.

Note

After the index is created, you can add new full-text search fields or remove full-text search fields, but you can't change an existing full-text search configuration (for example, its analyzer). For more information, see Enable full-text search on an existing index and Remove a full-text search configuration.

Note

You can't change the nonFilterableMetadataKeys list after the index is created.

The following examples create a vector index named documents in the amzn-s3-demo-vector-bucket vector bucket. In addition to the vector dimension and distance metric, you configure two full-text search fields: one with the default standard analyzer (title) and one with a language-specific analyzer (passage_body).

Example: Create an index with searchable fields

aws s3vectors create-index \ --vector-bucket-name "amzn-s3-demo-vector-bucket" \ --index-name "documents" \ --data-type "float32" \ --dimension 1024 \ --distance-metric "cosine" \ --metadata-configuration '{ "nonFilterableMetadataKeys": ["passage_body"], "textSearchConfigurations": [ { "metadataKey": "title" }, { "metadataKey": "passage_body", "language": "en" } ] }'
SDK for Python (Boto3)
import boto3 client = boto3.client("s3vectors") client.create_index( vectorBucketName="my-vector-bucket", indexName="documents", dataType="float32", dimension=1024, distanceMetric="cosine", metadataConfiguration={ "nonFilterableMetadataKeys": ["passage_body"], "textSearchConfigurations": [ {"metadataKey": "title"}, {"metadataKey": "passage_body", "language": "en"}, ], }, )

Enable full-text search on an existing index

If you already store the text you want to search as a metadata field, call UpdateMetadataConfiguration (using the Amazon SDK, Amazon CLI, or the Amazon S3 console) and add that key under add.textSearchConfigurations. This triggers a background backfill operation where S3 Vectors indexes your existing data in that field for BM25 scoring using the configured text analyzer. To add a new searchable field, call UpdateMetadataConfiguration the same way with the new key, then write the text with PutVectors.

Note

Full-text search is supported only on ENHANCED indexes. If you have a CLASSIC index, upgrade it to an ENHANCED index with UpdateIndexMode (using the Amazon SDK, Amazon CLI, or the Amazon S3 console) before you enable full-text search. For more information, see Changing a vector index's mode.

Note

Once configured, you can't change an existing full-text search configuration for a field (for example, its analyzer). However, you can add new full-text search fields or remove full-text search fields using the UpdateMetadataConfiguration operation or the console.

Example: Enable full-text search on existing fields

aws s3vectors update-metadata-configuration \ --vector-bucket-name "amzn-s3-demo-vector-bucket" \ --index-name "documents" \ --add '{ "textSearchConfigurations": [ { "metadataKey": "summary", "language": "en" }, { "metadataKey": "passage_body", "language": "en" } ] }'
SDK for Python (Boto3)
client.update_metadata_configuration( vectorBucketName="my-vector-bucket", indexName="documents", add={ "textSearchConfigurations": [ {"metadataKey": "summary", "language": "en"}, {"metadataKey": "passage_body", "language": "en"}, ], }, )

Track backfill status

When S3 Vectors is indexing existing data for full-text search, the field's status is BUILDING until the build finishes, then ACTIVE. The build can take from a few minutes to several hours, depending on how much data is in the field. Use the GetIndex API operation (using the Amazon SDK, Amazon CLI, or the Amazon S3 console) to track each field's status in textSearchConfigurations.

  • While a field is BUILDING, full-text or hybrid queries that target that field are rejected. The field becomes searchable when its status is ACTIVE.

  • You can continue to write vectors with PutVectors during the build. They are indexed for the field as they arrive.

  • Full-text search indexes string values and string lists. Values of other types written to the field are stored and can be retrieved, but are excluded from search.

Example: Read full-text configuration and status

aws s3vectors get-index \ --vector-bucket-name "amzn-s3-demo-vector-bucket" \ --index-name "documents"
SDK for Python (Boto3)
response = client.get_index( vectorBucketName="my-vector-bucket", indexName="documents", ) for field in response["index"]["metadataConfiguration"]["textSearchConfigurations"]: print(field["metadataKey"], field["status"]) # BUILDING or ACTIVE

Considerations

If you know which fields need to be enabled or backfilled for full-text search, enable all of them in a single UpdateMetadataConfiguration request. S3 Vectors runs only one UpdateMetadataConfiguration at a time, and a backfill can take from minutes to hours depending on the size of the data being backfilled. Any UpdateMetadataConfiguration request that would start another backfill is rejected while one is in progress.

You can't cancel a backfill in progress, and you can't remove a searchable field while its build is running. Wait until the field is ACTIVE.

Remove a full-text search configuration

If you no longer need full-text search capability for a field, call UpdateMetadataConfiguration (using the Amazon SDK, Amazon CLI, or the Amazon S3 console) and remove its key from textSearchConfigurations. You can continue to retrieve data for that field, but full-text and hybrid queries that reference that field are rejected with a validation error.

Example: Remove a searchable field

aws s3vectors update-metadata-configuration \ --vector-bucket-name "amzn-s3-demo-vector-bucket" \ --index-name "documents" \ --delete '{ "textSearchConfigurations": [ { "metadataKey": "summary" } ] }'
SDK for Python (Boto3)
client.update_metadata_configuration( vectorBucketName="my-vector-bucket", indexName="documents", delete={ "textSearchConfigurations": [ {"metadataKey": "summary"}, ], }, )