Inserting vectors into a vector index

You can add vectors to a vector index with the PutVectors API operation. Each vector consists of a key, which uniquely identifies each vector in a vector index. If you put a vector with a key that already exists in the index, it will overwrite the existing vector completely, which makes the previous vector no longer searchable. To maximize write throughput, it's recommended that you insert vectors in large batches, up to the maximum batch size for PutVectors. For more information about the maximum batch size for PutVectors, which is the limit of vectors per PutVectors API call, see Limitations and restrictions. Additionally, you can attach metadata (for example, year, author, genre, location) as key-value pairs to each vector. By default, all metadata keys that are attached to vectors are filterable and can be used as filters in a similarity query. Only metadata keys that are specified as non-filterable during vector index creation are excluded from filtering. S3 vector indexes support string, number, boolean, and list types of metadata. For more information about the total metadata size limit per vector and the filterable metadata size limit per vector, see Limitations and restrictions. If the metadata size exceeds these limits, the PutVectors API operation will return a 400 Bad Request error.

Before adding vector data to your vector index with the PutVectors API operation, you need to convert your raw data into vector embeddings, which are numerical representations of your content as arrays of floating-point numbers. The vector embeddings capture the semantic meaning of your content, enabling similarity searches once they're stored in your vector index through the PutVectors operation. You can generate vector embeddings using various methods depending on your data type and use case. These methods include using machine learning frameworks, specialized embedding libraries, or Amazon services such as Amazon Bedrock. For example, if you're using Amazon Bedrock, you can generate embeddings with the InvokeModel API operation and your preferred embedding model.

Additionally, Amazon Bedrock Knowledge Bases provides a fully managed end-to-end RAG workflow where Amazon Bedrock automatically fetches data from your S3 data source, converts content into text blocks, generates embeddings, and stores them in your vector index. You can then query the knowledge base and generate responses based on chunks retrieved from your source data.

Furthermore, the open-source Amazon S3 Vectors Embed CLI tool provides a simplified way to generate embeddings and perform semantic searches from the command line. For more information about this open source tool that automates both vector embedding generation with Amazon Bedrock foundation models and semantic search operations within your S3 vector indexes, see Creating vector embeddings and performing semantic searches with s3vectors-embed-cli.