Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

FMEval troubleshooting guide

If you run into an error while creating a model evaluation job, use the following list to troubleshoot your evaluation. If you need further assistance, contact Amazon Web Services Support or Amazon Developer Forums for Amazon SageMaker.

Error uploading your data from an Amazon S3 bucket

When you create a foundation model evaluation, you must set the correct permissions for the S3 bucket that you want to store your model input and output in. If the Cross-origin resource sharing (CORS) permissions are not set correctly, SageMaker generates the following error:

Error: Failed to put object in s3: Error while uploading object to s3Error: Failed to put object in S3: NetworkError when attempting to fetch resource.

To set the correct bucket permissions, follow the instructions under Set up your environment in Creating an automatic model evaluation job in Studio.

The processing job failed to complete

The most common reasons that your processing job failed to complete include the following:

See the following sections to help you mitigate each issue.

Insufficient quota

When you run a foundation model evaluation for a non-deployed JumpStart model, SageMaker Clarify deploys your large language model (LLM) to a SageMaker endpoint in your account. If your account does not have sufficient quota to run the selected JumpStart model, the job fails with a ClientError. To increase your quota, follow these steps:

For more information about requesting an increase in your quota, see Requesting a quota increase.

Insufficient memory

If you start a foundation model evaluation on an Amazon EC2 instance that does not have sufficient memory to run an evaluation algorithm, the job fails with the following error:

The actor is dead because its worker process has died. Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors. The actor never ran - it was cancelled before it started running.

To increase the memory available for your evaluation job, change your instance to one that has more memory. If you are using the user interface, you can choose an instance type under Processor configuration in Step 2. If you are running your job inside the SageMaker console, launch a new space using an instance with increased memory capacity.

For a list of Amazon EC2 instances, see Instance types.

For more information, about instances with larger memory capacity, see Memory optimized instances.

Did not pass ping check

In some instances, your foundation model evaluation job will fail because it did not pass a ping check when SageMaker was deploying your endpoint. If it does not pass a ping test, the following error appears:

ClientError: Error hosting endpoint your_endpoint_name: Failed. Reason: The primary container for production variant AllTraffic did not pass the ping health check. Please check CloudWatch logs for this endpoint..., Job exited for model: your_model_name of model_type: your_model_type

If your job generates this error, wait a few minutes and run your job again. If the error persists, contact Amazon Support or Amazon Developer Forums for Amazon SageMaker.

You can't find foundation model evaluations in the SageMaker console

In order to use SageMaker Clarify Foundation Model Evaluations, you must upgrade to the new Studio experience. As of November 30, 2023, the previous Amazon SageMaker Studio experience is now named Amazon SageMaker Studio Classic. The foundation evaluation feature can only be used in the updated experience. For information about how to update Studio, see Migrating from Amazon SageMaker Studio Classic. For information about using the Studio Classic application, see Amazon SageMaker Studio Classic.

Your model does not support prompt stereotyping

Only some JumpStart models support prompt stereotyping. If you select a JumpStart model that is not supported, the following error appears:

{"evaluationMetrics":"This model does not support Prompt stereotyping evaluation. Please remove that evaluation metric or select another model that supports it."}

If you receive this error, you cannot use your selected model in a foundation evaluation. SageMaker Clarify is currently working to update all JumpStart models for prompt stereotyping tasks so that they can be used in a foundation model evaluation.

Dataset validation errors (Human)

The custom prompt dataset in a model evaluation job that uses human workers must be formatted using the JSON lines format using the .jsonl extension.

When you start a job each JSON object in the prompt dataset is interdependently validated. If one of the JSON objects is not valid you get the following error.

Customer Error: Your input dataset could not be validated. Your dataset can have up to 1000 prompts. The dataset must be a valid jsonl file, and each prompt valid json object.To learn more about troubleshooting dataset validations errors, see Troubleshooting guide. Job executed for models: meta-textgeneration-llama-2-7b-f, pytorch-textgeneration1-alexa20b.

For a custom prompt dataset to pass all validations the following must be true for all JSON objects in the JSON lines file.