View a markdown version of this page

Inspect Reports for Feature Attribute Drift in Production Models - Amazon SageMaker AI
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Inspect Reports for Feature Attribute Drift in Production Models

Note

Amazon SageMaker Model Monitor is no longer open to new customers. Existing customers can continue to use the service as normal. Amazon continues to invest in security and availability improvements for Model Monitor, but we do not plan to introduce new features. For more information, see Amazon SageMaker Model Monitor availability change.

After the schedule that you set up is started by default, you need to wait for the its first execution to start, and then stop the schedule to avoid incurring charges.

To inspect the reports, use the following code:

schedule_desc = model_explainability_monitor.describe_schedule()
execution_summary = schedule_desc.get("LastMonitoringExecutionSummary")
if execution_summary and execution_summary["MonitoringExecutionStatus"] in ["Completed", "CompletedWithViolations"]:
    last_model_explainability_monitor_execution = model_explainability_monitor.list_executions()[-1]
    last_model_explainability_monitor_execution_report_uri = last_model_explainability_monitor_execution.output.destination
    print(f'Report URI: {last_model_explainability_monitor_execution_report_uri}')
    last_model_explainability_monitor_execution_report_files = sorted(S3Downloader.list(last_model_explainability_monitor_execution_report_uri))
    print("Found Report Files:")
    print("\n ".join(last_model_explainability_monitor_execution_report_files))
else:
    last_model_explainability_monitor_execution = None
    print("====STOP==== \n No completed executions to inspect further. Please wait till an execution completes or investigate previously reported failures.")

If there are any violations compared to the baseline, they are listed here:

if last_model_explainability_monitor_execution:
    model_explainability_violations = last_model_explainability_monitor_execution.constraint_violations()
    if model_explainability_violations:
        print(model_explainability_violations.body_dict)

If your model is deployed to a real-time endpoint, you can see visualizations in SageMaker Studio of the analysis results and CloudWatch metrics by choosing the Endpoints tab, and then double-clicking the endpoint.