教程:使用示例书店应用程序进行 A/B 测试 - Amazon CloudWatch
Amazon Web Services 文档中描述的 Amazon Web Services 服务或功能可能因区域而异。要查看适用于中国区域的差异,请参阅中国的 Amazon Web Services 服务入门

教程:使用示例书店应用程序进行 A/B 测试

本节介绍如何使用 Amazon CloudWatch Evidently 进行 A/B 测试的教程。本教程使用 Amazon 书店演示应用程序。书店演示应用程序是创建店面和后端的示例全堆栈 Web 应用程序。有关此演示应用程序的更多信息,请参阅 Amazon 书店演示应用程序

步骤 1:启动书店演示应用程序

首先启动书店演示应用程序。

要使用 Amazon CloudFormation 启动书店演示应用程序

  1. 使用以下链接打开 Amazon CloudFormation 控制台并加载书店演示应用程序的模板。

    https://console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/new?stackName=MyBookstore&templateURL=https://bookstoretemplate.s3.amazonaws.com/bookstoreTemplate.yaml
  2. 选择 Next (下一步)

  3. 确认 Stack name(堆栈名称)设置为 MyBookstore 并且 ProjectName 设置为 mybookstore,然后选择 Next(下一步)。

  4. 对于配置堆栈选项,请选择下一步

  5. 在下一页底部,选择 I acknowledge that Amazon CloudFormation might create IAM resources with custom names.(我确认 Amazon CloudFormation 可能创建具有自定义名称的 IAM 资源。),然后选择 Create stack(创建堆栈)。

    完成 Amazon CloudFormation 堆栈部署大约需要 30 分钟。

  6. 完成部署后,您可以访问书店应用程序。

    要远程访问书店应用程序,请登录 Amazon CloudFormation 控制台,然后选择刚刚部署的堆栈。然后选择 Outputs(输出)选项卡,接着选择 WebApplication 旁边的链接。

    要在本地访问书店应用程序,请登录 CodeCommit 控制台。然后在 Repositories(存储库)下选择应用程序的名称,接着选择 Clone URL(克隆 URL)。借助 HTTPS、SSH 或 HTTPS (GRC),使用 git clone 拉取代码。以下示例使用 HTTPS 链接。

    git clone https://git-codecommit.us-west-2.amazonaws.com/v1/repos/mybookstore-WebAssets

    您可以在 IAM 控制台中为 CodeCommit 生成 HTTPS Git 凭证。为此,请选择用户,然后选择 Security credentials(安全凭证)选项卡。

  7. 打开书店应用程序后,请选择 Sign up(注册)。请使用有效的电子邮件地址。您将收到电子邮件中的确认码,然后请输入该代码以完成注册过程。然后登录该账户。

步骤 2:添加 Evidently 端点和未经身份验证的角色

如果您想在自己的网站应用程序上测试 Evidently,则需要执行此步骤。

要配置 Evidently 端点和未经身份验证的角色

  1. 将 Evidently 区域和端点同时添加到书店应用程序包中 src/ 目录中的 config.jsconfig.ts 文件:

    evidently: { // Region REGION: "us-west-2", // Evidently endpoint ENDPOINT: "https://evidently.us-west-2.amazonaws.com", }
  2. 将未经身份验证的角色同时添加到带有前缀的 config.jsconfig.ts,如以下示例所示:

    arn:aws:iam:: //EXAMPLE UN_AUTH_ROLE: "arn:aws:iam::1234567891011:role/MyBookstore-CognitoUnAuthorizedRole-10PGTHJ0DZ3D2"

    config.jsconfig.ts 文件现在应该是这样。

    export default { // Default setting MAX_ATTACHMENT_SIZE: 5000000, // Default setting apiGateway: { REGION: "us-west-2", API_URL: "https://hdk8s9ev64.execute-api.us-west-2.amazonaws.com/prod", }, cognito: { // Default setting no need to change REGION: "us-west-2", // Default setting no need to change USER_POOL_ID: "us-west-2_HcWLl7xMr", // Default setting no need to change APP_CLIENT_ID: "71jti5mq3j65p0ci1lfturefno", // Default setting no need to change IDENTITY_POOL_ID: "us-west-2:0f283cf1-43ac-4410-b310-0c531ac6ce9e", ** *//* Newly added *Unauthenticated role* for bookstore app from Cognito identity pool UN_AUTH_ROLE: "arn:aws:iam::750664202209:role/MyBookstore-CognitoUnAuthorizedRole-10PGTHJ0DZ3D2" }, ** *// Evidently endpoint* evidently: { REGION: "us-west-2", ENDPOINT: "https://evidently.us-west-2.amazonaws.com", } };
  3. 向书店未经身份验证的角色添加调用 Evidently API 的权限:

    • 通过以下网址打开 IAM 控制台:https://console.aws.amazon.com/iam/

    • 在导航窗格中,选择 Roles(角色)

    • 查找书店的未经身份验证的角色并将以下策略附加到该角色:

      { "Version": "2012-10-17", "Statement": [ { "Action": [ "evidently:*" ], "Resource": "*", "Effect": "Allow" } ] }

步骤 3:创建项目、功能和实验

然后,您可以在 CloudWatch Evidently 控制台中创建项目、功能和实验。

要为本教程创建项目、功能和实验

  1. 访问 https://console.aws.amazon.com/cloudwatch/,打开 CloudWatch 控制台。

  2. 创建项目。

  3. 在导航窗格中,依次选择 Application monitoring(监控应用程序)、Evidently

  4. 请选择 Create project(创建项目)并填写字段。将项目名称命名为 BookstoreProject。有关更多信息,请参阅 创建新项目

  5. 创建项目后,在该项目中创建功能。将功能命名为 showDiscount。在此功能中,创建 Boolean 类型的两个变体。使用值 Variation1-False 将第一个变体命名为 False,然后使用值 Variation2-True 将第二个变体命名为 True

    有关创建功能的更多信息,请参阅 向项目添加功能

  6. 完成功能创建后,请在项目中创建实验。将实验命名为 pageLoadTime

    本实验将使用名为 pageLoadTime 的自定义指标,用以衡量被测试页面的页面加载时间。用于实验的自定义指标使用 Amazon EventBridge 创建而成。有关 EventBridge 的更多信息,请参阅什么是 Amazon EventBridge?

    要创建该自定义指标,请在创建实验时执行以下操作:

    • 对于 Metric source(指标来源),请在 Metrics(指标)下选择 Custom metrics(自定义指标)。

    • 对于 Metric name(指标名称),请输入 pageLoadTime

    • 对于 Goal(目标),请选择 Decrease(减少)。这表示我们希望该指标的值较低,以指明该功能变体最佳。

    • 对于 Metric rule(指标规则),请输入以下内容:

      • 对于 Entity ID (实体 ID),请输入 UserDetails.userId

      • 对于 Value key(值键),请输入 details.pageLoadTime

      • 对于 Units(单位)中,请输入 ms

    • 请选择 Add metric(添加指标)。

    对于 Audiences(受众),请选择 100% 以便所有用户都能进入实验。将变体之间的流量拆分设置为每个变体 50%。

    然后,请选择 Create experiment(创建实验)来创建实验。创建它后,只有您告知 Evidently 将其启动,它才会启动。

步骤 4:为功能评估设置代码

当您使用 CloudWatch Evidently 评估功能时,必须使用 EvaluateFeature 操作为每个用户会话随机选择功能变体。此操作根据您在实验中指定的百分比,将用户会话分配给功能的每个变体。

要为书店演示应用程序设置功能评估代码

  1. 使用 npm(Node.js 包管理器)安装适用于 JavaScript 的 Amazon 开发工具包:

    npm install aws-sdk

    您可以复制以下脚本来替换 package.json 文件并运行 npm install -f

    { "name": "aws-bookstore-demo-app", "version": "0.1.0", "private": true, "dependencies": { "@aws-sdk/credential-provider-cognito-identity": "3.30.0", "@awsui/components-react": "^3.0.148", "@types/graphql": "^14.0.5", "@types/jest": "^23.3.10", "@types/node": "^10.12.12", "@types/react": "^16.7.12", "@types/react-bootstrap": "^0.32.15", "@types/react-dom": "^16.0.11", "@types/react-router-bootstrap": "^0.24.5", "@types/react-router-dom": "^4.3.1", "aws-amplify": "^1.0.10", "aws-sdk": "^2.1033.0", "bootstrap": "^3.3.7", "react": "^16.5.0", "react-bootstrap": "^0.32.4", "react-dom": "^16.5.0", "react-router-bootstrap": "^0.24.4", "react-router-dom": "^4.3.1", "react-scripts": "^3.0.1", "typescript": "^4.2.4" }, "scripts": { "start": "react-scripts start", "build": "react-scripts build", "test": "react-scripts test --env=jsdom", "eject": "react-scripts eject" }, "devDependencies": { "@aws-sdk/types": "3.1.0" }, "browserslist": [ ">0.2%", "not dead", "not ie <= 11", "not op_mini all" ] }
  2. 要设置凭证以调用 Amazon 开发工具包,请将以下代码添加到 Home.tsx 文件中。

    import Evidently from 'aws-sdk/clients/evidently'; import { STS, AssumeRoleWithWebIdentityCommand } from '@aws-sdk/client-sts'; import { PutProjectEventsRequest , EvaluateFeatureRequest} from 'aws-sdk/clients/evidently'; import { CredentialProvider, Credentials } from '@aws-sdk/types'; import { CognitoIdentityClient, GetOpenIdTokenCommand, GetIdCommand, } from '@aws-sdk/client-cognito-identity'; export type ClientBuilder = ( endpoint: string, region: string, credentials: CredentialProvider ) => Promise<Evidently>; interface HomeState { isLoading: boolean; startTime?: any; id?: string; showDiscount: boolean }
  3. 通过添加这些变量名称来修改构造函数,以对主组件进行修改。

    private client!: Evidently; private stsClient: STS; private cognitoIdentityClient: CognitoIdentityClient; constructor(props: HomeProps) { super(props); this.state = { isLoading: true, id: "", showDiscount: false, }; this.stsClient = new STS({ region: config.cognito.REGION }); this.cognitoIdentityClient = new CognitoIdentityClient({ region: config.cognito.REGION, }); }
  4. 通过在 async componentDidMount() 之前立即添加以下代码来设置凭证:

    private defaultClientBuilder: ClientBuilder = async ( endpoint: string, region: string, credentialProvider: CredentialProvider ) => { let credentials: Credentials = await credentialProvider(); return new Evidently({ endpoint, region, credentials, }); }; //Get credentials from *Unauthenticated role* //You need to add *@aws-sdk/credential-provider-cognito-identity* & *@aws-sdk/client-sts* in your package.json private cognitoCredentialsProvider = async (): Promise<Credentials> => { return this.cognitoIdentityClient .send( new GetIdCommand({ IdentityPoolId: config.cognito.IDENTITY_POOL_ID, }) ) .then((getIdResponse) => this.cognitoIdentityClient.send( new GetOpenIdTokenCommand({ IdentityId: getIdResponse.IdentityId, }) ) ) .then((getOpenIdTokenResponse) => this.stsClient.send( new AssumeRoleWithWebIdentityCommand({ RoleArn: config.cognito.UN_AUTH_ROLE, RoleSessionName: 'evidently', WebIdentityToken: getOpenIdTokenResponse.Token, }) ) ) .then((assumeRoleResponse) => { if (!assumeRoleResponse.Credentials) { throw new Error('STS response did not contain credentials.'); } const credentials = { accessKeyId: assumeRoleResponse.Credentials.AccessKeyId as string, secretAccessKey: assumeRoleResponse.Credentials .SecretAccessKey as string, sessionToken: assumeRoleResponse.Credentials.SessionToken, expiration: assumeRoleResponse.Credentials.Expiration, }; return credentials; }); };

完成上一过程后,Home.tsx 文件应类似于以下内容。

import React, { Component } from "react"; import screenshot from "../../images/screenshot.png"; import yourpastorders from "../../images/yourpastorders.png"; import bestSellers from "../../images/bestSellers.png"; import yourshoppingcart from "../../images/yourshoppingcart.png"; import { Hero } from "../../common/hero/Hero"; import { CategoryNavBar } from "../category/categoryNavBar/CategoryNavBar"; import { SearchBar } from "../search/searchBar/SearchBar"; import { BestSellersBar } from "../bestSellers/bestSellersBar/BestSellersBar"; import { CategoryGalleryTeaser } from "../category/CategoryGalleryTeaser"; import { FriendsBought } from "../friends/FriendsBought"; import { LinkContainer } from "react-router-bootstrap"; *import Evidently from 'aws-sdk/clients/evidently'; import { STS, AssumeRoleWithWebIdentityCommand } from '@aws-sdk/client-sts'; import { PutProjectEventsRequest , EvaluateFeatureRequest} from 'aws-sdk/clients/evidently'; import { CredentialProvider, Credentials } from '@aws-sdk/types'; * *import { CognitoIdentityClient, GetOpenIdTokenCommand, GetIdCommand, } from '@aws-sdk/client-cognito-identity'; import { Spinner } from '@awsui/components-react'; import "./home.css"; import config from '../../config'; export type ClientBuilder = ( endpoint: string, region: string, credentials: CredentialProvider ) => Promise<Evidently>; * interface HomeProps { isAuthenticated: boolean; } *interface HomeState { isLoading: boolean; startTime?: any; id?: string; showDiscount: boolean } * export default class Home extends Component<HomeProps, HomeState> { *private client!: Evidently; private stsClient: STS; private cognitoIdentityClient: CognitoIdentityClient;* *constructor(props: HomeProps) { super(props); this.state = { isLoading: true, id: "", showDiscount: false, }; this.stsClient = new STS({ region: config.cognito.REGION }); this.cognitoIdentityClient = new CognitoIdentityClient({ region: config.cognito.REGION, }); } * *private defaultClientBuilder: ClientBuilder = async ( endpoint: string, region: string, credentialProvider: CredentialProvider ) => { let credentials: Credentials = await credentialProvider(); return new Evidently({ endpoint, region, credentials, }); }; //Get credentials from *Unauthenticated role* //You need to add *@aws-sdk/credential-provider-cognito-identity* & *@aws-sdk/client-sts* in your package.json private cognitoCredentialsProvider = async (): Promise<Credentials> => { return this.cognitoIdentityClient .send( new GetIdCommand({ IdentityPoolId: config.cognito.IDENTITY_POOL_ID, }) ) .then((getIdResponse) => this.cognitoIdentityClient.send( new GetOpenIdTokenCommand({ IdentityId: getIdResponse.IdentityId, }) ) ) .then((getOpenIdTokenResponse) => this.stsClient.send( new AssumeRoleWithWebIdentityCommand({ RoleArn: config.cognito.UN_AUTH_ROLE, RoleSessionName: 'evidently', WebIdentityToken: getOpenIdTokenResponse.Token, }) ) ) .then((assumeRoleResponse) => { if (!assumeRoleResponse.Credentials) { throw new Error('STS response did not contain credentials.'); } const credentials = { accessKeyId: assumeRoleResponse.Credentials.AccessKeyId as string, secretAccessKey: assumeRoleResponse.Credentials .SecretAccessKey as string, sessionToken: assumeRoleResponse.Credentials.SessionToken, expiration: assumeRoleResponse.Credentials.Expiration, }; return credentials; * ** }); }; *async componentDidMount() { const randomizedID = new Date().getTime().toString(); this.setState({id: randomizedID}) if (!this.props.isAuthenticated) { return; } if (this.client == null) { this.client = await this.defaultClientBuilder( config.evidently.ENDPOINT, config.evidently.REGION, this.cognitoCredentialsProvider ); } this.setState({ isLoading: false }); const evaluateFeatureRequest: EvaluateFeatureRequest = { entityId: randomizedID, // Your feature name feature: 'showDiscount', // Your project name' project: 'BookstoreProject', }; this.client.evaluateFeature(evaluateFeatureRequest).promise().then(res => { console.log('evaluateFeature response: ', res); if(res.value?.boolValue !== undefined) { this.setState({showDiscount: res.value.boolValue}); } }) } * renderLanding() { return ( <div className="lander"> <h1>Bookstore Demo</h1> <hr /> <p>This is a sample application demonstrating how different types of databases and Amazon services can work together to deliver a delightful user experience. In this bookstore demo, users can browse and search for books, view recommendations, see the leaderboard, view past orders, and more. You can get this sample application up and running in your own environment and learn more about the architecture of the app by looking at the <a href="https://github.com/aws-samples/aws-bookstore-demo-app" target="_blank">github repository</a>.</p> <div className="button-container col-md-12"> <LinkContainer to="/signup"> <a href="/signup">Sign up to explore the demo</a> </LinkContainer> </div> <img src={screenshot} className="img-fluid full-width" alt="Screenshot"></img> <div className="product-section"> <h1>Databases on Amazon</h1> <hr /> <h2>Purpose-built databases for all your application needs</h2> <div className="col-md-12"> <p>As the cloud continues to drive down the cost of storage and compute, a new generation of applications have emerged, creating a new set of requirements for databases. These applications need databases to store terabytes to petabytes of new types of data, provide access to the data with millisecond latency, process millions of requests per second, and scale to support millions of users anywhere in the world. To support these requirements, you need both relational and non-relational databases that are purpose-built to handle the specific needs of your applications. Amazon offers the broadest range of databases purpose-built for your specific application use cases. </p> </div> </div> <div className="product-section"> <div className="col-md-12"> <hr /> <h3><a href="https://aws.amazon.com/dynamodb/">Amazon DynamoDB</a></h3> <h4>NoSQL Database</h4> <p><a href="https://aws.amazon.com/dynamodb/">Amazon DynamoDB</a> is a fast and flexible <a href="https://aws.amazon.com/nosql/">NoSQL database service</a> for all applications that need consistent, single-digit millisecond latency at any scale. It is a fully managed cloud database and supports both document and key-value store models. Its flexible data model and reliable performance make it a great fit for mobile, web, gaming, ad tech, IoT, and many other applications. New for DynamoDB is global tables, which fully automate the replication of tables across Amazon regions for a fully managed, multi-master, multi-region database. DynamoDB also adds support for on-demand and continuous backups for native data protection.</p> <p>For more information visit the <a href="https://aws.amazon.com/dynamodb/">Amazon DynamoDB product page.</a></p> </div> <div className="col-md-12"> <hr /> <h3><a href="https://aws.amazon.com/neptune/">Amazon Neptune</a></h3> <h4>Graph Database</h4> <p><a href="https://aws.amazon.com/neptune/">Amazon Neptune</a> is a fast, reliable, fully-managed <a href="https://aws.amazon.com/nosql/graph/">graph database</a> service that makes it easy to build and run applications that work with highly connected datasets. The core of Amazon Neptune is a purpose-built, high-performance graph database engine optimized for storing billions of relationships and querying the graph with milliseconds latency. Amazon Neptune supports popular graph models Apache TinkerPop and W3C's RDF, and their associated query languages TinkerPop Gremlin and RDF SPARQL, allowing you to easily build queries that efficiently navigate highly connected datasets. Neptune powers graph use cases such as recommendation engines, fraud detection, knowledge graphs, drug discovery, and network security.</p> <p>For more information visit the <a href="https://aws.amazon.com/neptune/">Amazon Neptune product page.</a></p> </div> <div className="col-md-12"> <hr /> <h3><a href="https://aws.amazon.com/opensearch-service/">Amazon OpenSearch Service</a></h3> <h4>In-Memory Data Store</h4> <p><a href="https://aws.amazon.com/opensearch-service/">Amazon OpenSearch Service</a> makes it easy to deploy, operate, and scale an <a href="https://aws.amazon.com/nosql/key-value/">in-memory data store</a> or cache in the cloud. The service improves the performance of web applications by allowing you to retrieve information from fast, managed, in-memory caches, instead of relying entirely on slower disk-based databases. </p> </div> <div className="col-md-12"> <hr /> <h3><a href="https://aws.amazon.com/opensearch-service/">Amazon OpenSearch Service</a></h3> <h4>Fully managed, scalable, and reliable OpenSearch service</h4> <p><a href="https://aws.amazon.com/opensearch-service/">Amazon Opensearch Service</a> is a fully managed service that makes it easy for you to deploy, secure, operate, and scale OpenSearch to search, analyze, and visualize data in real-time. </p> <p>For more information visit the <a href="https://aws.amazon.com/opensearch-service/">Amazon OpenSearch Service product page.</a></p> </div> </div> </div>); } renderHome() { return ( <div className="bookstore"> *{this.state.showDiscount && <div>Enjoy 25% off your order!</div>} * <Hero /> <SearchBar /> <CategoryNavBar /> <BestSellersBar /> <div className="well-bs col-md-12 ad-container-padding"> <div className="col-md-4 ad-padding"> <div className="container-category no-padding"> <LinkContainer to="/past"> <img src={yourpastorders} alt="Past orders"></img> </LinkContainer> </div> </div> <div className="col-md-4 ad-padding"> <div className="container-category no-padding"> <LinkContainer to="/cart"> <img src={yourshoppingcart} alt="Shopping cart"></img> </LinkContainer> </div> </div> <div className="col-md-4 ad-padding"> <div className="container-category no-padding"> <LinkContainer to="/best"> <img src={bestSellers} alt="Best sellers"></img> </LinkContainer> </div> </div> </div> <CategoryGalleryTeaser /> <FriendsBought /> </div> ); } render() { return ( <div className="Home"> {this.props.isAuthenticated ? this.renderHome() : this.renderLanding()} </div> ); } }

步骤 5:为实验指标设置代码

对于自定义指标,请使用 Evidently 的 PutProjectEvents API 向 Evidently 发送指标结果。以下示例说明了如何设置自定义指标并向 Evidently 发送实验数据。

async componentDidMount() 之前立即添加以下代码:

componentWillMount() { this.setState({startTime: new Date()}) }

然后,添加以下函数来计算页面加载时间,并使用 PutProjectEvents 向 Evidently 发送指标值。将以下代码放入 Home.tsx 并在 EvaluateFeature API 中调用此函数:

getPageLoadTime = () => { const timeSpent = (new Date().getTime() - this.state.startTime.getTime()) * 1.000001; const pageLoadTimeData = `{ "details": { "pageLoadTime": ${timeSpent} }, "userDetails": { "userId": "${this.state.id}", "sessionId": "${this.state.id}"} }`; const putProjectEventsRequest: PutProjectEventsRequest = { project: 'BookstoreProject', events: [ { timestamp: new Date(), type: 'aws.evidently.custom', data: JSON.parse(pageLoadTimeData) }, ], }; this.client.putProjectEvents(putProjectEventsRequest).promise(); }

现在,async componentDidMount() 中的评估功能 API 代码应类似于以下内容:

this.client.evaluateFeature(evaluateFeatureRequest).promise().then(res => { if(res.value?.boolValue !== undefined) { this.setState({showDiscount: res.value.boolValue}); } // Send custom metric to Evidently *this.getPageLoadTime()* })

每次用户访问书店应用程序时,都会向 Exlenly 发送自定义指标进行分析。Evidently 会分析每个指标并在 Evidently 控制面板上实时显示结果。以下示例显示了指标有效负载:

[ { "timestamp": 1637368646.468, "type": "aws.evidently.custom", "data": "{\"details\":{\"pageLoadTime\":2058.002058},\"userDetails\":{\"userId\":\"1637368644430\",\"sessionId\":\"1637368644430\"}}" } ]

步骤 6:开始实验并启动书店应用程序

最后一步是开始实验并启动书店应用程序。

要开始教程实验

  1. 访问 https://console.aws.amazon.com/cloudwatch/,打开 CloudWatch 控制台。

  2. 在导航窗格中,依次选择 Application monitoring(监控应用程序)、Evidently

  3. 请选择 BookstoreProject 项目。

  4. 请选择 Experiments(实验)选项卡。

  5. 请选择 pageLoadTime 旁边的按钮,然后依次选择 Actions(操作)、Start experiment(开始实验)。

  6. 请选择实验结束的时间。

  7. 请选择 Start experiment(开始实验)。

    实验会立即开始。

然后,使用以下命令启动书店应用程序:

npm install -f && npm start

应用程序启动后,您可以登录并创建书店账户。创建账户并登录后,您将被分配到正在测试的两个变体之一。一个变体会显示“此订单享受 25% 的折扣”,另一个无折扣。