Version 5 (V5) of the Amazon Tools for PowerShell has been released!
To start using the new version of the tools, see the Amazon Tools for PowerShell User Guide (V5), especially the topic for Migrating to V5.
Pipelining, output, and iteration in the Amazon Tools for PowerShell
Pipelining
PowerShell encourages users to connect cmdlets into pipelines
PS >Get-EC2Instance | Stop-EC2Instance
Cmdlet output
To better support pipelining, some data from the responses of the Amazon SDK for .NET might be discarded by
default. The output from Amazon Tools for PowerShell cmdlets isn't reshaped to include the service response and result
instances as Note properties on the emitted collection object. Instead, for those calls
that emit a single collection as output, the collection is now enumerated to the PowerShell pipeline.
This means that the SDK response and result data cannot exist in the pipeline because there is no
containing collection object to which it can be attached.
Although most users probably won't need this data, it can be useful for diagnostic purposes because
you can see exactly what was sent to and received from the underlying Amazon service calls that were made
by the cmdlet. Starting with the Amazon Tools for PowerShell V4, cmdlets can use the -Select * parameter
and argument to return the entire service response.
Note
In versions of the Amazon Tools for PowerShell prior to V4, a session variable called $AWSHistory
was introduced that maintains a record of Amazon cmdlet invocations and the service responses that were
received for each invocation. In V4 of the Tools for PowerShell, this session variable was deprecated in favor of
the -Select * parameter and argument, which can be used to return the entire service
response. This parameter is described in this topic.
Note
This is prerelease documentation for a feature in preview release. It is subject to change.
The $AWSHistory variable will be removed in V5 of the Amazon Tools for PowerShell. For more
information, see the blog post Notice of upcoming major version 5 of Amazon Tools for PowerShell
To illustrate how all data from a response can be returned, consider the following examples.
The first example simply returns a list of Amazon S3 buckets. This is the default behavior.
PS >Get-S3BucketCreationDate BucketName ------------ ---------- 9/22/2023 10:54:35 PM amzn-s3-demo-bucket1 9/22/2023 11:04:37 AM amzn-s3-demo-bucket2 9/22/2023 12:54:34 PM amzn-s3-demo-bucket3
The second example returns an Amazon SDK for .NET response object. Because -Select * was specified,
the output includes the entire API response, which contains the collection of buckets in the
Buckets property. In this example, the Format-List cmdlet isn't strictly
necessary, but is present to ensure that all properties are displayed.
PS >Get-S3Bucket -Select * | Format-ListLoggedAt : 10/1/2023 9:45:52 AM Buckets : {amzn-s3-demo-bucket1, amzn-s3-demo-bucket2, amzn-s3-demo-bucket3} Owner : Amazon.S3.Model.Owner ContinuationToken : ResponseMetadata : Amazon.Runtime.ResponseMetadata ContentLength : 0 HttpStatusCode : OK
Iteration through paged data
The following sections describe various types of iteration that are possible.
Automatic iteration
For service APIs that impose a default maximum number of returned objects for a given call or that support pageable result sets, most cmdlets implement automatic iteration, which enables the default behavior of "page-to-completion". In this scenario, a cmdlet makes as many calls as necessary on your behalf to return the complete data set to the pipeline.
In the following example, which uses the Get-S3Object cmdlet, the
$result variable contains S3Object instances for every key in a bucket
called amzn-s3-demo-bucket1, which is potentially a very large data set.
PS >$result = Get-S3Object -BucketName amzn-s3-demo-bucket1
The following example reduces the number of results for each page during automatic iteration from the default value of 1000 to 500. The example performs twice as many automatic iteration calls because only half as many results are returned for each call.
PS >$result = Get-S3Object -BucketName amzn-s3-demo-bucket1 -MaxKey 500
Note
In the Amazon Tools for PowerShell V4, some cmdlets for paged operations don't implement automatic iteration.
If a cmdlet doesn't have the -NoAutoIteration parameter, which is discussed in the next
section, then it doesn't implement automatic iteration.
Disable automatic iteration
If you want the Tools for PowerShell to return only the first page of data, you can add the
-NoAutoIteration parameter to prevent additional pages of data from being
returned.
The following example uses the -NoAutoIteration and -MaxKey parameters
to limit the number of returned S3Object instances to no more than the first 500 found in
the bucket.
PS >$result = Get-S3Object -BucketName amzn-s3-demo-bucket1 -MaxKey 500 -NoAutoIteration
To determine if more data was available but not returned, use the -Select * parameter
and argument and check if there is a value in the next token property.
The following example returns $true if there are more than 500 objects in the bucket
and $false otherwise.
PS >$result = Get-S3Object -BucketName amzn-s3-demo-bucket1 -MaxKey 500 -NoAutoIteration -Select *PS >$null -eq $result.NextMarker
Note
The names of the next token response property and cmdlet parameter vary between cmdlets. For details, refer to the help documentation for each cmdlet.
Manual iteration
The following example returns all S3 objects from a bucket using a dodo loop
performs iterations until Get-S3Object sets $result.NextMarker to
$null, indicating that no more paged data remains. The output of the loop is assigned
to the $s3Objects variable.
$s3Objects = do { $splatParams = @{ BucketName = 'amzn-s3-demo-bucket1' MaxKey = 500 Marker = $result.NextMarker NoAutoIteration = $true Select = '*' } $result = Get-S3Object @splatParams $result.S3Objects } while ($null -ne $result.NextMarker)
This example uses PowerShell splatting