Reading from Twilio entities - Amazon Glue
Partitioning queries
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Reading from Twilio entities

Prerequisite

A Twilio object you would like to read from. You will need the object name such as SMS-Message or SMS-CountryPricing.

Supported entities for source:

Entity Interface Can be filtered Supports limit Supports Order by Supports Select * Supports partitioning
SMS-Message REST Yes Yes No Yes Yes
SMS-CountryPricing REST No No No Yes No
Voice-Call REST Yes Yes No Yes No
Voice-Application REST Yes Yes No Yes No
Voice-OutgoingCallerID REST Yes Yes No Yes No
Voice-Queue REST Yes Yes No Yes No
Conversations-Conversation REST Yes Yes No Yes No
Conversations-User REST No Yes No Yes No
Conversations-Role REST No Yes No Yes No
Conversations-Configuration REST No No No Yes No
Conversations-AddressConfiguration REST Yes Yes No Yes No
Conversations-WebhookConfiguration REST No No No Yes No
Conversations-ParticipantConversation REST No No No Yes No
Conversations-Credential REST No Yes No Yes No
Conversations-ConversationService REST No Yes No Yes No

Example:

twilio_read = glueContext.create_dynamic_frame.from_options(
    connection_type="twilio",
    connection_options={
        "connectionName": "connectionName",
        "ENTITY_NAME": "sms-message",
        "API_VERSION": "2010-04-01",
        "Edge_Location": "sydney.us1"
    }

Twilio entity and field details:

Entity Field Data type Supported operators
SMS-Message account_sid String N/A
api_version String N/A
body String N/A
date_created Datetime N/A
date_sent Datetime >=, <=, =
date_updated Datetime N/A
direction String N/A
error_code Integer N/A
error_message String N/A
from Integer =
messaging_service_sid String N/A
num_media String N/A
num_segments String N/A
price String N/A
price_unit Struct N/A
sid Integer N/A
status String N/A
subresource_uris Map N/A
to Integer =
uri Datetime N/A
SMS-CountryPricing country String N/A
iso_country String N/A
url String N/A
outbound_sms_prices List N/A
inbound_sms_prices List N/A
price_unit String N/A
Voice-Call account_sid String N/A
annotation String N/A
answered_by String N/A
api_version String N/A
caller_name String N/A
date_created Datetime N/A
date_updated Datetime N/A
direction String N/A
duration String N/A
end_time Datetime >=, <=, =
forwarded_from String N/A
from String =
from_formatted String N/A
group_sid String N/A
parent_call_sid String N/A
phone_number_sid String N/A
price String N/A
price_unit String N/A
sid String N/A
start_time Datetime >=, <=, =
status String =
subresource_uris String N/A
to String =
to_formatted String N/A
trunk_sid String N/A
uri String N/A
queue_time String N/A
Voice-Application account_sid String N/A
api_version String N/A
date_created Datetime N/A
date_updated Datetime N/A
friendly_name String =
message_status_callback String N/A
sid String N/A
sms_fallback_method String N/A
sms_fallback_url String N/A
sms_method String N/A
sms_status_callback String N/A
sms_url String N/A
status_callback String N/A
status_callback_method String N/A
uri String N/A
voice_caller_id_lookup Boolean N/A
voice_fallback_method String N/A
voice_fallback_url String N/A
voice_method String N/A
voice_url String N/A
public_application_connect_enabled Boolean N/A
Voice-OutgoingCallerID sid String N/A
date_created Datetime N/A
date_updated Datetime N/A
account_sid String N/A
friendly_name String =
phone_number String =
uri String N/A
Voice-Queue date_created Datetime N/A
date_updated Datetime N/A
current_size Integer N/A
friendly_name String N/A
uri String N/A
account_sid String N/A
average_wait_time Integer N/A
sid String N/A
max_size Integer N/A
Conversations-Conversation account_sid String N/A
chat_service_sid String N/A
messaging_service_sid String N/A
sid String N/A
friendly_name String N/A
unique_name String N/A
attributes String N/A
state String =
date_created Datetime N/A
date_updated Datetime N/A
timers Struct N/A
url String N/A
links Struct N/A
bindings Struct N/A
start_date Datetime =
end_date Datetime =
Timers.DateInactive String N/A
Timers.DateClosed String N/A
Conversations-User sid String N/A
account_sid String N/A
chat_service_sid String N/A
role_sid String N/A
identity String N/A
friendly_name String N/A
attributes String N/A
is_online Boolean N/A
is_notifiable Boolean N/A
date_created Datetime N/A
date_updated Datetime N/A
url String N/A
links Struct N/A
Conversations-Role sid String N/A
account_sid String N/A
chat_service_sid String N/A
friendly_name String N/A
type String N/A
permissions String N/A
date_created Datetime N/A
date_updated Datetime N/A
url String N/A
Conversations-Configuration account_sid Long N/A
default_chat_service_sid String N/A
default_messaging_service_sid String N/A
default_inactive_timer String N/A
default_closed_timer String N/A
url String N/A
links Map N/A
Conversations-AddressConfiguration sid String N/A
account_sid String N/A
type String N/A
address String N/A
friendly_name String N/A
auto_creation Struct N/A
date_created Datetime N/A
date_updated Datetime N/A
url String N/A
address_country String N/A
AutoCreation.Enabled Boolean N/A
AutoCreation.Type String N/A
AutoCreation.ConversationServiceSid String N/A
AutoCreation.WebhookUrl String N/A
AutoCreation.WebhookMethod String N/A
AutoCreation.WebhookFilters List N/A
AutoCreation.StudioFlowSid String N/A
AutoCreation.StudioRetryCount Integer N/A
Conversations-WebhookConfiguration account_sid String N/A
method String N/A
filters List N/A
pre_webhook_url String N/A
post_webhook_url String N/A
target String N/A
url String N/A
Converations-ParticipantConversation account_sid String N/A
chat_service_sid String N/A
participant_sid String N/A
participant_user_sid String N/A
participant_identity String N/A
participant_messaging_binding Struct N/A
Conversation_sid String N/A
conversation_unique_name String N/A
conversation_friendly_name String N/A
conversation_attributes String N/A
conversation_date_created Datetime N/A
conversation_date_updated Datetime N/A
conversation_created_by String N/A
conversation_state String N/A
conversation_timers Struct N/A
links Map N/A
address String =
identity String =
Conversation-Credentials sid String N/A
account_sid String N/A
friendly_name String N/A
type String N/A
sandbox String N/A
date_created Datetime N/A
dated_updated Datetime N/A
url String N/A
certificate String N/A
private_key String N/A
api_key String N/A
secret String N/A
Conversations-ConversationService sid String N/A
account_sid String N/A
friendly_name String N/A
date_created Datetime N/A
date_updated Datetime N/A
url String N/A
links Map N/A

Partitioning queries

Fields supporting partitioning:

In Twilio, the DateTime datatype fields support field-based partitioning.

You can provide the additional Spark options PARTITION_FIELD, LOWER_BOUND, UPPER_BOUND, and NUM_PARTITIONS if you want to utilize concurrency in Spark. With these parameters, the original query would be split into NUM_PARTITIONS number of sub-queries that can be executed by Spark tasks concurrently.

  • PARTITION_FIELD: the name of the field to be used to partition the query.

  • LOWER_BOUND: an inclusive lower bound value of the chosen partition field.

    For the Datetime field, we accept the Spark timestamp format used in Spark SQL queries.

    Examples of valid value:

    "2024-05-01T20:55:02.000Z"

  • UPPER_BOUND: an exclusive upper bound value of the chosen partition field.

  • NUM_PARTITIONS: the number of partitions.

Example:

twilio_read = glueContext.create_dynamic_frame.from_options(
    connection_type="twilio",
    connection_options={
        "connectionName": "connectionName",
        "ENTITY_NAME": "sms-message",
        "API_VERSION": "2010-04-01",
        "PARTITION_FIELD": "date_sent"
        "LOWER_BOUND": "2024-05-01T20:55:02.000Z"
        "UPPER_BOUND": "2024-06-01T20:55:02.000Z"
        "NUM_PARTITIONS": "10"
    }