Manage CloudStream data sources

Supported vendors

The configuration for cloud data sources is nearly the same for every vendor. For vendor-specific configuration details, see the following:

• Databricks
• Snowflake

Create a cloud data source

To create a cloud data source for CloudStream, use the following steps:

1. Go to CloudStream > Data Sources.
2. Click + Add Data Source.
3. Under Categories, click Data Warehouse and select a vendor.
4. In the Name field, enter a unique name for the data source related to your use case.
5. Click Continue.

You can connect to up to 10 data sources in a profile.

Establish a connection

Before you configure the data to import, you must establish a connection to your cloud data source. A connection is the reusable configuration of your vendor credentials that connects Tealium to your cloud data source.

On the Connection Configuration screen, confirm the name of the data source, then select an existing connection from the list or create a connection by clicking the + icon.

Click Save to return to the Connection Configuration screen, then click Establish Connection.

After you successfully establish a connection, select a table from the Table Selection list.

For more information about connecting to your vendor, see:

• Databricks
• Snowflake

Enable processing

Turn on Enable Processing to begin importing data immediately after you save and publish your changes. Or, leave this setting off while you complete the configuration and turn it on later.

Frequency

Select how often you want to fetch data:

• Near real-time: The process runs every 2 seconds.
• Hourly: The process runs at the start of each hour.
• Daily: The process runs once a day at the time you select (UTC).
• Weekly: The process runs once a week on a specified day and time (UTC).

Configure the query

In the Query Mode and Configuration screen, select the appropriate query mode and optionally include a SQL WHERE clause to import only those records that match your custom condition.

Select a query mode

The query mode determines how to select new rows, modified rows, or both for import.

• If you select Timestamp + Incrementing (Recommended), you must select two columns, a timestamp column and a strictly incrementing column.
• If you select Timestamp or Incrementing, you must select a column to use to detect either new and modified rows or new rows only.
• If you select Full Resync, all rows are imported during each import cycle. This option is useful when data changes frequently or when you want to ensure that all records are up-to-date. Full resync query mode requires that you use Hourly or Daily frequency, and you must specify the Timestamp Column or Incrementing Column.

For more information, see Query modes.

Configure the query

1. In the Query > Select Columns section, select the columns to import. To change the table or view, click Previous and select a different table.

2. (Optional) To add a custom condition, include a SQL WHERE clause.

   The WHERE clause does not support subqueries from multiple tables. To import data from multiple tables, create a view and select the view in the data source configuration.

3. For full resync query mode, enter the Limit for the maximum number of records to import. Daily queries can import up to 1,000,000 records, and hourly queries can import up to 50,000 records.

4. Click Test Query to validate your SQL query and preview the results.

Map columns

On the column mapping page, you’ll see the columns you selected in the previous step. Each column is automatically assigned a data type and a suggested cloud attribute name, which you can change as needed.

Columns not mapped to a cloud attribute are ignored.

Summary

In this final step, view the summary, make any needed corrections, and then save and publish your profile. To edit your configuration, click Previous to return to the step where you want to make changes.

1. Confirm the cloud attribute mappings.
2. Click Finish to create the data source and exit the configuration screen. The new data source is listed in the Data Sources dashboard.
3. Click Save/Publish to save and publish your changes.

Duplicate a data source

To duplicate a data source:

1. Click the data source.
2. Click the Duplicate button.
3. Enter a name for the duplicated data source.

Processed rows and errors

To see import activity, navigate to Data Sources and expand the data source.

CloudStream can ingest up to 500 data records per second.

Statuses

After configuring a cloud data source, it may display one of the following statuses:

Status	Description
Failed	A connection error has occurred, such as an authentication failure, and imports are halted until the error is resolved. Row-level errors during an import do not trigger this status and are logged while the data source remains in Running.
Inactive	The data source was created but was never turned on or transitioned to any other status.
Initializing	The connector is starting for the first time or resuming from a Stopped state. This is a temporary state before transitioning to Running or Scheduled.
Running	The connector is actively querying and importing data.
Scheduled	The next import is scheduled to run. This state can follow Initializing or Running.
Stopped	The data source was previously enabled but is now turned off. No data imports occur until it is enabled.
Unassigned	The task is awaiting allocation to a cloud worker.