Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 54 additions & 9 deletions sources/platform/integrations/data-and-storage/snowflake.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,11 +63,11 @@ After the External Access Integration is in place, grant the app the remaining p

### Step 4: Verify the connection

![Apify Snowflake Integration home page](../images/snowflake/snowflake-home.png)
![Apify Snowflake Integration home page](../images/snowflake/snowflake-home.webp)

Once installation is complete, open the app and go to the **API Token** page to verify the connection is working. If the token secret was configured correctly during setup, the page shows your Apify username and email.

![API Token page showing a connected Apify account](../images/snowflake/snowflake-api-token.png)
![API Token page showing a connected Apify account](../images/snowflake/snowflake-api-token.webp)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new screenshots no longer match the text around them. For example, this page no longer shows the username and email, Next: Run Actor and Fetch Data are now Run Actor and Fetch data, and the options table lacks Flatten. Was reverting the text updates from 39a5d5e intentional?


To update your Apify API token later, go to **Configurations** > **Credentials** in App settings, find the secret bound to this app, and update its value.

Expand All @@ -89,7 +89,7 @@ Choose an Actor from two sources:

You can also type a custom Actor ID (for example, `E2jjCZBezvAZnX8Rb`) directly into the input.

![Run Actor page with Actor source and selection dropdown](../images/snowflake/snowflake-actor-select.png)
![Run Actor page with Actor source and selection dropdown](../images/snowflake/snowflake-actor-select.webp)

### Step 2: Configure and run

Expand All @@ -102,21 +102,21 @@ Expand **Advanced Run Options** to set:

Click **Next: Run Actor** to start the run.

![Actor Input JSON form with Advanced Run Options](../images/snowflake/snowflake-actor-input.png)
![Actor Input JSON form with Advanced Run Options](../images/snowflake/snowflake-actor-input.webp)

### Step 3: Monitor the run

The app polls the run status every five seconds and shows live updates. You can also follow the link to **View run in Apify Console** for full logs.

![Actor run in progress with live status updates](../images/snowflake/snowflake-actor-run-status.png)
![Actor run in progress with live status updates](../images/snowflake/snowflake-actor-run-status.webp)

Use the **Abort Actor run** button to stop the run at any time.

### Step 4: Import results

When the run completes, click **Next: Import dataset** to go straight to the **Import Data from Dataset** page with the run's default dataset pre-filled. Follow the [import steps](#import-a-dataset) to load the results into Snowflake.

![Completed Actor run with Next: Import dataset button](../images/snowflake/snowflake-actor-run-complete.png)
![Completed Actor run with Import dataset button](../images/snowflake/snowflake-actor-run-complete.webp)

## Import a dataset

Expand All @@ -126,7 +126,7 @@ The **Import Data from Dataset** page lets you fetch items from an Apify dataset

Choose one of your named datasets from the dropdown or enter a custom dataset ID. You can find dataset IDs in [Apify Console](https://console.apify.com/storage/datasets) under **Storage** > **Datasets**.

![Import Dataset page with dataset selector and Advanced options](../images/snowflake/snowflake-import-dataset.png)
![Import Dataset page with dataset selector and Advanced options](../images/snowflake/snowflake-import-dataset.webp)

Expand **Advanced options** to refine what data is fetched:

Expand All @@ -146,13 +146,13 @@ Click **Fetch Data** to retrieve a preview of the dataset.

After fetching data, select how you want to load it into Snowflake. You can either [create a new table](#create-new-table) or [insert into an existing table](#insert-into-existing-table).

![Dataset items preview and export mode selector](../images/snowflake/snowflake-dataset-preview.png)
![Dataset items preview and export mode selector](../images/snowflake/snowflake-dataset-preview.webp)

#### Create new table

The app creates a new table in the `APIFY.DATASETS_EXPORTS` schema in a dedicated database. Select which columns to include, then click **Next: Export**.

![Export settings with column selection](../images/snowflake/snowflake-export-settings.png)
![Export settings with column selection](../images/snowflake/snowflake-export-settings.webp)

Tables created by the app are only visible to `ACCOUNTADMIN` by default. To grant access to other roles, see [Manage table visibility](#manage-table-visibility).

Expand All @@ -172,6 +172,51 @@ Alternatively, open the app in Snowsight, go to the **Access management** tab, a

![Access management tab for granting app_public role](../images/snowflake/snowflake-access-management.png)

## Schedule recurring imports

The **Schedules** page creates recurring Actor or task runs whose results are imported into Snowflake automatically. Each schedule lives in your Apify account: on every run, Apify starts the Actor and, when the run succeeds, imports its dataset into your Snowflake table. Schedules keep running when the app is closed, until you pause or delete them.

![Schedules page with the list of schedules and the Create new schedule button](../images/snowflake/snowflake-schedules.webp)

### Before you start

Scheduled imports run outside Snowflake and connect back to your account to write the data, so each schedule needs:

- A Snowflake [programmatic access token (PAT)](https://docs.snowflake.com/en/user-guide/programmatic-access-tokens) for a user whose role can write to the destination table. Snowflake only accepts a PAT from a user who is subject to a [network policy](https://docs.snowflake.com/en/user-guide/network-policies). Apify Actor runs don't connect from a fixed range of IP addresses, so the policy must allow connections from them.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What should readers actually put in the network policy: allow all IP addresses, or something narrower? Right now they can't tell what to configure, and it affects security.

- For imports that create a new table, the `app_public` role granted to that user's role, as described in [Manage table visibility](#manage-table-visibility). The app checks for this grant and won't create the schedule without it.

### Create a schedule

On the **Schedules** page, select **Create new schedule** and follow the four steps:

1. **Schedule setup** - enter a **Schedule name** and pick a **Frequency**: every few minutes or hours, daily, weekly, monthly, yearly, or a custom **Cron** expression. The app previews the next five runs. Schedules run in your browser's time zone.

![Schedule setup step with the frequency tabs and the next runs preview](../images/snowflake/snowflake-schedule-setup.webp)

1. **Actor or task setup** - pick **Choose an Actor** and enter its input, or pick **Choose a task** and optionally enter input overrides. Select **Continue** to start a preview run. The schedule repeats this run, so its dataset shows you the data each scheduled run produces.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This page doesn't explain tasks anywhere since the "Run a task" section was dropped. Should that section come back, or should this step link to tasks?

1. **Import dataset** - when the preview run finishes, select **Continue** and configure the import as described in [Import a dataset](#import-a-dataset). The dataset is locked to the preview run's dataset, since every scheduled run imports the dataset it produces itself. For the same reason, **Unwind** and **Flatten** aren't available, and **Offset** and **Limit** apply to the preview only.
1. **Review** - check the schedule and its destination table. Then fill in the Snowflake connection details: **Snowflake account identifier**, **Snowflake username**, **Snowflake programmatic access token (PAT)**, and optionally **Snowflake warehouse**. For the username, enter your login name, which you find in Snowsight under your name > **Connect a tool to Snowflake** > **Login name**. Select **Save and start**.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this field take the login name or the user name? The app's helper query on the Review step fills it with CURRENT_USER(), which returns the user name, and the two can differ.


![Review step with the Snowflake destination credentials and the Save and start button](../images/snowflake/snowflake-schedule-review.webp)

The Snowflake credentials are stored only in the schedule's configuration in your Apify account.
Comment on lines +198 to +202

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Readers need to know where the credentials are stored before they enter the PAT, so move that sentence into the Review step.

Suggested change
1. **Review** - check the schedule and its destination table. Then fill in the Snowflake connection details: **Snowflake account identifier**, **Snowflake username**, **Snowflake programmatic access token (PAT)**, and optionally **Snowflake warehouse**. For the username, enter your login name, which you find in Snowsight under your name > **Connect a tool to Snowflake** > **Login name**. Select **Save and start**.
![Review step with the Snowflake destination credentials and the Save and start button](../images/snowflake/snowflake-schedule-review.webp)
The Snowflake credentials are stored only in the schedule's configuration in your Apify account.
1. **Review** - check the schedule and its destination table. Then fill in the Snowflake connection details: **Snowflake account identifier**, **Snowflake username**, **Snowflake programmatic access token (PAT)**, and optionally **Snowflake warehouse**. For the username, enter your login name, which you find in Snowsight under your name > **Connect a tool to Snowflake** > **Login name**. The credentials are stored only in the schedule's configuration in your Apify account. Select **Save and start**.
![Review step with the Snowflake destination credentials and the Save and start button](../images/snowflake/snowflake-schedule-review.webp)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The app stores the PAT in the schedule's Apify task configuration, so can anyone with access to the Apify account, like organization members, read it there? If so, should the page say that, so users scope the PAT's role accordingly?


### Schedule an existing import

After a successful import on the **Import Dataset** page, select **Schedule export** to repeat the Actor run that produced the dataset, with the same input, run options, and destination table. You only name the schedule and set its frequency before the review. This works for a run's default dataset, and not for imports fetched with **Unwind** or **Flatten**.

### Manage schedules

Select a schedule in the list to open its detail page, where you can:

- **Run now** - start a run immediately, outside the schedule.
- **Pause** - stop the schedule's runs. A paused schedule shows **Resume** instead, which restarts them.
- **Delete** - remove the schedule and everything it created in your Apify account.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What exactly does Delete remove, and does it touch the Snowflake table? If it only removes the Apify schedule and task, something like this would be clearer:

Suggested change
- **Delete** - remove the schedule and everything it created in your Apify account.
- **Delete** - remove the schedule and the Apify schedule and task it created. Data already imported into Snowflake stays in your tables.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A schedule is made of an Apify schedule, task, and webhook, so the suggestion should cover the webhook too:

Suggested change
- **Delete** - remove the schedule and everything it created in your Apify account.
- **Delete** - remove the schedule and the Apify schedule, task, and webhook it created. Data already imported into Snowflake stays in your tables.

- **Edit timing** - change the frequency.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens to a schedule when the PAT expires, and how do users update the stored credentials? Edit timing only changes the frequency.

- **Run history** - check each run's status, start time, and the number of rows it imported.

![Schedule detail page with the Run now, Pause, and Delete actions and the run history](../images/snowflake/snowflake-schedule-detail.webp)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this be reshot with realistic data? "PR test schedule" with 1 row per run looks like a test, and it highlights View in Apify Console, which the text never mentions.


## Next steps

With Apify data in Snowflake, you can query it with standard SQL, join it with your existing tables, and connect it to business intelligence (BI) tools like Tableau, Looker, or Snowflake's own data-sharing features.
Expand Down
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading