Repository navigation
docs: refresh Snowflake integration screenshots and add schedules section #3103
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
39a5d5e
1c5608c
d2322d4
3ff4aca
bb9fae8
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -63,11 +63,11 @@ After the External Access Integration is in place, grant the app the remaining p | |||||||||||||||||
|
|
||||||||||||||||||
| ### Step 4: Verify the connection | ||||||||||||||||||
|
|
||||||||||||||||||
|  | ||||||||||||||||||
|  | ||||||||||||||||||
|
|
||||||||||||||||||
| Once installation is complete, open the app and go to the **API Token** page to verify the connection is working. If the token secret was configured correctly during setup, the page shows your Apify username and email. | ||||||||||||||||||
|
|
||||||||||||||||||
|  | ||||||||||||||||||
|  | ||||||||||||||||||
|
|
||||||||||||||||||
| To update your Apify API token later, go to **Configurations** > **Credentials** in App settings, find the secret bound to this app, and update its value. | ||||||||||||||||||
|
|
||||||||||||||||||
|
|
@@ -89,7 +89,7 @@ Choose an Actor from two sources: | |||||||||||||||||
|
|
||||||||||||||||||
| You can also type a custom Actor ID (for example, `E2jjCZBezvAZnX8Rb`) directly into the input. | ||||||||||||||||||
|
|
||||||||||||||||||
|  | ||||||||||||||||||
|  | ||||||||||||||||||
|
|
||||||||||||||||||
| ### Step 2: Configure and run | ||||||||||||||||||
|
|
||||||||||||||||||
|
|
@@ -102,21 +102,21 @@ Expand **Advanced Run Options** to set: | |||||||||||||||||
|
|
||||||||||||||||||
| Click **Next: Run Actor** to start the run. | ||||||||||||||||||
|
|
||||||||||||||||||
|  | ||||||||||||||||||
|  | ||||||||||||||||||
|
|
||||||||||||||||||
| ### Step 3: Monitor the run | ||||||||||||||||||
|
|
||||||||||||||||||
| The app polls the run status every five seconds and shows live updates. You can also follow the link to **View run in Apify Console** for full logs. | ||||||||||||||||||
|
|
||||||||||||||||||
|  | ||||||||||||||||||
|  | ||||||||||||||||||
|
|
||||||||||||||||||
| Use the **Abort Actor run** button to stop the run at any time. | ||||||||||||||||||
|
|
||||||||||||||||||
| ### Step 4: Import results | ||||||||||||||||||
|
|
||||||||||||||||||
| When the run completes, click **Next: Import dataset** to go straight to the **Import Data from Dataset** page with the run's default dataset pre-filled. Follow the [import steps](#import-a-dataset) to load the results into Snowflake. | ||||||||||||||||||
|
|
||||||||||||||||||
|  | ||||||||||||||||||
|  | ||||||||||||||||||
|
|
||||||||||||||||||
| ## Import a dataset | ||||||||||||||||||
|
|
||||||||||||||||||
|
|
@@ -126,7 +126,7 @@ The **Import Data from Dataset** page lets you fetch items from an Apify dataset | |||||||||||||||||
|
|
||||||||||||||||||
| Choose one of your named datasets from the dropdown or enter a custom dataset ID. You can find dataset IDs in [Apify Console](https://console.apify.com/storage/datasets) under **Storage** > **Datasets**. | ||||||||||||||||||
|
|
||||||||||||||||||
|  | ||||||||||||||||||
|  | ||||||||||||||||||
|
|
||||||||||||||||||
| Expand **Advanced options** to refine what data is fetched: | ||||||||||||||||||
|
|
||||||||||||||||||
|
|
@@ -146,13 +146,13 @@ Click **Fetch Data** to retrieve a preview of the dataset. | |||||||||||||||||
|
|
||||||||||||||||||
| After fetching data, select how you want to load it into Snowflake. You can either [create a new table](#create-new-table) or [insert into an existing table](#insert-into-existing-table). | ||||||||||||||||||
|
|
||||||||||||||||||
|  | ||||||||||||||||||
|  | ||||||||||||||||||
|
|
||||||||||||||||||
| #### Create new table | ||||||||||||||||||
|
|
||||||||||||||||||
| The app creates a new table in the `APIFY.DATASETS_EXPORTS` schema in a dedicated database. Select which columns to include, then click **Next: Export**. | ||||||||||||||||||
|
|
||||||||||||||||||
|  | ||||||||||||||||||
|  | ||||||||||||||||||
|
|
||||||||||||||||||
| Tables created by the app are only visible to `ACCOUNTADMIN` by default. To grant access to other roles, see [Manage table visibility](#manage-table-visibility). | ||||||||||||||||||
|
|
||||||||||||||||||
|
|
@@ -172,6 +172,51 @@ Alternatively, open the app in Snowsight, go to the **Access management** tab, a | |||||||||||||||||
|
|
||||||||||||||||||
|  | ||||||||||||||||||
|
|
||||||||||||||||||
| ## Schedule recurring imports | ||||||||||||||||||
|
|
||||||||||||||||||
| The **Schedules** page creates recurring Actor or task runs whose results are imported into Snowflake automatically. Each schedule lives in your Apify account: on every run, Apify starts the Actor and, when the run succeeds, imports its dataset into your Snowflake table. Schedules keep running when the app is closed, until you pause or delete them. | ||||||||||||||||||
|
|
||||||||||||||||||
|  | ||||||||||||||||||
|
|
||||||||||||||||||
| ### Before you start | ||||||||||||||||||
|
|
||||||||||||||||||
| Scheduled imports run outside Snowflake and connect back to your account to write the data, so each schedule needs: | ||||||||||||||||||
|
|
||||||||||||||||||
| - A Snowflake [programmatic access token (PAT)](https://docs.snowflake.com/en/user-guide/programmatic-access-tokens) for a user whose role can write to the destination table. Snowflake only accepts a PAT from a user who is subject to a [network policy](https://docs.snowflake.com/en/user-guide/network-policies). Apify Actor runs don't connect from a fixed range of IP addresses, so the policy must allow connections from them. | ||||||||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What should readers actually put in the network policy: allow all IP addresses, or something narrower? Right now they can't tell what to configure, and it affects security. |
||||||||||||||||||
| - For imports that create a new table, the `app_public` role granted to that user's role, as described in [Manage table visibility](#manage-table-visibility). The app checks for this grant and won't create the schedule without it. | ||||||||||||||||||
|
|
||||||||||||||||||
| ### Create a schedule | ||||||||||||||||||
|
|
||||||||||||||||||
| On the **Schedules** page, select **Create new schedule** and follow the four steps: | ||||||||||||||||||
|
|
||||||||||||||||||
| 1. **Schedule setup** - enter a **Schedule name** and pick a **Frequency**: every few minutes or hours, daily, weekly, monthly, yearly, or a custom **Cron** expression. The app previews the next five runs. Schedules run in your browser's time zone. | ||||||||||||||||||
|
|
||||||||||||||||||
|  | ||||||||||||||||||
|
|
||||||||||||||||||
| 1. **Actor or task setup** - pick **Choose an Actor** and enter its input, or pick **Choose a task** and optionally enter input overrides. Select **Continue** to start a preview run. The schedule repeats this run, so its dataset shows you the data each scheduled run produces. | ||||||||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This page doesn't explain tasks anywhere since the "Run a task" section was dropped. Should that section come back, or should this step link to tasks? |
||||||||||||||||||
| 1. **Import dataset** - when the preview run finishes, select **Continue** and configure the import as described in [Import a dataset](#import-a-dataset). The dataset is locked to the preview run's dataset, since every scheduled run imports the dataset it produces itself. For the same reason, **Unwind** and **Flatten** aren't available, and **Offset** and **Limit** apply to the preview only. | ||||||||||||||||||
| 1. **Review** - check the schedule and its destination table. Then fill in the Snowflake connection details: **Snowflake account identifier**, **Snowflake username**, **Snowflake programmatic access token (PAT)**, and optionally **Snowflake warehouse**. For the username, enter your login name, which you find in Snowsight under your name > **Connect a tool to Snowflake** > **Login name**. Select **Save and start**. | ||||||||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should this field take the login name or the user name? The app's helper query on the Review step fills it with |
||||||||||||||||||
|
|
||||||||||||||||||
|  | ||||||||||||||||||
|
|
||||||||||||||||||
| The Snowflake credentials are stored only in the schedule's configuration in your Apify account. | ||||||||||||||||||
|
Comment on lines
+198
to
+202
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Readers need to know where the credentials are stored before they enter the PAT, so move that sentence into the Review step.
Suggested change
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The app stores the PAT in the schedule's Apify task configuration, so can anyone with access to the Apify account, like organization members, read it there? If so, should the page say that, so users scope the PAT's role accordingly? |
||||||||||||||||||
|
|
||||||||||||||||||
| ### Schedule an existing import | ||||||||||||||||||
|
|
||||||||||||||||||
| After a successful import on the **Import Dataset** page, select **Schedule export** to repeat the Actor run that produced the dataset, with the same input, run options, and destination table. You only name the schedule and set its frequency before the review. This works for a run's default dataset, and not for imports fetched with **Unwind** or **Flatten**. | ||||||||||||||||||
|
|
||||||||||||||||||
| ### Manage schedules | ||||||||||||||||||
|
|
||||||||||||||||||
| Select a schedule in the list to open its detail page, where you can: | ||||||||||||||||||
|
|
||||||||||||||||||
| - **Run now** - start a run immediately, outside the schedule. | ||||||||||||||||||
| - **Pause** - stop the schedule's runs. A paused schedule shows **Resume** instead, which restarts them. | ||||||||||||||||||
| - **Delete** - remove the schedule and everything it created in your Apify account. | ||||||||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What exactly does Delete remove, and does it touch the Snowflake table? If it only removes the Apify schedule and task, something like this would be clearer:
Suggested change
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. A schedule is made of an Apify schedule, task, and webhook, so the suggestion should cover the webhook too:
Suggested change
|
||||||||||||||||||
| - **Edit timing** - change the frequency. | ||||||||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What happens to a schedule when the PAT expires, and how do users update the stored credentials? Edit timing only changes the frequency. |
||||||||||||||||||
| - **Run history** - check each run's status, start time, and the number of rows it imported. | ||||||||||||||||||
|
|
||||||||||||||||||
|  | ||||||||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could this be reshot with realistic data? "PR test schedule" with 1 row per run looks like a test, and it highlights View in Apify Console, which the text never mentions. |
||||||||||||||||||
|
|
||||||||||||||||||
| ## Next steps | ||||||||||||||||||
|
|
||||||||||||||||||
| With Apify data in Snowflake, you can query it with standard SQL, join it with your existing tables, and connect it to business intelligence (BI) tools like Tableau, Looker, or Snowflake's own data-sharing features. | ||||||||||||||||||
|
|
||||||||||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The new screenshots no longer match the text around them. For example, this page no longer shows the username and email, Next: Run Actor and Fetch Data are now Run Actor and Fetch data, and the options table lacks Flatten. Was reverting the text updates from 39a5d5e intentional?