Schedule production runs

Grant required roles

To run workflows in BigQuery, you can use a custom service account or your Google Account (Preview). However, custom service account credentials are the default option for scheduled runs. Using Google Account user account credentials is discouraged for scheduled runs.

To run workflows in BigQuery, your custom service account must have the following required roles:

• BigQuery Data Editor (roles/bigquery.dataEditor) on projects to which Dataform needs both read and write access, which usually includes the project hosting your Dataform repository.
• BigQuery Data Viewer (roles/bigquery.dataViewer) on projects to which Dataform needs read-only access.
• BigQuery Job User (roles/bigquery.jobUser) on the project hosting your Dataform repository.

To let Dataform use your custom service account, the default Dataform service agent must have the following roles on the custom service account resource:

• Service Account Token Creator (roles/iam.serviceAccountTokenCreator)
• Service Account User (roles/iam.serviceAccountUser)

To grant these roles, follow these steps:

1. In the Google Cloud console, go to the IAM page.

   Go to IAM

2. Click Grant access.

3. In the New principals field, enter your custom service account ID.

4. In the Select a role menu, select the following roles one by one, using Add another role for each additional role:

   • BigQuery Data Editor
   • BigQuery Data Viewer
   • BigQuery Job User

5. Click Save.

6. In the Google Cloud console, go to the Service accounts page.

   Go to Service accounts

7. Select your custom service account.

8. Go to Principals with access, and then click Grant access.

9. In the New principals field, enter your default Dataform service agent ID.

   Your default Dataform service agent ID is in the following format:

   service-PROJECT_NUMBER@gcp-sa-dataform.iam.gserviceaccount.com
   

10. In the Select a role list, add the following roles:

    • Service Account User
    • Service Account Token Creator

11. Click Save.

Create a Dataform repository

1. In the Google Cloud console, go to the Dataform page.

   Go to Dataform

2. Click add Create repository.

3. On the Create repository page, do the following:

   1. In the Repository ID field, enter quickstart-production.

   2. In the Region list, select europe-west4.

   3. In the Service account field, click Enter manually, and then enter the name of your custom service account. Ensure you enter your custom service account in this field.

   4. Click Create.

4. Click Go to repositories.

Create a release configuration and workflow configuration

To create production compilation results of the quickstart-production repository and schedule a run of production tables, follow these steps:

1. In the Google Cloud console, go to the Dataform page.

   Go to Dataform

2. Click quickstart-production.

3. Click Releases & scheduling, then click Create production release.

4. In the Create release configuration pane, configure the following settings:

   1. In the Release ID field, enter production.
   2. In the Git commitish field, leave the default value main .
   3. In the Schedule frequency section, in the Repeats menu, select Custom.
   4. In the Custom schedule field, enter 0 16 * * *.

   5. In the Timezone menu, select a UTC+1 timezone, for example, Central European Standard Time (CET).

      Every day at 4 PM UTC+1, Dataform compiles the quickstart-production repository and applies the compilation settings configured in this release configuration to create production compilation results.

5. Click Create.

   The production release configuration creates a compilation result of the entire quickstart-production repository every day at 4PM UTC+1.

6. Ensure that you're on the Releases & scheduling tab. Go to the Workflow configurations section and click Create.

7. In the Create workflow configuration pane, configure the following settings:

   1. In the Configuration ID field, enter production.
   2. In the Release configuration menu, select production.
   3. In the Schedule frequency section, in the Repeats menu, select Custom.
   4. In the Custom schedule field, enter 0 17 * * *.

   5. In the Timezone menu, select a UTC+1 timezone, for example, Central European Standard Time (CET).

      Every day at 5PM UTC+1, Dataform runs the latest production compilation result of the quickstart-production repository.

   6. Click All actions.

      Dataform runs all the workflow actions in the production compilation result.

8. Click Create.

   The production workflow configuration runs the entire latest compilation result created by the production release configuration every day at 5PM UTC+1.

View past production compilation results

To view past scheduled production compilation results, follow these steps:

1. In the Google Cloud console, go to the Dataform page.

   Go to Dataform

2. Select the quickstart-production repository.

3. Click Releases & scheduling.

4. In the Release configurations section, click production.

View past production workflow runs

To view past production workflow runs, follow these steps:

1. In the Google Cloud console, go to the Dataform page.

   Go to Dataform

2. Select the quickstart-production repository.

3. Click Workflow Execution Logs.

4. Select a workflow run to see more detailed information, including the status of each action and any logs.