Databricks icon

Databricks

Interact with Databricks API

Databricks SQL → Execute Query

AI-generated

Overview

This node executes SQL queries on a specified Databricks SQL warehouse. It is useful for running data retrieval or manipulation commands directly on Databricks SQL warehouses, enabling integration of Databricks data operations within automated workflows. For example, users can query tables to fetch data subsets or perform analytics within their data pipelines.

Use Case Examples

  1. Run a SELECT query to retrieve the top 10 rows from a table in a Databricks SQL warehouse.
  2. Execute an UPDATE or DELETE SQL statement to modify data in a Databricks SQL warehouse as part of an automated data processing workflow.

Properties

Name Meaning
Warehouse The SQL warehouse to use for executing the query. It supports selection from a list, by ID, or by URL.
Query The SQL query string to execute on the selected warehouse. This is a required field where users input their SQL commands.

Output

JSON

  • json - Each output item represents a row from the query result, with column names as keys and corresponding values as values.

Dependencies

  • Requires Databricks API credentials including host URL and bearer token for authentication.

Troubleshooting

  • Common issues include invalid warehouse ID or URL format, which will cause errors before query execution.
  • Query execution may time out if the query takes too long; the node waits up to 5 minutes polling the query status.
  • If the query fails or is canceled, an error is thrown with details from the Databricks API response.
  • Network errors or API errors are logged and can be handled by enabling 'Continue On Fail' to prevent workflow interruption.

Links

  • Databricks SQL Documentation - Official documentation for Databricks SQL, useful for understanding SQL warehouse concepts and query syntax.

Discussion