How Snowflake Snowpark enables Python, Java, and Scala code to run directly inside Snowflake — transformations, ML model inference, and application logic executing where the data lives, without data movement or external compute infrastructure.
Snowflake Snowpark is a developer framework that allows Python, Java, and Scala code to run directly inside Snowflake — executing on Snowflake's compute infrastructure, against data that is already in Snowflake, without moving data out to an external processing environment.
Before Snowpark, Python data processing required extracting data from Snowflake, processing it in Pandas or PySpark, and writing results back. Each round-trip added latency, introduced data movement costs, and created operational complexity. Snowpark eliminates the round-trip — Python code runs where the data lives.
What Snowpark Provides
**Snowpark DataFrame API.** A Python (or Java/Scala) API for data transformation that closely mirrors the PySpark DataFrame API. Operations on Snowpark DataFrames are translated to SQL and executed on Snowflake's compute engine. The translation happens lazily — operations are composed into a query plan and executed only when an action is triggered (show(), collect(), write()).
from snowflake.snowpark import Session
from snowflake.snowpark.functions import col, sum as snow_sum
session = Session.builder.configs(connection_params).create()
orders = session.table("orders")
revenue_by_region = (
orders
.filter(col("status") == "completed")
.group_by("region")
.agg(snow_sum("revenue").alias("total_revenue"))
)
revenue_by_region.show()
The Snowpark DataFrame code is compiled to a SQL query and executed on Snowflake's virtual warehouse. The Python code runs in the Snowpark client library; the actual data processing happens in Snowflake.
**User-Defined Functions (UDFs) in Python.** Python functions can be registered as Snowflake UDFs and called from SQL. The Python function runs on Snowflake's compute nodes, not on an external server. This allows Python logic — string manipulation, complex calculations, external library calls — to be applied at query scale inside Snowflake.
from snowflake.snowpark.functions import udf
from snowflake.snowpark.types import StringType
@udf(return_type=StringType(), input_types=[StringType()])
def extract_domain(email: str) -> str:
return email.split('@')[1] if '@' in email else None
# Call from SQL
session.sql("SELECT extract_domain(email) FROM customers").show()
**Vectorised UDFs (UDTFs with Pandas).** For vectorised Python operations, Snowpark supports Pandas UDFs that process a batch of rows as a Pandas DataFrame rather than one row at a time. Vectorised UDFs are significantly faster than scalar UDFs for operations that can be expressed with Pandas/NumPy.
**Stored Procedures in Python.** Python stored procedures execute Python code on Snowflake's compute nodes as part of a stored procedure call. Unlike UDFs (which transform individual rows), stored procedures can execute complex multi-step logic including multiple SQL statements, conditional logic, loops, and error handling.
**Snowpark ML Modeling.** Snowflake Snowpark ML provides scikit-learn-compatible ML estimators and transformers that run on Snowflake's compute, alongside preprocessing utilities (StandardScaler, OneHotEncoder, train_test_split) that work directly on Snowpark DataFrames without exporting data.
Snowpark vs PySpark
Snowpark and PySpark (Spark on Databricks) serve similar purposes — Python data processing at scale — but from different starting positions:
**Snowpark runs in Snowflake:** all processing happens on Snowflake's virtual warehouses. No separate cluster to manage. Data does not leave Snowflake. Cost is Snowflake warehouse credits.
**PySpark runs on Spark clusters:** either managed by Databricks, EMR, or self-managed. Spark has richer ML capabilities (MLlib, integration with Delta Live Tables), more mature streaming processing (Structured Streaming), and a larger ecosystem. Data must be accessible to the Spark cluster.
When Snowpark is preferable:
- Organisation is already Snowflake-centric and wants to add Python processing without managing a separate Spark cluster
- Use case is primarily data transformation and feature engineering at warehouse scale
- Team is comfortable with SQL and wants a Python API over Snowflake data
When Spark/PySpark is preferable:
- Complex streaming and real-time processing requirements
- Large ML workloads requiring GPU acceleration or complex MLlib pipelines
- Organisation is Databricks-centric with existing Delta Lake investment
- Need for the broader Spark ecosystem (MLflow, Delta Live Tables, Koalas)
Snowpark for ML Inference at Scale
One of Snowpark's most valuable use cases: deploying trained ML models for batch inference inside Snowflake. A model trained externally (scikit-learn, XGBoost, PyTorch) is serialised and deployed as a Snowflake User-Defined Function or stored procedure. Inference runs against the full dataset inside Snowflake without exporting data.
This pattern — train externally, infer in Snowflake via Snowpark — separates the training environment (where you need flexible ML infrastructure) from the inference environment (where you want co-location with the data and governance controls).
For data engineering teams working with Snowflake and evaluating Python processing capabilities, our data architecture consulting practice covers Snowflake architecture and Snowpark implementation — contact us to discuss your requirements.
A former Microsoft data architect audits your data foundation, identifies your top priorities, and sends you a written plan. Free. No pitch.
Book a Call →