Course contents

SnowPro Core COF-C03 Study GuidePart 4: Data loading and storageSnowpipe Streaming

Snowpipe Streaming

Snowpipe Streaming

In shortSnowpipe Streaming lets an application send rows to a table through an SDK. In the high-performance architecture the rows pass through a PIPE object that can transform them. No file or stage is involved.

Ingestion path · Snowpipe Streaming
Source
Rows in an application, an IoT gateway or a Kafka topic
Trigger
The client sends rows through a channel
Processor
Snowflake serverless ingestion
Destination
A table, queryable within seconds
Compute and billing
High-performance: per uncompressed GB received · classic: serverless compute plus active client connections

The problem

Some sources never produce files. A Kafka topic emits rows continuously. Writing them to files only to load those files again adds delay and moving parts. Amine's real-time stock dashboard needs rows in seconds, not minutes.

How it works

The application uses a Snowflake SDK (or the Kafka connector) to send rows through a channel. A named channel keeps rows in order and tracks progress with offset tokens; an elastic channel is managed by Snowflake and doesn't guarantee order. Snowflake commits the rows and they become queryable within seconds. Two architectures exist. They differ in where the channel points and in how they're billed.

Snowpipe Streaming's two architectures. High-performance, generally available since September 2025: an application or Kafka connector v4 sends rows into a channel, a PIPE object can transform and pre-cluster them, and the rows land in the table within seconds; billing is per uncompressed GB received. Classic: the Ingest SDK client opens a channel on the table and commits rows directly; billing is serverless compute plus active client connections.HIGH-PERFORMANCE · GA SEPTEMBER 2025 · billed per uncompressed GB receivedCLASSIC · billed as serverless compute + active clientsSOURCEApp / Kafka v4rows in memoryCHANNELChannelordered rowsSERVER SIDEPIPE objecttransform,pre-clusterTABLETablequeryablein secondsSOURCEApp / KafkaIngest SDK clientCHANNELChannelopened on the tableTABLETablequeryablein secondsinsert rowscommit

HIGH-PERFORMANCE · GA SEPTEMBER 2025 · billed per uncompressed GB received

SOURCEApp / Kafka v4rows in memory

CHANNELChannelordered rows

SERVER SIDEPIPE objecttransform, pre-cluster

TABLETablequeryable in seconds

CLASSIC · billed as serverless compute + active clients

SOURCEApp / KafkaIngest SDK client

insert rows

CHANNELChannelopened on the table

commit

TABLETablequeryable in seconds

Figure 4.11 Snowpipe Streaming's two architectures. In the high-performance one, channels write through a PIPE object that can transform and pre-cluster rows server-side, and billing is per uncompressed GB received. In the classic one, the Ingest SDK writes to the table directly, and billing is serverless compute plus active client connections.

The high-performance architecture became generally available in September 2025 (November 2025 on Azure and Google Cloud) and is the default choice for new projects. It counts only data values toward the GB measured, not structural bytes such as JSON keys.

The Kafka connector

The Snowflake Connector for Kafka runs in Kafka Connect, reads topics and writes them to Snowflake tables. Version 4, generally available since April 2026 and recommended for new deployments, writes through Snowpipe Streaming's high-performance architecture only. By default it creates one column per JSON key, and the Kafka metadata goes into a RECORD_METADATA VARIANT column when the table has one. The classic version 3 can load through Snowpipe (it stages files, then a pipe loads them) or through Snowpipe Streaming, and the tables it creates have two VARIANT columns: RECORD_CONTENT holds the message and RECORD_METADATA holds the topic, partition, offset and timestamp.

Choosing an ingestion path

Bulk COPYSnowpipeSnowpipe Streaming
SourceFiles in a stageFiles in a stageRows in an application
TriggerYou run the statement (by hand, a task, an orchestrator)Cloud event or REST callThe application sends rows
ProcessorYour virtual warehouseSnowflake serverless computeSnowflake serverless ingestion
DestinationTableTableTable
Who pays for whatWarehouse credits per second while it runsCredits per GB ingestedPer uncompressed GB (high-performance) or compute + clients (classic)
Typical latencyAs often as you schedule itMinutesSeconds
Default on bad dataABORT_STATEMENTSKIP_FILEErrors reported per channel to the client
Load historyTable metadata, 64 daysPipe metadata, 14 daysOffset tokens per named channel
?Does the data arrive as files?
No, rows from an app or Kafka
Snowpipe Streaming

Rows through a channel, queryable in seconds, no stage. Serverless.

Yes
Keep going ↓

Files in a stage. Next: must they be loaded?

?Must the data be copied into Snowflake storage?
No, the files must stay where they are
External table (or Iceberg)

Query in place, read-only, warehouse at query time.

Yes
Keep going ↓

Load with COPY or Snowpipe. Next: when?

?Should each file load soon after it arrives?
Yes, continuous arrival
Snowpipe

Event or REST call per file, serverless, billed per GB.

No, a scheduled batch is fine
COPY INTO on a warehouse

You run it (task, orchestrator); warehouse credits per second.

✓Scenario words: "rows within seconds, no files" → Streaming · "keep the files in the partner's bucket" → external table · "as files arrive" → Snowpipe · "nightly batch" → COPY.
Figure 4.12 Picking the path from a scenario in three questions. The same logic answers most "which loading method" scenarios.
Exam takeaway
  • Snowpipe Streaming ingests rows, not files; no stage is involved.
  • Latency in seconds, versus minutes for Snowpipe.
  • Both are serverless; neither uses your warehouse.
  • Kafka connector tables: v3 gets RECORD_CONTENT and RECORD_METADATA (VARIANT); v4 gets one column per JSON key.