rwdocs
Design & Architecture
View

Datavant Tokenization

Datavant is a third-party PHI tokenization vendor. We run their tokenizer inside Snowflake to convert Natera patient identifiers into Datavant tokens, then join Natera records to partner datasets (Forian, Veradigm claims) for real-world-evidence analysis — without ever exposing raw PHI.

Overview

The tokenization service runs on Snowpark Container Services (SCS), Snowflake’s managed container runtime. It replaced the legacy EC2 approach (which incurred ~9 seconds of subprocess startup per batch) in April 2026. A sidecar pattern runs the Datavant binary persistently in serve mode alongside a Flask/gunicorn wrapper, exposing SQL-callable service functions with no Python scripts or external orchestration.

The de-identification boundary

RECORD_ID values are joinable to the dbt de-identified views, which use the same salt and HMAC construction. The salt is per-environment, sourced from AWS Secrets Manager and read at run time from the grant-locked LIMS_UTILS.HASH_SALT_KEY — never inlined into object DDL.

Architecture

The container runs two processes: the Datavant binary in persistent serve mode (port 9090, caches credentials/salts/keys), and a Flask/gunicorn wrapper (port 8080) that translates Snowflake’s service-function batch protocol into HTTP calls to the sidecar.

text
Snowflake Service Function (SQL)  → HTTP POST batch (up to 1,000 rows)  → gunicorn/Flask (port 8080)  → HTTP POST → Datavant serve mode (localhost:9090)  → Response returned (no CSV I/O, no subprocess startup)

All objects reside in CLINICOGENOMICS.LIMS_UTILS: the compute pool, service, the HASH_PATIENT_ID UDF, the TOKENIZE_PATIENTS and TRANSFORM_TO_TRANSIT_TOKENS service functions, and the token tables.

Token lifecycle

  1. Tokenize (TOKENIZE_PATIENTS) — PII in, Natera-side token_1 / token_2 out.
  2. Transform (TRANSFORM_TO_TRANSIT_TOKENS) — Natera tokens into partner-specific transit tokens for delivery.
  3. Onboard (CLI-only) — upload tokens to Datavant Connect via SFTP. Not exposed as a service function (concurrent uploads collide on the same remote path).