# Datavant Tokenization

Datavant is a third-party PHI tokenization vendor. We run their tokenizer inside Snowflake to convert Natera patient identifiers into Datavant tokens, then join Natera records to partner datasets (Forian, Veradigm claims) for real-world-evidence analysis — without ever exposing raw PHI.

## Overview

The tokenization service runs on Snowpark Container Services (SCS), Snowflake’s managed container runtime. It replaced the legacy EC2 approach (which incurred ~9 seconds of subprocess startup per batch) in April 2026. A sidecar pattern runs the Datavant binary persistently in serve mode alongside a Flask/gunicorn wrapper, exposing SQL-callable service functions with no Python scripts or external orchestration.

## The de-identification boundary

> **Never pass raw PATIENT_ID**
>
> Raw patient identifiers must never be passed directly to service functions. Always hash via `HASH_PATIENT_ID()` first. The hash is keyed HMAC-SHA256: `RECORD_ID = HMAC-SHA256(key=salt, msg=patient_id)`, a 64-char lower-hex digest.

`RECORD_ID` values are joinable to the dbt de-identified views, which use the same salt and HMAC construction. The salt is per-environment, sourced from AWS Secrets Manager and read at run time from the grant-locked `LIMS_UTILS.HASH_SALT_KEY` — never inlined into object DDL.

## Architecture

The container runs two processes: the Datavant binary in persistent serve mode (port 9090, caches credentials/salts/keys), and a Flask/gunicorn wrapper (port 8080) that translates Snowflake’s service-function batch protocol into HTTP calls to the sidecar.

```text
Snowflake Service Function (SQL)
  → HTTP POST batch (up to 1,000 rows)
  → gunicorn/Flask (port 8080)
  → HTTP POST → Datavant serve mode (localhost:9090)
  → Response returned (no CSV I/O, no subprocess startup)
```

All objects reside in `CLINICOGENOMICS.LIMS_UTILS`: the compute pool, service, the `HASH_PATIENT_ID` UDF, the `TOKENIZE_PATIENTS` and `TRANSFORM_TO_TRANSIT_TOKENS` service functions, and the token tables.

## Token lifecycle

1. **Tokenize** (`TOKENIZE_PATIENTS`) — PII in, Natera-side `token_1` / `token_2` out.
2. **Transform** (`TRANSFORM_TO_TRANSIT_TOKENS`) — Natera tokens into partner-specific transit tokens for delivery.
3. **Onboard** (CLI-only) — upload tokens to Datavant Connect via SFTP. Not exposed as a service function (concurrent uploads collide on the same remote path).

> **Cast inputs to constrained widths**
>
> Always cast input columns to constrained widths (`VARCHAR(200)`, not bare `VARCHAR`). Bare `VARCHAR` defaults to ~16MB, causing the batch estimator to send ~150 rows instead of the optimal 1,000 — a 5–7× throughput hit. And use `TO_VARCHAR(dob::DATE, 'MM/DD/YYYY')` for DOB.