ETL for your database

Loading the data export into a database takes more than copying files. The data is not append-only: some tables must be truncated before every load, others must be merged with UPSERT or MERGE logic because existing rows change as well as new rows arriving. Load it the wrong way and the database quietly holds wrong data.

The ETL service (extract, transform, load) does this for you. It runs on our infrastructure and delivers the data into your database the right way. For that it needs access to your database and to the cloud storage the export lands in. That does not mean access to your whole database or storage; each database page below says exactly what to grant, and your security team will want to read it. One thing to know before that conversation: Attribution has no static IP address. Each run connects from a different, one-off host, so access cannot be restricted to an IP allow-list; it rests on the credentials you create for the service.

The export runs once a day, so the load runs once a day. Each run:

Connects to your database with the credentials you provide.
Creates the schema if it is not there yet.
Loads the day's export from your cloud storage into temporary tables.
Runs the SQL that removes superseded rows and inserts the current ones into the destination tables.

After a run, your database holds the same data Attribution uses to build your dashboard, and your analysts can build on it; see Data schema for what is there.

The ETL service is set up under Settings → Data Export, next to the storage export it reads from. It supports:

If your database is not listed, or to load the export yourself, see Manually loading data export, or write to [email protected].