Skip to content
flowlakes FlowLakes documentation home

Data sources

A data source is a configured connection that brings external data into FlowLakes. Some sources load rows into an application table you choose; others sync directory identities and assets into discovery. You set up the connection once, map its fields to where the data should land, and the platform keeps pulling it in. This page stays high level; the specifics of each connector live in its own setup screen.

When you’d use it

  • Load records from outside a database, a REST API, or a spreadsheet into an application table.
  • Sync your directory so users and groups from Entra ID stay current in the platform.
  • Feed asset discovery from directory, cloud, and network sources.
  • Bring in a one-off dataset with a file import.

What you manage

Record-loading sources live on the Data Sources screen (/analytics/data-sources). For each one you set:

  • The type. A database (PostgreSQL or MySQL), a REST API, or a file import (CSV or spreadsheet).
  • The connection. The host, endpoint, or file, plus the credential it uses. Credentials are stored separately and referenced, never typed into the source.
  • The target and mapping. The application table to load into, and how each source field maps to a column.
  • The test and sync. Check the connection, then pull data on demand or on a schedule.

Directory and asset sources (Entra ID, LDAP, cloud accounts, and network scanners) are set up as discovery sources on the Sources screen under Assets & Discovery → Configure (/discovery/sources); they become assets and configuration items rather than rows in one table.

Good to know

  • Secrets live in the credential store, not in the source config. A source references a credential; the value is encrypted and never shown back.
  • A record-loading source needs a target table and a field mapping. Without them it has nowhere to put the data.
  • Directory and network syncs feed discovery, not a single table. What they bring in is deduplicated across sources into your asset inventory.
  • An Entra ID source lands its pulls on its own. Its landing mode is direct by default: every scheduled pull is committed to the platform right after it is staged. Switch it to review before commit only when a person will commit each pull on the source page; a pull left unreviewed expires after 72 hours, and the page shows the waiting pull with a Commit action until then. The landing mode is read when a run starts, so a pull that was already running when you switched to direct still waits: commit it, or let the next run land its own pull and this one will expire.
  • One schedule, shown in two places. The schedule on the source page and the schedule on the Data Sources configuration page are the same value, and the source page is where you set it. A six-hourly schedule runs at 00:00, 06:00, 12:00 and 18:00 UTC, within a minute of the boundary, not six hours after the last run.
  • A landed pull supersedes the older ones. A directory pull is a snapshot. Once one lands, older pulls still waiting can no longer be committed, because landing an older snapshot would deactivate everything that appeared since. The source page shows only the newest waiting pull; the Runs tab says for each run whether it landed, is waiting, or is stale.
  • When Entra and Active Directory both describe a person, trust priority decides per field. Each directory source carries a priority. A profile field both carry is written by the higher one; the lower fills only fields still empty; equal priorities keep whichever wrote it first. Raise the source that should win under Edit.
  • Users and applications from Entra land without any app installed. Groups, app-role assignments and group memberships need the Entra Access Management app for their tables; until it is installed the source page lists them as not synced and the sync pulls only what can land.

How it connects

Data sources own bringing data in. Where it goes depends on the type: application records land in the table you map, while directory and asset data flow into CMDB and discovery. The secret a source needs comes from Credentials and secrets, and directory identity ties into Identity and access.

  • Configure a source in the admin guides.
  • Exact shapes: connector types database, REST API, and file import for records, plus directory, cloud, and network for discovery; each record source declares a target table and field mapping; every term in the glossary.