Basic Features

Here are some of the supported features in a nutshell:

• Data Collaboration - Get event notifications with Activity feeds. Send alerts & notifications using webhooks. Add Announcements to notify the team of upcoming changes. Add Tasks to request descriptions or glossary term approval workflows. Add user mentions and collaborate using conversation threads.

• Data Quality and Profiler - Standardized tests and data quality metadata. Groups related tests as Test Suites. Supports custom SQL data quality tests. Has an interactive dashboard to drill down to the details.

• Data Lineage - Supports rich column-level lineage. Effectively filters queries to extract lineage. Edit lineage manually as required and connect the entities with a no-code editor.

• Comprehensive Roles and Policies - Handles complex access control use cases and hierarchical teams.

• Webhooks - Supports webhook integrations. Integrate with Slack, and Microsoft Teams.

• Connectors - Supports 55 connectors to various databases, dashboards, pipelines and messaging services.

• Glossary - Add a Controlled Vocabulary to describe important concepts and terminologies within your organization. Add Glossaries, Terms, Tags, Descriptions, and Reviewers.

• Data Security - Supports Google, Okta, custom OIDC, Auth0, Azure, Amazon Cognito, and OneLogin as identity providers for SSO. Also, supports SAML-based authentication for AWS SSO and Google.

• Secrets Manager Interface - Communicates with any key management store.

• And lots more...

More set of features would be included with the Darkseal rapidly growing.

All Your Data In One Place

Darkseal provides all the data context you need for different use cases in a single place.

Data Discovery

Darkseal enables you to discover your data using a variety of strategies, including: keyword search, data associations (e.g. frequently joined tables, lineage) and complex queries. Using Darkseal you can search across tables, topics, dashboards, pipelines, and services. Darkseal supports detailed metadata for assets and their components (e.g., columns, charts), including support for complex data types such as arrays and structs.

Activity Feeds

The Darkseal home screen features a change activity feed that enables you to view a summary of data change events. This feed shows all changes to data sorted with the most recent changes at the top. The entities in the activity feed are clickable including tables, dashboards, team names, etc. There are activity feeds for:

• All data

• Data for which you are an owner

• Data you are following

Events Notification

The webhook interface allows you to build applications that receive all the data changes happening in your organization through APIs. Register URLs to receive metadata event notifications. Slack integration through incoming webhooks is one of many applications of this feature.

Descriptive Metadata

Add descriptions and tags to tables, columns, and other assets. Darkseal indexes assets based on descriptions, tags, names, and other metadata to enable keyword, advanced search, and filtering to enable you and others in your organization to discover your data.

Complex Data Types

Add descriptions and tags to nested fields in complex data types like arrays and structs. Locate these assets using keyword search or advanced search.

Role-Based Access Control

Darkseal supports Role Based Access Control (RBAC) policies for metadata operations. Each user may be assigned one or more roles. Each role has a defined policy. Policies are composed of a set of rules. Rules allow/deny access to metadata operations such as updating descriptions, tags, owners, and lineage.

Data Lineage

Trace the path of data across tables, pipelines, and dashboards.

You can also edit lineage to provide a richer understanding of the provenance of data. The Darkseal no-code editor provides a drag-and-drop interface. Drop tables, pipelines, and dashboards onto the lineage graph. You may add new edges or delete existing edges to better represent data lineage.

Data Reliability

Build trust in your data by creating tests to monitor that the data is complete, fresh, and accurate.

Metadata Versioning

Darkseal captures changes in both technical metadata (e.g. table schemas) and business metadata (e.g. tags, ownership, descriptions) as new versions of any entity. Metadata changes generate events that indicate which entity changed, who change it, and how it changed. You may use these events to integrate metadata into other tools or trigger actions.

Darkseal maintains the version history for all entities using a number with the format major.minor, starting with 0.1 as the initial version of an entity.

Changes in metadata result in version changes as follows:

• Backward-compatible changes result in a Minor version change. A change in the description, tags, or ownership will increase the version of the entity metadata by 0.1.

• Backward-incompatible changes in a Major version change. For example, when a column in a table is deleted, the version increases by 1.0.

Elasticsearch Integration

Darkseal supports SSL-enabled Elasticsearch (including self-signed certs). Darkseal automatically runs an indexing workflow as new entities are added or updated through ingestion workflows.