Trigger Data Ingestion On Demand - Preview

Data Lake is deprecated. As of September 2024, Data Lake is deprecated and will reach end-of-life. It will be removed on September 30, 2025. If you use Data Lake, you should migrate to alternative solutions before the service is removed. To learn more, see Atlas Data Lake Deprecation Guide.

You can manually trigger an ingestion of snapshot data from the Atlas cluster to Atlas Data Lake datasets if you configured On Demand extraction in your Data Lake pipeline. You can trigger data ingestion from the Atlas UI, the Data Lake Pipelines API, or the Atlas CLI.

Procedure

To trigger the specified data lake pipeline for your project using the Atlas CLI, run the following command:

atlas dataLakePipelines trigger <pipelineName> [options]

To learn more about the command syntax and parameters, see the Atlas CLI documentation for atlas dataLakePipelines trigger.

Tip

See: Related Links

Watch for a Pipeline Run to Complete

To watch for the specified data lake pipeline run to complete using the Atlas CLI, run the following command:

atlas dataLakePipelines runs watch <pipelineName> [options]

To learn more about the command syntax and parameters, see the Atlas CLI documentation for atlas dataLakePipelines runs watch.

Tip

See: Related Links

To trigger data ingestion through the API, send a POST request to the Data Lake trigger endpoint with the name of the pipeline for which you want to trigger data ingestion. To learn more about the API syntax and parameters for the trigger endpoint, see Trigger On-Demand Snapshot Ingestion.

Tip

You can send a GET request to the Data Lake availableSnapshots endpoint to retrieve the list of backup snapshots that you can use to trigger an on-demand pipeline run. To learn more about the API syntax and options for the availableSnapshots endpoint, see Return Available Backup Snapshots for One Data Lake Pipeline.

Log in to MongoDB Atlas.

Go to Atlas Data Lake in the Atlas UI.

If it's not already displayed, select the organization that contains your project from the Organizations menu in the navigation bar.
If it's not already displayed, select your project from the Projects menu in the navigation bar.
In the sidebar, click Data Lake under the Deployment heading.

Click the vertical ellipsis () for the Data Lake for which you configured On Demand ingestion and select Trigger an On Demand Pipeline Run.

Select the snapshot, from which to ingest data, from the dropdown.

The dropdown shows a list of all the snapshots on your Atlas cluster. However, you can select only the snapshots from which Data Lake hasn't yet ingested data; the grayed-out snapshots are snapshots from which your Data Lake has already ingested data.

You can also send a GET request to the Data Lake availableSnapshots endpoint to retrieve the list of backup snapshots that you can use to trigger an on-demand pipeline run. To learn more about the API syntax and options for the availableSnapshots endpoint, see Return Available Backup Snapshots for One Data Lake Pipeline.

Click Confirm.

Atlas displays a blue banner at the top of the page that shows the data ingestion status.

Back

Edit a Data Lake Pipeline

Pause and Resume Data Ingestion