Edit an Atlas Data Lake Pipeline - Preview
On this page
You can make changes to your Data Lake pipelines through the Atlas UI, Data Lake Pipelines API, and the Atlas CLI, including:
Edit the data extraction schedule
Edit the dataset retention policy
Edit the data storage region
Change the fields to exclude from your Data Lake datasets
Procedure
To modify the details of the specified data lake pipeline for your project using the Atlas CLI, run the following command:
atlas dataLakePipelines update <pipelineName> [options]
To learn more about the command syntax and parameters, see the Atlas CLI documentation for atlas dataLakePipelines update.
To edit a pipeline through the API, send a PATCH request to the
Data Lake pipelines endpoint
with the name of the pipeline that you want to edit. To learn more
about the pipelines endpoint syntax and parameters for updating a
pipeline, see Update One Data Lake Pipeline.
Tip
You can send a GET request to the
availableSchedules endpoint to retrieve the
list of backup schedule policy items that you can use to change the
schedule of your Data Lake pipelines of type PERIODIC_DPS.
Log in to MongoDB Atlas.
Go to Atlas Data Lake in the Atlas UI.
If it's not already displayed, select the organization that contains your project from the Organizations menu in the navigation bar.
If it's not already displayed, select your project from the Projects menu in the navigation bar.
In the sidebar, click Data Lake under the Deployment heading.
(Optional) Configure a Dataset Retention Policy.
A dataset retention policy can help manage your Data Lake storage by automatically deleting old datasets after a duration you specify.
To configure a dataset retention policy for a pipeline,
toggle Dataset Retention Policy to ON and
enter a time value, in days, weeks, or months.
Important
Changes to your pipeline's dataset retention policy take effect immediately. If you lower the retention duration for a pipeline with existing datasets, then datasets that are beyond the new duration expire immediately.
(Optional) Make changes to your data extraction schedule.
Before making changes to your Basic Schedule, ensure
that your desired data extraction frequency is similar to your
current backup schedule. For example, if you wish to switch to
Daily, you must have a Daily backup schedule configured
in your policy. Or, if you want to switch to a schedule of once a
week, you must have a Weekly backup schedule configured in
your policy. To learn more, see Backup Scheduling.
You can also send a GET request to the Data Lake
availableSchedules endpoint to retrieve the
list of backup schedule policy items that you can use to change
the schedule of your Data Lake pipeline.
(Optional) Make changes to your data storage region.
Atlas Data Lake provides optimized storage in the following AWS regions:
Data Lake Regions | AWS Regions |
|---|---|
Virginia, USA | us-east-1 |
Oregon, USA | us-west-2 |
Sao Paulo, Brazil | sa-east-1 |
Ireland | eu-west-1 |
London, England | eu-west-2 |
Frankfurt, Germany | eu-central-1 |
Mumbai, India | ap-south-1 |
Singapore | ap-southeast-1 |
Sydney, Australia | ap-southeast-2 |
(Optional) Make changes to the fields excluded from your Data Lake datasets.
Click Add Field and specify Field Name to add fields to the excluded fields list.
Click Delete All to remove all the fields from the excluded fields list.
Click next to a field to remove that field from the excluded fields list.