What is a Pipeline?

A Pipeline is a collection of Steps. Pipelines are used for automatic (or scheduled) processing of incoming data and for Dropzone Data Requests, without the need to open a Worksheet to do so. Pipelines are useful when you need to regularly process new incoming data of the same original schema in the same way i.e. applying the same set of Processing Steps.

A data Pipeline starts with at least one Import Step and ends with one Load Step. In between the Import and Load Steps are Processing steps. When you run a Pipeline, data will be extracted from the Source, processed according to the Processing Steps, and finally loaded to the Database.

Deploying Pipelines

To create a Pipeline, you must deploy the Dataflow from a Worksheet. To do this, open the Worksheet that you want to convert to a Pipeline. The Worksheet must have an Import Step and a Load to Database Step that has been used to export data to a database table. Processing Steps are optional, but if any are present, you must run them before deploying.

Select the "Deploy as Pipeline" button at the top of the Dataflow (right Sidebar). Enter the Pipeline name, the loading destination, and indicate if you want to overwrite, append, or merge to the destination table. Optionally, you can enter a Pipeline description.

To view your Pipeline, select the Pipeline icon in the left menubar.

Deleting Pipelines

To delete a Pipeline, open the Pipeline Manager using the icon in the left menubar. Select the three dots icon at the top right of the Pipeline you wish to delete and select "Delete." Upon confirming your decision, the Pipeline will be deleted.

Requesting Data to Pipelines

Open the Pipeline Manager using the icon in the top menubar. Select the "Request Data" button at the top right of the desired Pipeline. Enter the data sender's email, the name of the Dropzone Request, and optionally enter a description and instructions for the request. Select "Request Data" to confirm.

Once you send a request, the data sender will receive an email notification. You will be notified by email when they have fulfilled the request. Once the request has been fulfilled, navigate back to your Pipeline Manager and select the Dropzone Import card for this request in your Pipeline. Here, you will need to accept or reject the incoming data. By clicking accept, the requested data will be ingested into your Pipeline (and by extension to the destination table determined by the Pipeline).

Uploading Data to Pipelines

Pipelines are run when you upload new data to them. This process differs depending on the method used to import data. There are 3 methods to import data to Pipelines.

Drag and Drop (File Upload)

To upload new data to a drag-and-drop Pipeline, select the import card and drag-and-drop your data then click "Upload File."

Data Connectors

To upload new data to an external source Pipeline, select the import card and ensure that the correct table is selected. Click "Import Data" to run the Pipeline with new data.