Create Pipelines

Every data import in Dropbase is a pipeline (or requires one). A pipeline is a predetermined set of cleaning and validation steps applied between a source-destination pair. Sources can be a file that’s manually uploaded or received via Dropmail (through an email attachments). Destinations are usually a database table; for most users this will be their managed Dropbase Database.

When you import data to a Dropbase database table, we use a File Upload Pipeline under the hood.

Pipelines automatically perform a set of actions that handles much of the repetitive work performed in cleaning and loading data to the database. These actions include automatic column mapping, remembering cleaning steps, and remembering how to load data in your database (merge, append, overwrite).

Pipelines consist of multiple parts, depending on the pipeline type. There are currently 2 types of Pipelines you can choose from.

File Upload Pipelines

You can use File Upload Pipelines to import data to your database tables in Dropbase. This pipeline type requires multiple steps to set up. You need to specify a destination table, upload an import a source file (CSV or Excel), review data to make sure we’ve opened your file correctly, apply any cleaning steps, and finally specify how you want us to load the data in the database. If you do not have a destination table already, you can create a new one. Note that a destination table is required and you will not be able to properly build a pipeline without creating it first.

When you complete all steps in the Pipeline Wizard, you would have effectively imported data to your database table.


Dropmail Pipelines

Dropmail pipelines allow you to email CSV or Excel attachments to your database. To set it up you only need a destination table. If you don’t have one you can create one. Once you set up a Dropmail, you can email it files. If you set up Dropmail with auto-run and you have an existing set of cleaning steps, we will automatically apply them and attempt to load the data to the database as soon as we receive the email attachment.

Generating a Dropmail email is easy. You don’t need to know how the file looks like. When you receive your first file attachment via Dropmail, you’ll simply add any required cleaning steps and Dropmail will remember these for you. Next time you receive a file via Dropmail, if auto-run is enabled we will use the same cleaning steps and process the file.

If auto-run is not enabled, you can simply open the Dropmail pipeline and manually clean the incoming files each time you receive them.


Cleaning Steps

As soon as you import data into Dropbase you can start adding Cleaning Steps that clean your data the way you want, before loading it to the database. You can add a cleaning step by directly clicking on the column name of the column you want to clean up in the Spreadsheet UI, or by adding a step from the sidebar by clicking on the '+' button.

For a complete list of Cleaning Steps, go to the Steps section in the documentation.

Load to database

Once you have applied cleaning steps to your data and are happy with how it looks, you can load it to your Dropbase Database. Click on "Load to Database" to perform this action. When you click on the 'Load to Database" button, Dropbase will process all your Cleaning Steps and load the data to the database. When the Load Step successfully completes, you are automatically redirected to Database Manager where you can view your the table you just created or loaded data to.

Manage data

Dropbase lets you easily manage all your data from one place using the Dropbase Database Manager. With Database Manager you can view, organize, and edit all of your data, including your databases, folders, tables, and views.

Query Data

Once you have imported, added cleaning steps, and loaded data to your database, you can then Query your data and create Views. To do this, go to the Query menu and simply type any custom SQL and click “Preview”.