# Command Line Interface

Meltano provides a command line interface (CLI) that allows you to manage the configuration and orchestration of Meltano instances. It provides a single source of truth for the entire data pipeline. The CLI makes it easy to develop, run, and debug every step of the data life cycle.

# add

The add command allows you to add an extractor, loader, or transform to your Meltano instance.

# Extractor / Loader

When you add a extractor or loader to a Meltano instance, Meltano will:

1. Add it to the meltano.yml file
2. Installs it in the .meltano directory with venv and pip

# Examples

# Extractor / Loader Template
meltano add [extractor | loader] [name_of_plugin]

# Extractor Example
meltano add extractor tap-gitlab

# Loader Example
meltano add loader target-postgres

# Transform

When you add a transform to a Meltano instance, Meltano will:

1. Installs dbt transformer to enable transformations (if needed)
2. Add transform to meltano.yml file
3. Updates the dbt packages and project configuration

# Example

# Transform Template
meltano add [transform] [name_of_transform]

# Model

When you add a model to a Meltano instance, Meltano will:

1. Add a model bundle to your meltano.yml file to help you interactively generate SQL
2. Installed inside the .meltano directory which are then available to use in the Meltano webapp

# Example

meltano add model [name_of_model]

# Orchestration

When you add an orchestrator to a Meltano instance, Meltano will:

1. Adds an orchestrator plugin to your meltano.yml
2. Installs it

# Example

meltano add orchestrator [name_of_orchestrator]

# config

Enables you to change a plugin's configuration.

Meltano uses configuration layers to resolve a plugin's configuration:

1. Environment variables
2. Plugin definition's config: attribute in meltano.yml
3. Settings set via meltano config or in the UI (stored in the system database)
4. Default values set in the setting definition in discovery.yml

# How to use

# Displays the plugin's configuration.
meltano config <plugin_name>

# List the available settings for the plugin.
meltano config <plugin_name> list

# Sets the configuration's setting `<name>` to `<value>`.
meltano config <plugin_name> set <name> <value>

# Remove the configuration's setting `<name>`.
meltano config <plugin_name> unset <name>

# Clear the configuration (back to defaults).
meltano config <plugin_name> reset

# discover

Lists the available plugins you are interested in.

# How to Use

# List all available plugins
meltano discover all

# Only list available extractors
meltano discover extractors

# Only list available loaders
meltano discover loaders

# Only list available models
meltano discover models

# elt

This allows you to run your ELT pipeline to Extract, Load, and Transform the data with configurations of your choosing:

1. The job_id is autogenerated using the current date and time if it is not provided (via --job_id or $MELTANO_JOB_ID)
2. The run_id is a UUID autogenerated at each run
3. All the output generated by this command is also logged in .meltano/run/elt/{job_id}/{run_id}/elt.log

# How to use

meltano elt <extractor> <loader> [--job_id TEXT] [--transform run] [--dry]

# Parameters

• The --transform option can be:

  • run: run the Transforms
  • skip: skip the Transforms (Default)
  • only: only run the Transforms (skip the Extract and Load steps)

# Examples

meltano select --exclude tap-carbon-intensity '*' 'longitude'

meltano select --exclude tap-carbon-intensity '*' 'latitude'

This will exclude all longitude and latitude attributes.

# extract

Extract data to a loader and optionally transform the data

# How to Use

meltano extract [name of extractor] --to [name of loader]`

# init

Used to create a new meltano project with a basic infrastructure in place in the current directory that the user is in.

# How to use

# Format
meltano init [project_name] [--no_usage_stats]

# Parameters

• project_name - This determines the folder name for the project

# Options

• no_usage_stats - This flag disables sending anonymous usage data when creating a new project.

# install

Installs all the dependencies of your project based on the meltano.yml file.

# How to Use

meltano install

# invoke

• meltano invoke <plugin_name> PLUGIN_ARGS...: Invoke the plugin manually.

# list

Use --list to list the current selected tap attributes.

Note: --all can be used to show all the tap attributes with their selected status.

# permissions

Use this command to check and manage the permissions of a Snowflake account.

meltano permissions grant <spec_file> --db snowflake [--dry] [--diff] [--full-refresh]

Given the parameters to connect to a Snowflake account and a YAML file (a "spec") representing the desired database configuration, this command makes sure that the configuration of that database matches the spec in an additive manner. If there are differences, it will return the sql grant and revoke commands required to make it match the spec. If there are additional permissions set in the database this command will not create the necessary revoke commands with the exception of:

• Role membership for Users and Roles
• Database Read/Write privileges
• Shared Database privileges
• Warehouse Operate/Usage privileges

We currently support only Snowflake, as pgbedrock can be used for managing the permissions in a Postgres database.

# spec_file

The YAML specification file is used to define in a declarative way the databases, roles, users and warehouses in a Snowflake account, together with the permissions for databases, schemas and tables for the same account.

Its syntax is inspired by pgbedrock, with additional options for Snowflake.

All permissions are abbreviated as read or write permissions, with Meltano generating the proper grants for each type of object. This includes shared databases which have simpler and more limited permissions than non-shared databases.

Tables and views are listed under tables and handled properly behind the scenes.

If * is provided as the parameter for tables the grant statement will use the ALL <object_type>s in SCHEMA syntax. It will also grant to future tables and views. See Snowflake documenation for ON FUTURE

If a schema name includes an asterisk, such as snowplow_*, then all schemas that match this pattern will be included in grant statement. This can be coupled with the asterisk for table grants to grant permissions on all tables in all schemas that match the given pattern. This is useful for date-partitioned schemas.

All entities must be explicitly referenced. For example, if a premission is granted to a schema or table then the database must be explicitly referenced for permissioning as well.

A specification file has the following structure:

# Databases
databases:
    - db_name:
        shared: boolean
    - db_name:
        shared: boolean
    ... ... ...

# Roles
roles:
    - role_name:
        warehouses:
            - warehouse_name
            - warehouse_name
            ...

        member_of:
            - role_name
            - role_name
            ...

        privileges:
            databases:
                read:
                    - database_name
                    - database_name
                    ...
                write:
                    - database_name
                    - database_name
                    ...
            schemas:
                read:
                    - database_name.*
                    - database_name.schema_name
                    - database_name.schema_partial_*
                    ...
                write:
                    - database_name.*
                    - database_name.schema_name
                    - database_name.schema_partial_*
                    ...
            tables:
                read:
                    - database_name.*.*
                    - database_name.schema_name.*
                    - database_name.schema_partial_*.*
                    - database_name.schema_name.table_name
                    ...
                write:
                    - database_name.*.*
                    - database_name.schema_name.*
                    - database_name.schema_partial_*.*
                    - database_name.schema_name.table_name
                    ...

        owns:
            databases:
                - database_name
                ...
            schemas:
                - database_name.*
                - database_name.schema_name
                - database_name.schema_partial_*
                ...
            tables:
                - database_name.*.*
                - database_name.schema_name.*
                - database_name.schema_partial_*.*
                - database_name.schema_name.table_name
                ...

    - role_name:
    ... ... ...

# Users
users:
    - user_name:
        can_login: boolean
        member_of:
            - role_name
            ...
    - user_name:
    ... ... ...

# Warehouses
warehouses:
    - warehouse_name:
        size: x-small
    ... ... ...

For a working example, you can check the Snowflake specification file that we are using for testing meltano permissions.

# --db

The database to be used, either postgres or snowflake. Postgres is still experimental and may be fully supported in the future.

# --diff

When this flag is set, a full diff with both new and already granted commands is returned. Otherwise, only required commands for matching the definitions on the spec are returned.

# --dry

When this flag is set, the permission queries generated are not actually sent to the server and run; They are just returned to the user for examining them and running them manually.

Currently we are still evaluating the results generated by the meltano permissions grant command, so the --dry flag is required.

# --full-refresh

When this flag is set, the permission queries generated are revoke statements for all roles, warehouse, databases, schemas, and tables listed in the spec file. Currently it will not revoke ownership of database objects or disable users. The revoke commands are run prior to the grant commands.

# Connection Parameters

The following environmental variables must be available to connect to Snowflake:

$PERMISSION_BOT_USER
$PERMISSION_BOT_PASSWORD
$PERMISSION_BOT_ACCOUNT
$PERMISSION_BOT_DATABASE
$PERMISSION_BOT_ROLE
$PERMISSION_BOT_WAREHOUSE

# schedule

Meltano provides a schedule method to run specified ELT pipelines at regular intervals. Schedules are defined inside the meltano.yml project as such:

• meltano schedule <schedule_name> <extractor> <loader> <interval> [--transform]: Schedule an ELT pipeline to run using an orchestrator.
  • meltano schedule list: List the project's schedules.

schedules:
  - name: test
    interval: '@daily'
    extractor: tap-mock
    loader: target-mock
    transform: skip
    env: {}

# select

Use the select command to add select patterns to a specific extractor in your Meltano project.

• meltano select [--list] [--all] <tap_name> [ENTITIES_PATTERN] [ATTRIBUTE_PATTERN]: Manage the selected entities/attributes for a specific tap.

# How to use

Meltano select patterns are inspired by the glob syntax you might find in your operating system.

• *: matches any sequence of characters
• ?: matches one character
• [abc]: matches either a, b, or c
• [!abc]: matches any character but a, b, or c

# Examples

$ meltano select tap-carbon-intensity '*' 'name*'

This will select all attributes starting with name.

$ meltano select tap-carbon-intensity 'region'

This will select all attributes of the region entity.

# Exclude Parameter

Use --exclude to exclude all attributes that match the filter.

# ui

• meltano ui: Start the Meltano UI.

# start (default)

Start the Meltano UI.

# setup

Generate secure secrets in the ui.cfg so that the application is secure.

# --bits

Specify the size of the secrets, default to 256.

# user

# add

Create a Meltano user account, active and ready to be used.

# --overwrite, -f

Update the user instead of creating a new one.

# --role, -G

Add the user to the role. Meltano ships with two built-in roles: admin and regular.

# How to use

meltano user add admin securepassword --role admin

# upgrade

Upgrade Meltano to the latest version.

This function will following process to upgrade Meltano:

• Run pip3 install --upgrade meltano
• Run the database migrations
• Send a SIGHUP to the process running under the .meltano/run/gunicorn.pid, thus restarting the workers

# version

It is used to check which version of Meltano currently installed.

# How to use

meltano --version