ODPS Console

The ODPS Console is a command-line interface for interacting with Alibaba Cloud's MaxCompute ( formerly ODPS) big data processing platform. It provides a powerful and flexible way to manage and interact with MaxCompute projects, tables, resources, and jobs directly from the terminal.

MaxCompute is a fully managed, serverless data warehousing and analytics platform that allows you to process exabytes of data with high concurrency and throughput. The ODPS Console enables developers, data engineers, and analysts to perform common tasks such as running SQL queries, uploading/downloading data, managing resources, and submitting MapReduce jobs.

Features

The ODPS Console offers a comprehensive set of features organized in a modular plugin architecture:

• SQL Execution - Run interactive SQL queries and scripts against MaxCompute tables with support for:

  • Standard SQL syntax with extensions
  • Interactive query mode for faster execution
  • Query cost estimation and confirmation for large data operations
  • Result set visualization and export
  • Support for materialized views, virtual views, and external tables

• Data Tunnel - High-performance data upload and download between local files and MaxCompute tables with:

  • Support for various file formats (CSV, JSON, etc.)
  • Parallel data transfer for improved performance
  • Data size confirmation to prevent accidental large transfers
  • Custom tunnel endpoints for optimized data transfer

• MapReduce Jobs - Submit and manage MapReduce jobs on the MaxCompute platform:

  • Execute distributed computing tasks
  • Monitor job progress and status
  • Manage job configurations and parameters

• Resource Management - Create, update, and delete resources used by MaxCompute functions:

  • Support for JAR, Python, and other resource types
  • Function creation and management with class path specification
  • Resource dependency management for complex functions

• Workflow Management - Execute and monitor XFlow workflows:

  • Manage offline and online machine learning models
  • Execute complex data processing pipelines
  • Monitor workflow instances and status

• Authentication - Secure authentication with multiple credential options:

  • AccessKey/SecretKey authentication
  • STS token authentication for temporary credentials
  • Application-based authentication
  • Support for various account providers (aliyun, sts, etc.)

• Volume File System - Manage files in the MaxCompute volume file system:

  • Create and manage external volumes
  • File operations within the MaxCompute environment
  • Integration with Hadoop-compatible file system operations

• Cupid Integration - Support for managed service features:

  • Spark job submission and management
  • Job monitoring and visualization
  • Integration with Alibaba Cloud managed services

• Advanced Table Operations - Comprehensive table management capabilities:

  • Show tables with filtering and pattern matching
  • Detailed table schema inspection with extended metadata
  • Partition management for partitioned tables
  • Table lifecycle and storage management

• Instance Management - Complete control over job instances:

  • Submit, monitor, and manage SQL and other job instances
  • Instance priority settings for resource allocation
  • Wait and status checking commands
  • Instance termination capabilities

• Project Management - Multi-project support with:

  • Project switching and context management
  • Project-level metadata inspection
  • Cross-project resource access

• Interactive Mode - Enhanced command-line experience:

  • Persistent session with command history
  • Tab completion for commands and table names
  • Syntax highlighting and formatting
  • Multi-line command editing

Getting Started

Prerequisites

• Java 8 or higher
• An Alibaba Cloud account with MaxCompute service enabled
• Network access to MaxCompute service endpoints

Installation

Option 1: Download Pre-built Package (Recommended)

1. Download the latest release: bash wget https://github.com/aliyun/aliyun-odps-console/releases/latest/download/odpscmd_public.zip

If the link above doesn't work, try: bash wget https://maxcompute-repo.oss-cn-hangzhou.aliyuncs.com/odpscmd/latest/odpscmd_public.zip

1. Extract the package: bash unzip odpscmd_public.zip cd odpscmd_public

2. Verify the installation by checking the directory structure: bin/ # Executable scripts conf/ # Configuration files lib/ # Library dependencies plugins/ # Plugin modules

Option 2: Build from Source

1. Ensure you have the prerequisites installed:

   • Java 8 or higher
   • Apache Maven 3.0 or higher

2. Clone the repository and build: bash git clone https://github.com/aliyun/aliyun-odps-console.git cd aliyun-odps-console mvn clean package -DskipTests

3. The built package will be available at: odps-console-dist-public/target/odpscmd_public.zip

Configuration

Before using the ODPS Console, you need to configure it with your MaxCompute credentials and settings.

1. Create a configuration file at conf/odps_config.ini: ```ini # Basic configuration (required) project_name = your_project_name access_id = your_access_id access_key = your_access_key end_point = your_endpoint

# Optional configurations for enhanced functionality log_view_host = https://logview.odps.aliyun.com debug = false instance_priority = 9 ```

1. You can obtain your endpoint from the MaxCompute endpoints documentation.

2. For detailed configuration options, see the Configuration section below.

Quick Start

1. Run the console in interactive mode: bash ./bin/odpscmd

2. You should see a prompt like: ``` ODPS Console Aliyun ODPS Command Line Tool Version 0.52.3-public

odps@your_project_name> ```

1. Execute a simple command to verify your setup: sql odps@your_project_name> show tables;

2. For batch execution, you can run commands directly: ```bash # Execute a single SQL command ./bin/odpscmd -e "show tables;"

# Execute a SQL script file ./bin/odpscmd -f /path/to/script.sql ```

Common First Steps

1. List tables in your project: sql show tables;

2. Describe a table structure: sql desc table_name;

3. Run a simple query: sql select * from table_name limit 10;

4. Upload data to a table: sql tunnel upload /path/to/local/file.txt table_name;

5. Download data from a table: sql tunnel download table_name /path/to/local/file.txt;

Command Line Options

The ODPS Console supports several command-line options:

• -e "command" - Execute a single command and exit
• -f script.sql - Execute commands from a script file
• --config file - Specify a custom configuration file
• -h, --help - Display help information
• -v, --version - Display version information

Example:

./bin/odpscmd -e "select count(*) from my_table;" --config /path/to/custom_config.ini

Configuration

The ODPS Console requires configuration to connect to your MaxCompute project. The configuration is specified in an odps_config.ini file with the following parameters:

Basic Configuration Parameters

Network and Proxy Configuration

Logging and Debugging Configuration

Authentication Configuration

Job and Execution Configuration

Interactive Mode Configuration

Tunnel and Data Transfer Configuration

Performance and Optimization Configuration

System and Update Configuration

Session and Fallback Configuration

| Parameter | Description | Required | Default Value | |---