dcrdata

Overview

dcrdata is an original Decred block explorer, with packages and apps for data collection, presentation, and storage. The backend and middleware are written in Go. On the front end, Webpack enables the use of modern javascript features, as well as SCSS for styling.

• dcrdata
• Overview
• Release Status
• Repository Overview
• Requirements
• Docker Support
• Building
  • Preparation
  • Package the Static Web Assets
  • Building dcrdata with Go
  • Setting build version flags
  • Runtime Resources
• Updating
• Upgrading Instructions
  • From v3.x or later
  • From v2.x or earlier
• Getting Started
  • Configuring PostgreSQL (IMPORTANT! Seriously, read this.)
  • CockroachDB Support (experimental)
  • Creating the dcrdata Configuration File
  • Using Environment Variables for Configuration
  • Indexing the Blockchain
  • Starting dcrdata
  • Hiding the PostgreSQL Settings Table
  • Running the Web Interface During Synchronization
• System Hardware Requirements
  • dcrdata only (PostgreSQL on other host)
  • dcrdata and PostgreSQL on same host
• dcrdata Daemon
  • Block Explorer
• APIs
  • Insight API
  • dcrdata API
  • Endpoint List
• Important Note About Mempool
• Command Line Utilities
  • rebuilddb2
  • scanblocks
• Front End Development
  • CSS Guidelines
  • HTML
  • Javascript
  • Web Performance
• Helper Packages
• Internal-use Packages
• Plans
• Contributing
• License

Release Status

Always run the Current release or on the Current stable branch. Do not use master in production.

Repository Overview

../dcrdata                The main Go MODULE. See cmd/dcrdata for the explorer executable.
├── api/types             The exported structures used by the dcrdata and Insight APIs.
├── blockdata             Package blockdata is the primary data collection and
|                           storage hub, and chain monitor.
├── cmd
│   └── dcrdata           MODULE for the dcrdata explorer executable.
│       ├── api           dcrdata's own HTTP API
│       │   └── insight   The Insight API
│       ├── explorer      Powers the block explorer pages.
│       ├── middleware    HTTP router middleware used by the explorer
│       ├── notification  Manages dcrd notifications synchronous data collection.
│       ├── public        Public resources for block explorer (css, js, etc.)
│       └── views         HTML templates for block explorer
├── db
│   ├── cache             Package cache provides a caching layer that is used by dcrpg.
│   ├── dbtypes           Package dbtypes with common data types.
│   └── dcrpg             MODULE and package dcrpg providing PostgreSQL backend.
├── dev                   Shell scripts for maintenance and deployment.
├── docs                  Extra documentation.
├── exchanges             MODULE and package for gathering data from public exchange APIs
│   ├── rateserver        rateserver app, which runs an exchange bot for collecting
│   |                       exchange rate data, and a gRPC server for providing this
│   |                       data to multiple clients like dcrdata.
|   └── ratesproto        Package dcrrates implementing a gRPC protobuf service for
|                           communicating exchange rate data with a rateserver.
├── explorer/types        Types used primarily by the explorer pages.
├── gov                   MODULE for the on- and off-chain governance packages.
│   ├── agendas           Package agendas defines a consensus deployment/agenda DB.
│   └── politeia          Package politeia defines a Politeia proposal DB.
│       ├── piclient      Package piclient provides functions for retrieving data
|       |                   from the Politeia web API.
│       └── types         Package types provides several JSON-tagged structs for
|                           dealing with Politeia data exchange.
├── mempool               Package mempool for monitoring mempool for transactions,
|                           data collection, distribution, and storage.
├── netparams             Package netparams defines the TCP port numbers for the
|                           various networks (mainnet, testnet, simnet).
├── pubsub                Package pubsub implements a websocket-based pub-sub server
|   |                       for blockchain data.
│   ├── democlient        democlient app provides an example for using psclient to
|   |                       register for and receive messages from a pubsub server.
│   ├── psclient          Package psclient is a basic client for the pubsub server.
│   └── types             Package types defines types used by the pubsub client
|                           and server.
├── rpcutils              Package rpcutils contains helper types and functions for
|                           interacting with a chain server via RPC.
├── semver                Defines the semantic version types.
├── stakedb               Package stakedb, for tracking tickets
├── testutil
│   ├── apiload           An HTTP API load testing application
|   └── dbload            A DB load testing application
└── txhelpers             Package txhelpers provides many functions and types for
                            processing blocks, transactions, voting, etc.

Requirements

• Go 1.17 or 1.18
• Node.js 16.x or 17.x. Node.js is only used as a build tool, and is not used at runtime.
• Running dcrd running with --txindex --addrindex, and synchronized to the current best block on the network. On startup, dcrdata will verify that the dcrd version is compatible.
• PostgreSQL 11+

Docker Support

Dockerfiles are provided for convenience, but NOT SUPPORTED. See the Docker documentation for more information. The supported dcrdata build instructions are described below.

Building

The dcrdata build process comprises two general steps:

1. Bundle the static web page assets with Webpack (via the npm tool).
2. Build the dcrdata executable from the Go source files.

These steps are described in detail in the following sections.

NOTE: The following instructions assume a Unix-like shell (e.g. bash).

Preparation

• Install Go

• Verify Go installation:

sh go env GOROOT GOPATH

• Ensure $GOPATH/bin is on your $PATH.

• Clone the dcrdata repository. It is conventional to put it under GOPATH, but this is no longer necessary (or recommend) with Go modules. For example:

sh git clone https://github.com/decred/dcrdata $HOME/go-work/github/decred/dcrdata

• Install Node.js, which is required to lint and package the static web assets.

Note that none of the above is required at runtime.

Package the Static Web Assets

Webpack, a JavaScript module bundler, is used to compile and package the static assets in the cmd/dcrdata/public folder. Node.js' npm tool is used to install the required Node.js dependencies and build the bundled JavaScript distribution for deployment.

First, install the build dependencies:

cd cmd/dcrdata
npm clean-install # creates node_modules folder fresh

Then, for production, build the webpack bundle:

npm run build # creates public/dist folder

Alternatively, for development, npm can be made to watch for and integrate JavaScript source changes:

npm run watch

See Front End Development for more information.

Building dcrdata with Go

Change to the cmd/dcrdata folder and build:

cd cmd/dcrdata
go build -v

The go tool will process the source code and automatically download dependencies. If the dependencies are configured correctly, there will be no modifications to the go.mod and go.sum files.

Note that performing the above commands with older versions of Go within $GOPATH may require setting GO111MODULE=on.

As a reward for reading this far, you may use the build.sh script to mostly automate the build steps.

Setting build version flags

By default, the version string will be postfixed with "-pre+dev". For example, dcrdata version 5.1.0-pre+dev (Go version go1.12.7). However, it may be desirable to set the "pre" and "dev" values to different strings, such as "beta" or the actual commit hash. To set these values, build with the -ldflags switch as follows:

go build -v -ldflags \
  "-X main.appPreRelease=beta -X main.appBuild=`git rev-parse --short HEAD`"

This produces a string like dcrdata version 6.0.0-beta+750fd6c2 (Go version go1.16.2).

Runtime Resources

The config file, logs, and data files are stored in the application data folder, which may be specified via the -A/--appdata and -b/--datadir settings. However, the location of the config file may also be set with -C/--configfile. The default paths for your system are shown in the --help description. If encountering errors involving file system paths, check the permissions on these folders to ensure that the user running dcrdata is able to access these paths.

The "public" and "views" folders must be in the same folder as the dcrdata executable. Set read-only permissions as appropriate.

Updating

Update the repository (assuming you have master checked out in GOPATH):

cd $HOME/go-work/github/decred/dcrdata
git pull origin master

Look carefully for errors with git pull, and reset locally modified files if necessary.

Next, build dcrdata and bundle the web assets:

cd cmd/dcrdata
go build -v
npm clean-install
npm run build # or npm run watch

Note that performing the above commands with versions of Go prior to 1.16 within $GOPATH may require setting GO111MODULE=on.

Upgrading Instructions

From v3.x or later

No special actions are required. Simply start the new dcrdata and automatic database schema upgrades and table data patches will begin.

From v2.x or earlier

The database scheme change from dcrdata v2.x to v3.x does not permit an automatic migration. The tables must be rebuilt from scratch:

1. Drop the old dcrdata database, and create a new empty dcrdata database.

```sql -- Drop the old database. DROP DATABASE dcrdata;

-- Create a new database with the same "pguser" set in the dcrdata.conf. CREATE DATABASE dcrdata OWNER dcrdata; ```

1. Delete the dcrdata data folder (i.e. corresponding to the datadir setting). By default, datadir is in {appdata}/data:

2. Linux: ~/.dcrdata/data

3. Mac: ~/Library/Application Support/Dcrdata/data

4. Windows: C:\Users\<your-username>\AppData\Local\Dcrdata\data (%localappdata%\Dcrdata\data)

5. With dcrd synchronized to the network's best block, start dcrdata to begin the initial block data sync.

Getting Started

Configuring PostgreSQL (IMPORTANT! Seriously, read this.)

It is crucial that you configure your PostgreSQL server for your hardware and the dcrdata workload.

Read postgresql-tuning.conf carefully for details on how to make the necessary changes to your system. A helpful online tool for determining good settings for your system is called PGTune. Note that when using this tool to subtract 1.5-2GB from your system RAM so dcrdata itself will have plenty of memory. DO NOT simply use this file in place of your existing postgresql.conf. DO NOT simply copy and paste these settings into the existing postgresql.conf. It is necessary to edit the existing postgresql.conf, reviewing all the settings to ensure the same configuration parameters are not set in two different places in the file (postgres will not complain).

If you tune PostgreSQL to fully utilize remaining RAM, you are lim