Mastering DolphinScheduler Core Scripts: PIDs, ZK Nodes, and Env Overrides

#apachedolphinscheduler #coding #programming #opensource

In modern data processing and workflow management, Apache DolphinScheduler has gained widespread adoption among developers thanks to its operational flexibility and robust orchestration capabilities.

This guide delivers a step-by-step walkthrough of DolphinScheduler's critical scripts, providing an actionable blueprint to master every stage of installation, configuration, and cluster operations.

Prerequisites: Assumes configuration files under ./bin/env/ have been set up.

Installation Workflow Breakdown

./install.sh

Source Environment Variables: Reads environment settings—such as Master and Worker node topologies—from install_env.sh and dolphinscheduler_env.sh.
Directory Provisioning: Creates and sets file permissions for installation directories on target machines.
Package Distribution: Unpacks and distributes DolphinScheduler binaries to remote nodes.
Service Teardown: Gracefully shuts down all active services across the cluster.
ZooKeeper Cleanup: Purges the legacy /dolphinscheduler root node from ZooKeeper.
Service Bootstrap: Launches all DolphinScheduler service components.

File Distribution to Worker Nodes

workDir=`dirname $0`
workDir=`cd ${workDir}; pwd`

source ${workDir}/env/install_env.sh
# Extract workers string; Default: workers=${workers:-"ds1:default,ds2:default,ds3:default,ds4:default,ds5:default"}
# Convert string to array
workersGroup=(${workers//,/ })

# Iterate through array items
for workerGroup in ${workersGroup[@]}
do
  # Example item: ds1:default
  echo $workerGroup;
  # Extract Worker IP address
  worker=`echo $workerGroup | awk -F':' '{print $1}'`
  # Extract Worker group name; Defaults to "default"
  group=`echo $workerGroup | awk -F':' '{print $2}'`
  # Append to IP list
  workerNames+=($worker)
  # Append to Group list
  groupNames+=(${group:-default})
done

# Extract target deployment IPs: ips=${ips:-"ds1,ds2,ds3,ds4,ds5"}
hostsArr=(${ips//,/ })

# Iterate through target deployment hosts
for host in ${hostsArr[@]}
do
  # Establish SSH connections to verify if the installation directory exists; creates it if missing (Requires pre-configured passwordless SSH)
  if ! ssh -o StrictHostKeyChecking=no -p $sshPort $host test -e $installPath; then
    # Create installation directory, e.g., /home/dolphinscheduler/apache-dolphinscheduler
    ssh -o StrictHostKeyChecking=no -p $sshPort $host "sudo mkdir -p $installPath; sudo chown -R $deployUser:$deployUser $installPath"
  fi

  # Identify whether the current host serves as a Worker node
  echo "scp dirs to $host/$installPath starting"
  for i in ${!workerNames[@]}; do
    if [[ ${workerNames[$i]} == $host ]]; then
      workerIndex=$i
      break
    fi
  done

  # Inject designated worker groups into application.yaml
  [[ -n ${workerIndex} ]] && sed -i "s/- default/- ${groupNames[$workerIndex]}/" $workDir/../worker-server/conf/application.yaml

  # Transfer core directories to target hosts
  for dsDir in bin master-server worker-server alert-server api-server ui tools
  do
    echo "start to scp $dsDir to $host/$installPath"
    # Use quiet mode to reduce command line output
    scp -q -P $sshPort -r $workDir/../$dsDir $host:$installPath
  done

  # Restore worker groups to default settings
  [[ -n ${workerIndex} ]] && sed -i "s/- ${groupNames[$workerIndex]}/- default/" $workDir/../worker-server/conf/application.yaml

  echo "scp dirs to $host/$installPath complete"
done

Deleting Root Nodes on ZooKeeper

Execute the cleanup command:

bash ${workDir}/remove-zk-node.sh $zkRoot

Underlying script implementation details:

print_usage(){
  printf $"USAGE: $0 rootNode\n"
  exit 1
}

# Require exactly one input argument
if [ $# -ne 1 ]; then
  print_usage
fi

# Target ZooKeeper root node path, e.g., /dolphinscheduler
rootNode=$1

# Resolve script bin directory
BIN_DIR=`dirname $0`
BIN_DIR=`cd "$BIN_DIR"; pwd`
# Resolve DolphinScheduler home path
DOLPHINSCHEDULER_HOME=$BIN_DIR/..

# Refresh environment configurations
source ${BIN_DIR}/env/install_env.sh
source ${BIN_DIR}/env/dolphinscheduler_env.sh

# Export JDK runtime path
export JAVA_HOME=$JAVA_HOME

# Define configuration and dependency paths
export DOLPHINSCHEDULER_CONF_DIR=$DOLPHINSCHEDULER_HOME/conf
export DOLPHINSCHEDULER_LIB_JARS=$DOLPHINSCHEDULER_HOME/api-server/libs/*

# JVM optimization flags and command assembly
export DOLPHINSCHEDULER_OPTS="-Xmx1g -Xms1g -Xss512k -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:LargePageSizeInBytes=128m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 "
export STOP_TIMEOUT=5

CLASS=org.apache.zookeeper.ZooKeeperMain

exec_command="$DOLPHINSCHEDULER_OPTS -classpath $DOLPHINSCHEDULER_CONF_DIR:$DOLPHINSCHEDULER_LIB_JARS $CLASS -server $REGISTRY_ZOOKEEPER_CONNECT_STRING rmr $rootNode"

cd $DOLPHINSCHEDULER_HOME
$JAVA_HOME/bin/java $exec_command

# Resolved execution footprint:
# /bin/java -Xmx1g -Xms1g -Xss512k \
#   -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC \
#   -XX:+CMSParallelRemarkEnabled -XX:LargePageSizeInBytes=128m \
#   -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 \
#   -classpath /conf:/api-server/libs/* \
#   org.apache.zookeeper.ZooKeeperMain \
#   -server localhost:2181 rmr /dolphinscheduler

Cluster Lifecycle Orchestration

# Spin up all cluster services
bash ./bin/start-all.sh

# Gracefully bring down all cluster services
bash ./bin/stop-all.sh

Startup Sequence:

Sources deployment metadata from install_env.sh for api-server, master-server, worker-server, and alert-server.
Executes dolphinscheduler-daemon.sh via SSH to initiate or terminate services on target hosts.
Enforces initialization sequence: master-server → worker-server → alert-server → api-server.
Executes status-all.sh post-launch to run health checks on active components.

Service Health & Process Monitoring

Production process tracking relies on two primary methodologies:

PID File Tracking: Writing active process IDs to discrete PID files during bootstrap (typically inside /var/run/ or custom directories).
Process Table Inspection: Querying running tasks directly via the ps CLI tool.

Apache DolphinScheduler adopts the PID file mechanism, persisting process IDs to manage service state checks and graceful shutdowns.

Single-Node Management & Health Checks

The engine underlying cluster management is dolphinscheduler-daemon.sh.

When single nodes fail unexpectedly or during horizontal cluster expansion/contraction, executing global scripts like start-all.sh is impractical. Mastering node-level execution via dolphinscheduler-daemon.sh becomes essential for targeted operations.

Command Usage Pattern:

dolphinscheduler-daemon.sh (start|stop|status) <api-server|master-server|worker-server|alert-server|standalone-server>

Core Daemon Script Breakdown

dolphinscheduler-daemon.sh implementation:

usage="Usage: dolphinscheduler-daemon.sh (start|stop|status) <api-server|master-server|worker-server|alert-server|standalone-server> "

# Validate argument count
if [ $# -le 1 ]; then
  echo $usage
  exit 1
fi

startStop=$1
shift
command=$1
shift

echo "Begin $startStop $command......"

BIN_DIR=`dirname $0`
BIN_DIR=`cd "$BIN_DIR"; pwd`
DOLPHINSCHEDULER_HOME=`cd "$BIN_DIR/.."; pwd`
BIN_ENV_FILE="${DOLPHINSCHEDULER_HOME}/bin/env/dolphinscheduler_env.sh"

# Global environment config override: Applies `bin/env/dolphinscheduler_env.sh` over individual service configurations in `<server>/conf/dolphinscheduler_env.sh`
function overwrite_server_env() {
  local server=$1
  local server_env_file="${DOLPHINSCHEDULER_HOME}/${server}/conf/dolphinscheduler_env.sh"
  if [ -f "${BIN_ENV_FILE}" ]; then
    echo "Overwrite ${server}/conf/dolphinscheduler_env.sh using bin/env/dolphinscheduler_env.sh."
    cp "${BIN_ENV_FILE}" "${server_env_file}"
  else
    echo "Start server ${server} using env config path ${server_env_file}, because file ${BIN_ENV_FILE} not exists."
  fi
}

export HOSTNAME=`hostname`
export DOLPHINSCHEDULER_LOG_DIR=$DOLPHINSCHEDULER_HOME/$command/logs
export STOP_TIMEOUT=5

if [ ! -d "$DOLPHINSCHEDULER_LOG_DIR" ]; then
  mkdir $DOLPHINSCHEDULER_LOG_DIR
fi

pid=$DOLPHINSCHEDULER_HOME/$command/pid

cd $DOLPHINSCHEDULER_HOME/$command

# Resolve service log targets
if [ "$command" = "api-server" ]; then
  log=$DOLPHINSCHEDULER_HOME/api-server/logs/$command-$HOSTNAME.out
elif [ "$command" = "master-server" ]; then
  log=$DOLPHINSCHEDULER_HOME/master-server/logs/$command-$HOSTNAME.out
elif [ "$command" = "worker-server" ]; then
  log=$DOLPHINSCHEDULER_HOME/worker-server/logs/$command-$HOSTNAME.out
elif [ "$command" = "alert-server" ]; then
  log=$DOLPHINSCHEDULER_HOME/alert-server/logs/$command-$HOSTNAME.out
elif [ "$command" = "standalone-server" ]; then
  log=$DOLPHINSCHEDULER_HOME/standalone-server/logs/$command-$HOSTNAME.out
else
  echo "Error: No command named '$command' was found."
  exit 1
fi

state=""
function get_server_running_status() {
  state="STOP"
  if [ -f $pid ]; then
    TARGET_PID=`cat $pid`
    if [[ $(ps -p "$TARGET_PID" -o comm=) =~ "bash" ]]; then
      state="RUNNING"
    fi
  fi
}

case $startStop in
  (start)
    get_server_running_status
    if [[ $state == "RUNNING" ]]; then
      echo "$command running as process $TARGET_PID. Stop it first."
      exit 1
    fi
    echo starting $command, logging to $DOLPHINSCHEDULER_LOG_DIR
    overwrite_server_env "${command}"
    nohup /bin/bash "$DOLPHINSCHEDULER_HOME/$command/bin/start.sh" > $log 2>&1 &
    echo $! > $pid
    ;;

  (stop)
    if [ -f $pid ]; then
      TARGET_PID=`cat $pid`
      if kill -0 $TARGET_PID > /dev/null 2>&1; then
        echo stopping $command
        pkill -P $TARGET_PID
        sleep $STOP_TIMEOUT
        if kill -0 $TARGET_PID > /dev/null 2>&1; then
          echo "$command did not stop gracefully after $STOP_TIMEOUT seconds: killing with kill -9"
          pkill -P -9 $TARGET_PID
        fi
      else
        echo no $command to stop
      fi
      rm -f $pid
    else
      echo no $command to stop
    fi
    ;;

  (status)
    get_server_running_status
    if [[ $state == "STOP" ]]; then
      state="[ \033[1;31m $state \033[0m ]"
    else
      state="[ \033[1;32m $state \033[0m ]"
    fi
    echo -e "$command $state"
    ;;

  (*)
    echo $usage
    exit 1
    ;;
esac

Key Highlights of the Startup Script

A critical detail in the env configuration directory lies within dolphinscheduler_env.sh, which explicitly handles database-related settings:

# Database configuration setup: define database vendor, username, and authentication credentials
export DATABASE=${DATABASE:-postgresql}
export SPRING_PROFILES_ACTIVE=${DATABASE}
export SPRING_DATASOURCE_URL
export SPRING_DATASOURCE_USERNAME
export SPRING_DATASOURCE_PASSWORD

For developers familiar with Spring Boot, Java application configurations are traditionally driven by YAML files. This approach can easily puzzle developers using DolphinScheduler for the first time.

On the official website of Spring Boot Externalized Configuration Documentation, we can see descriptions like this:

Spring Boot applies a very specific PropertySource order designed to allow sensible overriding of values. Properties are evaluated in the following order:

Devtools global settings properties in the $HOME/.config/spring-boot directory when devtools is active.
@TestPropertySource annotations on your tests.
properties attribute on your tests. Available on @SpringBootTest and the test annotations for testing a particular slice of your application.
Command line arguments.
Properties from SPRING_APPLICATION_JSON (inline JSON embedded in an environment variable or system property).
ServletConfig init parameters.
ServletContext init parameters.
JNDI attributes from java:comp/env.
Java System properties (System.getProperties()).
OS environment variables.
A RandomValuePropertySource that has properties only in random.*.
Profile-specific application properties outside of your packaged jar (application-{profile}.properties and YAML variants).
Profile-specific application properties packaged inside your jar (application-{profile}.properties and YAML variants).
Application properties outside of your packaged jar (application.properties and YAML variants).
Application properties packaged inside your jar (application.properties and YAML variants).
@PropertySource annotations on your @Configuration classes. Please note that such property sources are not added to the Environment until the application context is refreshed. This is too late to configure certain properties such as logging.* and spring.main.* which are read before refresh begins.
Default properties (specified by setting SpringApplication.setDefaultProperties).

Notice that OS environment variables are included in this list, using an uppercase and underscore-separated format. You can refer to the official Spring Boot documentation linked above for full implementation details.

Gaining a deep understanding of these internal operational scripts empowers data engineers to manage Apache DolphinScheduler with higher confidence, ensuring smoother deployments, easier troubleshooting, and greater stability in production environments.