Nomad Series - Quick Start

Series Articles

• Nomad Series Articles

Key Nomad Terminology

Nomad Installation and Setup Terminology

• agent - An agent is a Nomad process running in either Server or Client mode.
• client - A Nomad client is responsible for running the tasks assigned to it. It also registers itself with the servers and watches for any work to be assigned. When running an agent, a client can be referred to as a Node.
• server - A Nomad server manages all jobs and clients, monitors tasks, and controls which tasks are placed on which client nodes. Servers replicate data between each other to ensure high availability.
• dev_agent - A dev agent is an agent configuration that provides useful defaults for running a single-node Nomad cluster. It runs in both server and client mode and does not persist its cluster state to disk, which allows the agent to start from a clean, repeatable state without having to remove disk-based state between runs.

A Nomad cluster typically consists of three to five server agents and many client agents.

Nomad Usage Terminology

You will encounter the following terms as Nomad schedules and runs workloads.

• task - A task is the smallest unit of work in Nomad (similar to a Pod in K8s). Tasks are executed by task drivers, which include docker and exec among others, making Nomad flexible in supporting various task types. A task specifies its required task driver, driver configuration, constraints, and required resources.
• group - A group is a set of tasks that run on the same Nomad client. (I personally consider this similar to a Deployment/StatefulSet/DaemonSet/Job in K8s)
• job - A job is the core control unit in Nomad. It defines an application and its configuration, and can contain one or more tasks. (I personally consider a job similar to a collection of multiple resource YAML manifests in K8s, including: SVC/Ingress/ConfigMap/Deploy/PVC…)
• job specification - A job specification, also known as a jobspec, defines the schema of a Nomad job. It describes the job type, the tasks and resources required to run the job, job information such as which clients the job can run on, and more.
• allocation - An allocation is a mapping between a task group in a job and a client node. When a job is run, Nomad selects a client capable of running it and allocates resources on that machine for the tasks defined in the job’s task groups. (I consider this similar to a running Pod in K8s)

Applications are defined in a jobspec as groups of tasks with a jobspec, and once submitted to Nomad, a job is created along with an allocation for each group defined in that jobspec.

Application Workflow on Nomad

A typical application workflow involves several steps and begins outside of Nomad.

A prerequisite for any application running on Nomad is having a workload artifact. Nomad supports various artifacts, including Docker images, raw binaries, Java applications, and virtual machine images using QEMU.

Nomad does not create these application artifacts, but CI tools such as CircleCI, GitHub Actions, or local builds can be used to create artifacts and push them to a repository from which Nomad can retrieve them when scheduling jobs.

After the application is created, the workflow continues within Nomad.

• Create a job spec - The job specification contains the tasks required by the application, including where artifacts reside, network configuration such as ports and service definitions, the number of required instances, and more.
• Deploy the job - The job specification is submitted to Nomad, and allocations are scheduled for the job on one or more clients based on the job configuration.
• Update and redeploy the job - Update the application code or job specification, then resubmit it to Nomad for scheduling.

Quick Start

Let’s get hands-on with Nomad and walk through deploying a sample application.

Prerequisites

• Docker installed
• Nomad binary (including CLI) installed
• (Optional) CNI plugins installed
• A Nomad cluster created and running (at least 1 Server and 1 Client)

Deploy the Sample Application Job

Here we directly git clone the official Nomad demo:

git clone https://github.com/hashicorp-education/learn-nomad-getting-started.git
cd learn-nomad-getting-started
git checkout -b nomad-getting-started v1.1

The sample application runs in Docker containers and consists of a database and a web frontend that reads data from the database. You will set up the database using a parameterized batch job, then start additional short-lived jobs that write data to the database using a periodic batch job.

Job Types

Service jobs are for long-running services (similar to a Deployment in K8s) that run until explicitly stopped.

Batch jobs are short-lived jobs that run until they exit successfully (similar to Job and CronJob in K8s).

• The parameterized block allows you to configure a batch job to accept required or optional inputs. You can trigger the job using the nomad job dispatch command.
• The periodic block allows you to schedule a Nomad job to run at set times. These are also known as Nomad cron jobs (similar to CronJob in K8s).

Sample Application Pytechco Overview

The sample application pytechco simulates employees working at a tech company. They come online, complete tasks, and then sign off.

Navigate to the jobs directory in the sample repo on your local machine:

cd jobs

Each jobspec file that makes up the application sets the driver attribute to docker and specifies an image stored in GHCR in the config block using the image attribute. The Redis job uses the official Redis image hosted on Docker Hub.

Redis Service

The contents of jobs/pytechco-redis.nomad.hcl:

task "redis-task" {
  driver = "docker"
  config {
    image = "redis:7.0.7-alpine"
  }
}

pytechco-redis.nomad.hcl - This service job runs and exposes a Redis database as a Nomad service for other application components to connect to. The jobspec sets the type to service, configures a Nomad service block (similar to a Service in K8s) to use Nomad native service discovery, and creates a service named redis-svc. The full specification:

job "pytechco-redis" {
  type = "service"

  group "ptc-redis" {
    count = 1
    network {
      port "redis" {
        to = 6379
      }
    }

    service {
      name     = "redis-svc"
      port     = "redis"
      provider = "nomad"
    }

    task "redis-task" {
      driver = "docker"

      config {
        image = "redis:7.0.7-alpine"
        ports = ["redis"]
      }
    }
  }
}

Explanation:

• type = “service”: A service type job, similar to a Deployment in K8s
• group “ptc-redis”: The group block ptc-redis
• count = 1: Replica count is 1
• to = 6379: Located in the network block, indicates connecting to port 6379 inside the container (the host or bridge port is randomly assigned)
• service {: The service block, similar to a Service in K8s
• provider = “nomad”: Starting from version 1.3, Nomad added native service discovery. 🐾 Note: If not specified, it defaults to using Consul service (which requires Consul to be installed).
• task “redis-task” {: The task block, similar to a Pod in K8s
• driver = “docker”: Unlike K8s, which by default only supports one runtime (the container runtime), Nomad natively supports multiple runtimes (called drivers in Nomad terminology), such as docker, java, raw binaries, QEMU, etc. Here we specify the docker driver.
• config {: This is the configuration specific to the docker driver, including image and ports
• image = “redis:7.0.7-alpine”: The redis:7.0.7-alpine image from Docker Hub
• ports = [“redis”]: The list of exposed ports — here it’s redis, which refers to the to = 6379 defined in the network block

Web Service

pytechco-web.nomad.hcl - This service job runs the web application frontend, displaying values stored in the database and active employees. The jobspec sets the type to service and uses a static port 5000 for the application. It also uses the nomadService built-in function to retrieve the address and port information of the Redis database service.

job "pytechco-web" {
  type = "service"

  group "ptc-web" {
    count = 1
    network {
      port "web" {
        static = 5000
      }
    }

    service {
      name     = "ptc-web-svc"
      port     = "web"
      provider = "nomad"
    }

    task "ptc-web-task" {
      template {
        data        = <<EOH
{{ range nomadService "redis-svc" }}
REDIS_HOST={{ .Address }}
REDIS_PORT={{ .Port }}
FLASK_HOST=0.0.0.0
REFRESH_INTERVAL=500
{{ end }}
EOH
        destination = "local/env.txt"
        env         = true
      }

      driver = "docker"

      config {
        image = "ghcr.io/hashicorp-education/learn-nomad-getting-started/ptc-web:1.0"
        ports = ["web"]
      }
    }
  }
}

Detailed explanation (most parts are similar to the Redis job, so they won’t be repeated here):

• static = 5000: Located in the network block, specifies a static port of 5000. Unlike to, which specifies the internal port with a randomly assigned external port, static makes both the internal and external ports the same — in this case, port 5000.
• template {: The template block, functionally similar to a ConfigMap in K8s.
• data = <<EOH: The actual configuration content, which references the nomadService built-in function to retrieve the Redis database service’s address and port information. The content is enclosed within EOH.
  • REDIS_HOST={{ .Address }}: The external (typically host) address of the running allocation
  • REDIS_PORT={{ .Port }}: The external port of the allocation (not 6379) — a random port between 20000-30000.
• destination = "local/env.txt": The location of the configuration file after allocation.
• env = true: Also sets REDIS_HOST, REDIS_PORT, etc. as environment variables inside the allocation container.

Setup Parameterized Batch

pytechco-setup.nomad.hcl - This parameterized batch job sets up the database with default values. You can dispatch it multiple times to reset the database. The jobspec sets the type to batch and has a parameterized block with a meta_required attribute that requires a budget value when dispatched.

job "pytechco-setup" {
  type = "batch"

  parameterized {
    meta_required = ["budget"]
  }

  group "ptc-setup" {
    count = 1

    task "ptc-setup-task" {

      template {
        data        = <<EOH
{{ range nomadService "redis-svc" }}
REDIS_HOST={{ .Address }}
REDIS_PORT={{ .Port }}
{{ end }}
PTC_BUDGET={{ env "NOMAD_META_budget" }}
EOH
        destination = "local/env.txt"
        env         = true
      }
      driver = "docker"

      config {
        image = "ghcr.io/hashicorp-education/learn-nomad-getting-started/ptc-setup:1.0"
      }
    }
  }
}

Explanation:

• type = "batch": The job type is batch, similar to a Job or CronJob in K8s
• parameterized {: This job has a parameterized block with a meta_required attribute that requires a budget value when dispatched
• PTC_BUDGET={{ env "NOMAD_META_budget" }}: Retrieves the value of the NOMAD_META_budget environment variable and assigns it to PTC_BUDGET

Employee Periodic Batch

pytechco-employee.nomad.hcl - This periodic batch job brings employees online. It randomizes the employee’s job type and other variables such as how long they work and how quickly they complete tasks. The jobspec sets the type to batch and has a periodic block with the cron attribute set to launch a new job every 3 seconds.

job "pytechco-employee" {
  type = "batch"

  periodic {
    cron             = "0/3 * * * * * *"
    prohibit_overlap = false
  }

  group "ptc-employee" {
    count = 1

    task "ptc-employee-task" {

      restart {
        attempts = 2
        delay    = "15s"
        interval = "10s"
        mode     = "fail"
      }

      template {
        data        = <<EOH
{{ range nomadService "redis-svc" }}
REDIS_HOST={{ .Address }}
REDIS_PORT={{ .Port }}
PTC_EMPLOYEE_ID={{ env "NOMAD_SHORT_ALLOC_ID"}}
{{ end }}
EOH
        destination = "local/env.txt"
        env         = true
      }
      driver = "docker"

      config {
        image = "ghcr.io/hashicorp-education/learn-nomad-getting-started/ptc-employee:1.0"
        // args = [
        //     "--employee-type", "sales_engineer"
        // ]
      }
    }
  }
}

Explanation:

• periodic {: Combined with type = "batch", this makes it similar to a CronJob in K8s
• restart {: The restart policy configuration for handling failures during batch execution.
• PTC_EMPLOYEE_ID={{ env "NOMAD_SHORT_ALLOC_ID"}}: Assigns the NOMAD_SHORT_ALLOC_ID environment variable (generated by Nomad, similar to a Pod name in K8s) to PTC_EMPLOYEE_ID
• // args = [: // is a comment in HCL, so this line has no effect. args is a configuration option within the docker driver’s config block.

Starting from Nomad 1.5.0, the datacenter attribute defaults to all available datacenters in the cluster (“*”). Therefore, the job specifications in this tutorial omit this attribute since the default is sufficient. If you are running an earlier version of the Nomad CLI and cluster binaries, you will need to include and set this attribute.

Deploy the Application

The specific commands are as follows:

Submit the database job:

$ nomad job run pytechco-redis.nomad.hcl
==> 2023-03-10T12:16:09-05:00: Monitoring evaluation "d44af37b"
    2023-03-10T12:16:09-05:00: Evaluation triggered by job "pytechco-redis"
    2023-03-10T12:16:10-05:00: Evaluation within deployment: "0ea05651"
    2023-03-10T12:16:10-05:00: Allocation "be0bda79" created: node "2c9f4b7e", group "ptc-redis"
    2023-03-10T12:16:10-05:00: Evaluation status changed: "pending" -> "complete"
==> 2023-03-10T12:16:10-05:00: Evaluation "d44af37b" finished with status "complete"
==> 2023-03-10T12:16:10-05:00: Monitoring deployment "0ea05651"
  ✓ Deployment "0ea05651" successful
    
    2023-03-10T12:16:24-05:00
    ID          = 0ea05651
    Job ID      = pytechco-redis
    Job Version = 0
    Status      = successful
    Description = Deployment completed successfully
    
    Deployed
    Task Group  Desired  Placed  Healthy  Unhealthy  Progress Deadline
    ptc-redis   1        1       1        0          2023-03-10T17:26:23Z

Submit the web job:

nomad job run pytechco-web.nomad.hcl

Get the webapp’s IP address. The following command retrieves the allocation ID of the web job, uses that ID to get the allocation status, searches for the IP address in the allocation status output, and formats the IP address as a link with the webapp port. Open the URL from the output in your browser to view the webapp frontend.

nomad node status -verbose \
    $(nomad job allocs pytechco-web | grep -i running | awk '{print $2}') | \
    grep -i ip-address | awk -F "=" '{print $2}' | xargs | \
    awk '{print "http://"$1":5000"}'

Submit the setup job:

nomad job run pytechco-setup.nomad.hcl

Dispatch the setup job by providing a budget value:

$ nomad job dispatch -meta budget="200" pytechco-setup
Dispatched Job ID = pytechco-setup/dispatch-1678468734-396cfa83
Evaluation ID     = 53a77034

==> 2023-03-10T12:18:54-05:00: Monitoring evaluation "53a77034"
    2023-03-10T12:18:54-05:00: Evaluation triggered by job "pytechco-setup/dispatch-1678468734-396cfa83"
    2023-03-10T12:18:54-05:00: Allocation "d6c60ffd" created: node "2c9f4b7e", group "ptc-setup"
    2023-03-10T12:18:54-05:00: Evaluation status changed: "pending" -> "complete"
==> 2023-03-10T12:18:54-05:00: Evaluation "53a77034" finished with status "complete"

Submit the employee job:

nomad job run pytechco-employee.nomad.hcl

Navigate to the Nomad UI, click the “Jobs” page, then click the pytechco-employee job. Since this is a cron batch job, you can see it creating a new job every 3 seconds.

Navigate back to the webapp URL. The result looks like this:

Update the Application

First, stop the employee job:

nomad job stop -purge pytechco-employee

Reset the database by dispatching the setup job again with a new budget of 500:

$ nomad job dispatch -meta budget="500" pytechco-setup
Dispatched Job ID = pytechco-setup/dispatch-1678469008-6ffe1c0c
Evaluation ID     = e8e420f0

==> 2023-03-10T12:23:28-05:00: Monitoring evaluation "e8e420f0"
    2023-03-10T12:23:28-05:00: Evaluation triggered by job "pytechco-setup/dispatch-1678469008-6ffe1c0c"
    2023-03-10T12:23:28-05:00: Allocation "74acf63e" created: node "2c9f4b7e", group "ptc-setup"
    2023-03-10T12:23:28-05:00: Evaluation status changed: "pending" -> "complete"
==> 2023-03-10T12:23:28-05:00: Evaluation "e8e420f0" finished with status "complete"

Open the pytechco-employee.nomad.hcl file and uncomment the args block in the task configuration so the job only creates employees of type sales_engineer. Then update the cron to 0/1 * * * * * * so the job runs every second. Save the file.

...
        cron = "0/1 * * * * * *"
...

        args = [
            "--employee-type", "sales_engineer"
        ]
...

Submit the employee job again and navigate to the web application. Notice that all online employees are now only sales engineers (sales_engineer) — previously there were multiple different employee types — and more of them are online at any given time.

$ nomad job run pytechco-employee.nomad.hcl
Job registration successful
Approximate next launch time: 2023-03-10T17:24:18Z (1s from now)

Clean Up the Application

Stop and purge all jobs:

nomad job stop -purge pytechco-employee
nomad job stop -purge pytechco-web
nomad job stop -purge pytechco-redis
nomad job stop -purge pytechco-setup

Summary

In this article, we explored Nomad’s terminology and deployed and updated Nomad jobs using the official demo. Along the way, we drew comparisons and contrasts between Nomad and K8s.

In upcoming articles, we’ll dive into hands-on topics including exclusive content on integrating Nomad with Traefik and Tailscale. Stay tuned! 💪💪💪

📚️ References

• Introduction to Nomad
• Deploy and Update a Job