Nomad Series - Nomad Networking Models

This article was last updated on: May 17, 2026 am

Series Articles

Overview

Nomad’s networking differs significantly from Docker’s, and even more so from Kubernetes’. Additionally, different Nomad versions (before and after Nomad 1.3) and whether components like Consul or CNI are integrated can lead to very different networking models. This article provides a detailed overview of Nomad’s main networking models.

Before Nomad 1.3 was released, it had no built-in support for discovering other applications running in the cluster. This is a very fundamental requirement when scheduling tasks in a cluster. Nomad relied on Consul to discover other “services” and provided first-class support for registering and retrieving service records, which made things easier. Consul provides records through various mechanisms such as REST API, DNS, and Consul templates, which render the exact IP/port of services in Go templates that can be injected into applications.

One of the challenges in learning Nomad is that it is often run alongside Consul. In that case, a major learning curve is that you must first understand how Consul works, deploy a Consul cluster, and mastering two pieces of software simultaneously is quite difficult. Nomad 1.3 addressed part of this problem (i.e., basic service discovery without running Consul), making it ideal for getting started with Nomad-based networking.

Scenario 1: Exposing an Application on the Host

Host Dynamic Port

Starting with the simplest use case: you have a Redis container and want to expose it to the host. The equivalent docker run command would be:

1
docker run --rm -p=6379 redis

This command exposes a dynamic port on the host. To find out the actual port number, you can run docker ps and find output similar to 0.0.0.0:49153->6379/tcp under PORTS.

1
2
3
$ redis-cli -p 49153                
127.0.0.1:49153> ping
PONG

So how do we achieve the same thing in Nomad?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
job "redis" {
  type        = "service"

  group "redis" {
    network {
      mode = "host"
      port "redis" {
        to = 6379
      }
    }

    task "redis" {
      driver = "docker"

      config {
        image = "redis"
        ports = ["redis"]
      }
    }
  }
}

In just a few lines of configuration, we have a running Docker container that exposes a dynamic port 30627:

Nomad Redis Job Map Port

We can connect to it using redis-cli on the host:

1
2
3
$ redis-cli -p 30627
127.0.0.1:30627> ping
PONG

│ 🐾Warning:
│
│ It is important to have ports in the task.config section. Nomad passes this information to the Docker daemon running on the host. So unless you specify which ports to advertise in the container, it won’t know whether to expose 6379.

Exposing a Static Port

A less common scenario is binding an application to a static port on the host. Simply add a static line in the port block:

1
2
3
4
5
network {
  port "redis" {
    static = 6379
  }
}

Host Static Port

When we deploy the same file again, we can see that the port allocation has changed from a dynamic port to the static port we assigned. However, make sure no other application is listening on the same interface and port, otherwise a conflict will inevitably occur.

A typical use case for static ports is Ingress. For example, Traefik can listen on static ports 80 and 443.

Scenario 2: Communicating with Redis Within the Same Group

For this scenario, assume we have an application that needs to communicate with Redis. In this case, Redis is used as a temporary cache, so we can deploy them in the same Group.

A Group can contain multiple Tasks. The important thing to know here is that the same Group will always have its own shared network namespace (similar to how multiple Containers in a Kubernetes Pod share a network namespace). This means that if you have 2 Tasks in a Group, they can both access the same network namespace. This allows the two Tasks to communicate with each other on the same network interface.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
job "hello" {
  type        = "service"

  group "app" {
    network {
      mode = "host"
      port "app" {
        static = 8080
      }
      port "redis" {
        static = 6379
      }
    }

    task "redis" {
      driver = "docker"

      config {
        network_mode = "host"
        image        = "redis"
        ports        = ["redis"]
      }
    }


    task "app" {
      driver = "docker"
      env {
        DEMO_REDIS_ADDR = "${NOMAD_ADDR_redis}"
      }

      config {
        network_mode = "host"
        image        = "mrkaran/hello-app:1.0.0"
        ports        = ["app"]
      }
    }
  }
}

Here’s a detailed explanation:

  • You can see that we defined task app and task redis under the same Group. This means Nomad will co-locate these two Tasks on the same client (because they not only tend to share the same network namespace, but also share a common allocation directory — making it very easy to share files across tasks).
  • We use NOMAD_ADDR_redis to get the IP:Port combination of the redis task. This is injected at runtime by Nomad. You can find the list of runtime environment variables here.
  • This is ideal for quick test/development setups where you don’t want to deal with service discovery and want to connect to your application with minimal overhead.

If you are migrating from a docker-compose based environment, the above configuration is a great fit (though the implementation is different — Nomad leverages host networking). You can use this template for your services. The biggest limitation of this approach is that it uses host networking.

Scenario 3: Communicating Across Different Groups

As mentioned above, the same Group is useful if you have related Tasks (like an init task where you want to fetch files before the task starts, similar to a Kubernetes Pod’s init container). However, the downside of using a Group is that you cannot scale Tasks independently. In the example above, we placed Redis and App in the same Group, but this means if you increase the Group’s count to scale the app, you’ll end up scaling the Redis container as well. This is undesirable because Redis may not need to scale proportionally with the application.

The approach for creating multiple Groups is to split tasks into their own groups:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
job "hello" {
  type        = "service"

  group "app" {
    count = 1

    network {
      mode = "host"
      port "app" {
        static = 8080
      }
    }

    task "app" {
      driver = "docker"
      env {
        DEMO_REDIS_ADDR = "localhost:6379"
      }

      config {
        image = "mrkaran/hello-app:1.0.0"
        ports = ["app"]
      }
    }
  }

  group "redis" {
    count = 1

    network {
      mode = "host"
      port "redis" {
        static = 6379
      }
    }

    task "redis" {
      driver = "docker"

      config {
        image = "redis"
        ports = ["redis"]
      }
    }
  }
}

After submitting this Job, you will get 2 allocation IDs (each Group creates one alloc). The key point here is that both Groups have their own network namespace. Therefore, we actually have no way to access other applications (we can’t rely on host networking as above, because there’s no guarantee that both Groups will be deployed on the same node).

Now since the groups are separate, the app container doesn’t know about redis (and vice versa):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
env | grep NOMAD
NOMAD_REGION=global
NOMAD_CPU_LIMIT=4700
NOMAD_IP_app=127.0.0.1
NOMAD_JOB_ID=hello
NOMAD_TASK_NAME=app
NOMAD_SECRETS_DIR=/secrets
NOMAD_CPU_CORES=1
NOMAD_NAMESPACE=default
NOMAD_ALLOC_INDEX=0
NOMAD_ALLOC_DIR=/alloc
NOMAD_JOB_NAME=hello
NOMAD_HOST_IP_app=127.0.0.1
NOMAD_SHORT_ALLOC_ID=a9da72dc
NOMAD_DC=dc1
NOMAD_ALLOC_NAME=hello.app[0]
NOMAD_PORT_app=8080
NOMAD_GROUP_NAME=app
NOMAD_PARENT_CGROUP=nomad.slice
NOMAD_TASK_DIR=/local
NOMAD_HOST_PORT_app=8080
NOMAD_MEMORY_LIMIT=512
NOMAD_ADDR_app=127.0.0.1:8080
NOMAD_ALLOC_PORT_app=8080
NOMAD_ALLOC_ID=a9da72dc-94fc-6315-bb37-63cbeef153b9
NOMAD_HOST_ADDR_app=127.0.0.1:8080

Service Discovery

The app Group needs to discover redis before connecting to it. There are multiple ways to do this, but we’ll cover two of the more common standard approaches.

Using Nomad Native Service Discovery

Nomad Native Service Discovery

This is a feature introduced in Nomad 1.3. Before this release, Nomad had to rely on Consul for this task. But with native service discovery built into Nomad, things are much simpler. Let’s make the following changes to the job file. In each Group, we’ll add a service definition:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
group "app" {
  count = 1

  network {
    mode = "host"
    port "app" {
      to = 8080
    }
  }

  service {
    name     = "app"
    provider = "nomad"
    port     = "app"
  }
  // task is the same
}

group "redis" {
  count = 1

  network {
    mode = "host"
    port "redis" {
      to = 6379
    }
  }

  service {
    name     = "redis"
    provider = "nomad"
    port     = "redis"
  }
  // task is the same
}

As shown above, we added a new service block and removed the static ports. When using service discovery, there’s no need to bind to static ports.

After submitting the job, we can use the nomad service list command to verify that the services are registered with Nomad.

1
2
3
4
nomad service list    
Service Name  Tags
app           []
redis         []

To get details about a specific service, we can use nomad service info:

1
2
3
4
5
6
$ nomad service info app      
Job ID  Address          Tags  Node ID   Alloc ID
hello   127.0.0.1:29948  []    d92224a5  5f2ac51f
$ nomad service info redis
Job ID  Address          Tags  Node ID   Alloc ID
hello   127.0.0.1:22300  []    d92224a5  8078c9a6

As shown above, we can see the dynamic port allocation for each service. To use this configuration in our application, we templatize it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
    task "app" {
      driver = "docker"

      template {
        data = <<EOH
{{ range nomadService "redis" }}
DEMO_REDIS_ADDR={{ .Address }}:{{ .Port }}
{{ end }}
EOH

        destination = "secrets/config.env"
        env         = true
      }

      config {
        image = "mrkaran/hello-app:1.0.0"
        ports = ["app"]
      }
    }

We added a template stanza that injects environment variables into the container. We iterate over nomadService and retrieve the address and port of the redis service. This allows tasks on different nodes to conveniently discover each other.

Using Consul Service Discovery

Consul Service Discovery

By simply adjusting the provider in the service block, we can use the Consul agent for service discovery.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
    service {
      name     = "app"
      provider = "consul"
      port     = "app"
    }


    task "app" {
      driver = "docker"

      template {
        data = <<EOH
{{ range service "redis" }}
DEMO_REDIS_ADDR={{ .Address }}:{{ .Port }}
{{ end }}
EOH

│ 🐾Warning:
│
│ Note that range nomadService has also changed to range service.

The prerequisite is to ensure that Consul is running and Nomad is connected to it. Refer to this documentation for details.

Everything else remains largely the same. With just two lines of code, you can switch between Nomad and Consul for service discovery.

Additionally, using Consul provides more advantages:

  • You can query service addresses using DNS:
1
2
3
doggo redis.service.consul @tcp://127.0.0.1:8600
NAME                    TYPE    CLASS   TTL ADDRESS     NAMESERVER     
redis.service.consul.   A       IN      0s  172.20.10.3 127.0.0.1:8600
  • Accessible by applications outside of Nomad. If Consul is used by other applications outside the Nomad cluster, they can still obtain the corresponding addresses (using DNS or REST API).

Of course, Nomad Native Service Discovery is ideal for local/edge environment setups, or even smaller production use cases, since it eliminates the need for Consul!

Scenario 4: Restricting Access to Certain Namespaces

Consul Service Mesh

In all the scenarios above, we found that services are exposed to the local Nomad client. If you run multiple Namespaces on your cluster, you may not want to expose them at all. Furthermore, you may want to express fine-grained control over which applications can access specific services. All of this can be achieved through a service mesh. Nomad provides a way to establish a “service mesh” via Consul Connect. Consul Connect enables mTLS and service authorization. Under the hood, it’s an Envoy proxy (or sidecar) that runs alongside your application. The Consul agent configures the Envoy configuration for you, so it’s all very seamless.

To achieve this, the first thing we need is the bridge network mode. This network mode is actually a CNI plugin that needs to be installed separately in /opt/cni/bin. Follow the steps mentioned here:

1
2
3
4
5
6
network {
  mode = "bridge"
  port "redis" {
    to = 6379
  }
}

The service in Redis is invoked by Consul Connect Ingress:

1
2
3
4
5
6
7
8
service {
  name     = "redis"
  provider = "consul"
  port     = "6379"
  connect {
    sidecar_service {}
  }
}

This is an empty block because we don’t need to define any upstreams here. The remaining values will use defaults.

Next, we create a service for app, which is a Consul Connect Egress:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
service {
  name     = "app"
  provider = "consul"
  port     = "app"
  connect {
    sidecar_service {
      proxy {
        upstreams {
          destination_name = "redis"
          local_bind_port  = 6379
        }
      }
    }
  }
}

Here we define an upstream for redis. When app wants to communicate with redis, it talks to localhost:6379, which is the local port where the Envoy sidecar is listening. We can verify this using netstat:

1
2
3
4
5
6
7
$ netstat -tulpvn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.2:19001         0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:23237           0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      -                   
tcp6       0      0 :::8080                 :::*                    LISTEN      1/./hello.bin

Traffic is sent from this port to the other Envoy proxy on its advertised port (automatically configured by Consul). That Envoy proxy then forwards the traffic to the redis container on port 6379. The proxied traffic is secured with mTLS encryption and authorized (via Consul Intentions — not covered in this article).

Scenario 5: Exposing Services to End Users

LB + Ingress

In the first scenario, we discussed how to use static ports. It turns out this is very useful if you want to define a Traffic Ingress service. Unlike Kubernetes, Nomad doesn’t have any Ingress Controller, so the best approach is to deploy these web proxies as a system job on every node (which ensures it runs on every client node) and bind them to static ports (such as 443/80). Then, configure a load balancer and register all Nomad nodes as target IPs, with the port being the static port you defined. These Ingress proxies (such as Traefik/Nginx) can communicate with your applications through any of the patterns mentioned above.

│ 📝Notes:
│
│ In the previous article, we did not configure a load balancer in front of all Traefik instances.
│ Instead, we directly accessed the Traefik 80/443 ports on a specific node.

Typically, you want to use a “host-based” routing pattern for the ingress proxy to make routing decisions.

For example, if you have an a.example.org DNS record pointing to an ALB. When a request reaches the ALB, it forwards to any one of the Traefik/NGINX instances. For NGINX to correctly route traffic to the a service, you can use the “Host” header.

Summary

These are some of the common networking patterns I’m aware of. Since some of these concepts are not very straightforward, I hope the explanations helped bring some clarity.

There’s much more to this topic, such as Consul Gateway and various CNI plugins that can fine-tune the underlying networking details in a cluster, but these are quite advanced topics that are beyond the scope of this article. We may explore them in future articles.

📚️References