Host discovery

Herd has a pluggable infrastructure for finding hosts in many places, from flat files to cloud provider API’s. To make herd find hosts, you can configure which providers to use and how to use them. This section of the documentation explains which providers there are and how to configure them.

Magic providers

As you could see in the getting started documentation, Herd can even find hosts if you do not configure it. It does this with so called magic providers, that detect hosts based on information on your filesystem or in your environment. For example, the known_hosts provider knows where known_hosts files usually live and will load them automatically if the exist. If you do not want to any magic providers to find hosts, you can use the --no-magic-provider command line flag to skip this.

Provider configuration

The provider section of the configuration looks as like the following in your config.yaml:

Providers:
  consul:
    provider: cache
    lifetime: 1h
    source:
      provider: consul
      timeout: 10s
      address: http://consul.service.consul:8500
  aws-main:
    provider: cache
    lifetime: 8h
    source:
      provider: aws
      accesskeyid: AK1............
      secretaccesskey: SE1..............
      excluderegions: [eu-south-1]
  aws-backup:
    provider: cache
    lifetime: 8h
    source:
      provider: aws
      accesskeyid: AK2............
      secretaccesskey: SE2..............
      excluderegions: [eu-south-1]
  tailscale:
    provider: tailscale
    domain: tail.example.com
  seveas:
    provider: seveas
    prefix: 'svs:'

As you can see, the name of a provider section foes not imply which provider is used, you have to specify the. As a result of this, you can easily use multiple instances of the same provider type, and can wrap a provider in a cache provider without affecting other parts of the configuration.

If you want to temporarily disable a provider in your configuration, you can add enabled: false to its parameters.

Host attributes

Providers do not just return a list of host names. The power of Herd is that it can filter by any attribute that providers return. For example, the AWS provider returns instance types or vpc id’s. When using multiple providers, sometimes the names of these attributes can collide. To prevent this, each provider supports a prefix parameter. This prefix will be prepended to all attribute names coming from this provider. In our example above, the seveas provider uses this parameter to add a svs: prefix to all attribute names.

Caching

Some providers can take quite a while to return data. Herd has built-in caching, implemented as a separate provider. The cache provider accepts the following parameters:

The cache provider is not only useful for caching, it also helps with fault tolerance. If a source provider fails to load, the cache provider will use cached data. If a providerd partially fails to load, such as when the consul provider can load data from certain datacenters but not all, the cache will complement the fresh data with cached data from the failed parts.

Custom providers

If you have your own inventory database or API, you can plug this into herd in two ways:

• Regularly export the data to an inventory file
• Write a custom provider to query your infrastructure database or API

Infrastructure files are explained a little lower, custom providers have their own section in the documentation. It takes only a little bit of Go code to write a custom provider.

Built-in provider reference

SSH known_hosts

This provider is loaded by default and does not require configuration. It will load ~/.ssh/known_hosts and /etc/ssh/ssh_known_hosts. If you want it to load other files, you can use the files parameter in the configuration.

This provider provides no attributes.

PuTTY

This provider only works on Windows. It looks in the registry to find host keys and host configurations. It returns all hosts it finds.

This provider takes no parameters and provides no host attributes.

Inventory files

As a simple way of integrating not-yet-supported data sources with herd, you can use the plain or json providers. The plain provider looks for a file named ${datadir}/inventory and parses that as a newline-separated list of hostnames. The json provider looks for ${datadir}/inventory and parses that as a list of host objects. The format of that file is quite simple, as can be seen in the following example:

[
  {
    "Name": "host-1.example.com",
    "Address": "10.0.0.1",
    "Attributes": {
      "application": "frontend",
      "owner": "seveas",
    }
  },
  {
    "Name": "host-1.example.com",
    "Address": "10.0.0.1",
    "Attributes": {
      "application": "database",
      "owner": "not-seveas",
    }
  }
]

In these host objects, addresses are optional and there can be as many attributes as you want. Attribute values can be of any type.

These providers take the following parameters:

The plain provider provides no host attributes, the json provider provides the attributes from the json file.

HTTP API

The HTTP API provider is not the most useful one on its own, unless your http API happens to provider inventory data in the exact format the json provider expects. Its use is mostly in embedding in custom providers that use HTTP APIs.

This provider, and providers that embed it, accept the following parameters:

An example of headers configuration:

Providers:
  api:
    provider: http
    Url: https://hosts.example.com/all
    Headers:
      Accept: application/json
      Authorization: Bearer secret-token-123456

Consul

The consul provider finds hosts in all datacenters in consul. If the name consul.service.consul resolves, or the environment variable CONSUL_HTTP_ADDR is set, the consul provider is automatically configured.

This provider accepts the following parameters:

This provider provides the following host attributes

Prometheus

If you monitor your hosts with prometheus, you can tell herd to find the hosts there. You can point it to the job(s) that do the node_exporter checks for easiest results. This provider uses the HTTP API provider and accepts the same parameters. The url should point to the prometheus targets api.

This provider accepts the following parameters:

This provider provides the following host attributes

Puppet

The puppet provider queries puppetdb for host facts and adds those as host attributes. It also populates Herd’s internal host key database, complementing keys forund in your known_hosts and/or PuTTY. It embeds the HTTP providerd and accepts the same parameters, plus its own.

This provider accepts the following parameters:

Puppet facts are usually nested, but Herd does not support this nesting, so the lists get flattened, with keys separated with :. For example, the os fact usually looks like this:

$ facter os
{
  architecture => "amd64",
  distro => {
    codename => "stretch",
    description => "Debian GNU/Linux 9 (stretch)",
    id => "Debian",
    release => {
      full => "9.13",
      major => "9",
      minor => "13"
    }
  },
  family => "Debian",
  hardware => "x86_64",
  name => "Debian",
  release => {
    full => "9.13",
    major => "9",
    minor => "13"
  },
  selinux => {
    enabled => false
  }
}

The attributes that herd then populates are:

$ herd list --no-refresh --template '{{ .Attributes|yaml }}' server-1.example.com | grep '^os:'
os:architecture: amd64
os:distro:codename: stretch
os:distro:description: Debian GNU/Linux 9 (stretch)
os:distro:id: Debian
os:distro:release:full: "9.13"
os:distro:release:major: "9"
os:distro:release:minor: "13"
os:family: Debian
os:hardware: x86_64
os:name: Debian
os:release:full: "9.13"
os:release:major: "9"
os:release:minor: "13"
os:selinux:enabled: false

AWS

If you use AWS EC2, herd can query its API to get your hosts’ information. You will need an access key id and its associated secret, and the provider exposes a lot of EC2 attributes, such as the owner id, state or private IP.

This provider accepts the following parameters:

This provider also has a magic provider. If AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY is set in the environment, it is used as IAM access key ID, and if AWS_SECRET_ACCESS_KEY or AWS_SECRET_KEY is set in the environment, it is uses as IAM secret access key.

This provider provides the following host attributes. All host tags are also added as attributes.

Azure

This provider retrieves host data from azure. You will need to provide a subscription, tenant and client identifiers and a set of credentials. If you do not provide credentials, herd will try to read them from the environment or from MSI.

This provider accepts the following parameters:

Only one set of credentials (client secret, certificate or username/password) is needed.

This provider provides the following host attributes. All host tags are also added as attributes.

Google cloud

This provider accepts the following parameters:

This provider provides the following host attributes. All host labels are also added as attributes.

TransIP

For VPS’es hosted at TransIP, herd can find information with this provider. You will need to enable API access for your account in the API controlpanel and generate an API key for herd to use.

This provider accepts the following parameters:

This provider provides the following host attributes:

Tailscale

Herd can autodetect all the hosts in your tailnet. This provider is not yet included by default, as it does not build on Windows at this time. It can be installed as a provider plugin.

$ go install github.com/seveas/herd/cmd/herd-provider-tailscale@latest

This provider accepts the following parameters

This provider provides the following host attributes: