NATS Logo by Example

Leafnode with JWT Auth in Topologies

This example demonstrates how to bootstrap the decentralized JWT authentication mode using the nsc tool, configuring a server to accept leaf node connections, configuring a server acting as the leaf node itself, and how a client making a request to the leaf node will get handled by a service connected to the main node.

CLI Go Python Deno Node Rust C# Java Ruby Elixir C
Jump to the output or the recording
$ nbe run topologies/leafnode-jwt/cli
View the source code or learn how to run this example yourself

Code

#!/bin/sh


set -xeuo pipefail

For this example, we are going to have a service connected to the main server and then another client send a request via a connection to the leaf node.

NATS_MAIN_URL="nats://0.0.0.0:4222"
NATS_LEAF_URL="nats://0.0.0.0:4223"

Create the operator, generate a signing key (which is a best practice), and initialize the default SYS account and sys user.

nsc add operator --generate-signing-key --sys --name local

A follow-up edit of the operator enforces signing keys are used for accounts as well. Setting the server URL is a convenience so that it does not need to be specified with call nsc push.

nsc edit operator --require-signing-keys \
  --account-jwt-server-url "$NATS_MAIN_URL"

Next we need to create an account intended for application usage. The SYS account should be used for operational purposes. These commands create the APP account, generates a signing key, and then creates a user named user.

nsc add account APP
nsc edit account APP --sk generate
nsc add user --account APP user

Check out the current settings of nsc.

nsc env

The nats CLI provides a way to manage different contexts by name. Here we define the server and the credentials (via nsc integration) (notice the operator/account/user hierarchy). We save two one for the main server and one for the leaf node. Note how didn’t provide credentials for the leaf node..

nats context save main-user \
  --server "$NATS_MAIN_URL" \
  --nsc nsc://local/APP/user 


nats context save main-sys \
  --server "$NATS_MAIN_URL" \
  --nsc nsc://local/SYS/sys


nats context save leaf-user \
  --server "$NATS_LEAF_URL"

This command generates the bit of configuration to be used by the server to setup the embedded JWT resolver.

nsc generate config --nats-resolver --sys-account SYS > resolver.conf

Create the most basic server config which enables leaf node connections and include the JWT resolver config.

echo 'Creating the main server conf...'
cat <<- EOF > main.conf
port: 4222
leafnodes: {
  port: 7422
}


include resolver.conf
EOF

The second config is for the leaf node itself. This needs to define the leaf node remotes which is the main server it will be connecting to.

echo 'Creating the leaf node conf...'
cat <<- EOF > leaf.conf
port: 4223
leafnodes: {
  remotes: [
    {
      url: "nats-leaf://0.0.0.0:7422",
      credentials: "$NKEYS_PATH/creds/local/APP/user.creds"
    }
  ]
}
EOF

Start the main server first.

nats-server -c main.conf 2> /dev/null &
MAIN_PID=$!


sleep 1

We need to put up the APP account JWT we created previously to the main server so that the user credentials file used both for the client providing the service and the client making the request is trusted.

echo 'Pushing the account JWT...'
nsc push -a APP

Now we can start the leaf node which uses the credentials for the remote connection.

nats-server -c leaf.conf 2> /dev/null &
LEAF_PID=$!


sleep 1

Connecting directly to the main server with the user creds, we can create a simple service that will reply to any request published to greet with the text hello. This is put in the background since this will block while serving.

nats --context main-user reply 'greet' 'hello' &
SERVICE_PID=$!

Tiny sleep to ensure the service is connected.

sleep 1

For this CLI invocation, we connect to the leaf server and send a request to greet. Notice two things, one is that no credentials file is specified since the leaf server does not have authentication setup. Instead the leafnodes.remotes section of the config defines the main server and provides the credentials so that the leaf node is authenticated for forwarding messaging. This request will be transparently fullfilled by the service connected to the main server.

nats --context leaf-user request 'greet' ''

Finally stop the service and servers.

kill $SERVICE_PID
kill $LEAF_PID
kill $MAIN_PID

Output

Network 65af27f1_default  Creating
Network 65af27f1_default  Created
             _             _               
 _ __   __ _| |_ ___      | |__   _____  __
| '_ \ / _` | __/ __|_____| '_ \ / _ \ \/ /
| | | | (_| | |_\__ \_____| |_) | (_) >  < 
|_| |_|\__,_|\__|___/     |_.__/ \___/_/\_\
                                           
nats-box v0.12.0
+ NATS_MAIN_URL=nats://0.0.0.0:4222
+ NATS_LEAF_URL=nats://0.0.0.0:4223
+ nsc add operator --generate-signing-key --sys --name local
[ OK ] generated and stored operator key "OBA4MEJE33O7FM2IHCHSPDK362OP3EQ7U54UFM6UB3VIMWBPZMNPUC7N"
[ OK ] added operator "local"
[ OK ] When running your own nats-server, make sure they run at least version 2.2.0
[ OK ] created operator signing key: OBGXY46O6M7D5MPR75SLTK5AWN5P2R34SZFAYWWIEURNDIREIULAD22S
[ OK ] created system_account: name:SYS id:ADMPNQC7QE2LFABYBGEA7AEN37O2T5Z26GQPAVNB4P43RGF3KDAGFYIW
[ OK ] created system account user: name:sys id:UAOJIINY45JT5JLI7FYWRX5HEBBFXVOVOIVTKM5MFH2T2VSAFLH4DJSZ
[ OK ] system account user creds file stored in `/nsc/nkeys/creds/local/SYS/sys.creds`
+ nsc edit operator --require-signing-keys --account-jwt-server-url nats://0.0.0.0:4222
[ OK ] strict signing key usage set to: true
[ OK ] set account jwt server url to "nats://0.0.0.0:4222"
[ OK ] edited operator "local"
+ nsc add account APP
[ OK ] generated and stored account key "ADBFUJSUHBMTN67HZJD327RCTLRGIYKSTWAZIUXS5U2H3CDGW5FCG7T7"
[ OK ] added account "APP"
+ nsc edit account APP --sk generate
[ OK ] added signing key "AB65VFZBW4RN34LQCHICB3BTGQ6YVDAC653CAP2S4OIWHEUU7KQ444MI"
[ OK ] edited account "APP"
+ nsc add user --account APP user
[ OK ] generated and stored user key "UDEMHMTMMGCNTKAK5T532WGAMEDHBFF77CPDRSTHYV7ZIDMY2Y6HQN3E"
[ OK ] generated user creds file `/nsc/nkeys/creds/local/APP/user.creds`
[ OK ] added user "user" to account "APP"
+ nsc env
+----------------------------------------------------------------------------------------------------------+
|                                             NSC Environment                                              |
+--------------------+-----+-------------------------------------------------------------------------------+
| Setting            | Set | Effective Value                                                               |
+--------------------+-----+-------------------------------------------------------------------------------+
| $NSC_CWD_ONLY      | No  | If set, default operator/account from cwd only                                |
| $NSC_NO_GIT_IGNORE | No  | If set, no .gitignore files written                                           |
| $NKEYS_PATH        | Yes | /nsc/nkeys                                                                    |
| $NSC_HOME          | No  | /nsc/.config/nats/nsc                                                         |
| $NATS_CA           | No  | If set, root CAs in the referenced file will be used for nats connections     |
|                    |     | If not set, will default to the system trust store                            |
| $NATS_KEY          | No  | If set, the tls key in the referenced file will be used for nats connections  |
| $NATS_CERT         | No  | If set, the tls cert in the referenced file will be used for nats connections |
+--------------------+-----+-------------------------------------------------------------------------------+
| From CWD           |     | No                                                                            |
| Default Stores Dir |     | /nsc/nats/nsc/stores                                                          |
| Current Store Dir  |     | /nsc/nats/nsc/stores                                                          |
| Current Operator   |     | local                                                                         |
| Current Account    |     | APP                                                                           |
| Root CAs to trust  |     | Default: System Trust Store                                                   |
+--------------------+-----+-------------------------------------------------------------------------------+

+ nats context save main-user --server nats://0.0.0.0:4222 --nsc nsc://local/APP/user
NATS Configuration Context "main-user"

      Server URLs: nats://0.0.0.0:4222
      Credentials: /nsc/nkeys/creds/local/APP/user.creds (OK)
       NSC Lookup: nsc://local/APP/user
             Path: /nsc/.config/nats/context/main-user.json

+ nats context save main-sys --server nats://0.0.0.0:4222 --nsc nsc://local/SYS/sys
NATS Configuration Context "main-sys"

      Server URLs: nats://0.0.0.0:4222
      Credentials: /nsc/nkeys/creds/local/SYS/sys.creds (OK)
       NSC Lookup: nsc://local/SYS/sys
             Path: /nsc/.config/nats/context/main-sys.json

+ nats context save leaf-user --server nats://0.0.0.0:4223
NATS Configuration Context "leaf-user"

      Server URLs: nats://0.0.0.0:4223
             Path: /nsc/.config/nats/context/leaf-user.json

+ nsc generate config --nats-resolver --sys-account SYS
Creating the main server conf...
+ echo 'Creating the main server conf...'
+ cat
+ echo 'Creating the leaf node conf...'
Creating the leaf node conf...
+ cat
+ MAIN_PID=99
+ sleep 1
+ nats-server -c main.conf
Pushing the account JWT...
+ echo 'Pushing the account JWT...'
+ nsc push -a APP
[ OK ] push to nats-server "nats://0.0.0.0:4222" using system account "SYS":
       [ OK ] push APP to nats-server with nats account resolver:
              [ OK ] pushed "APP" to nats-server NB6YKJ4ZAT3MYWMHMY2UYRSX6DXPUHPOJQOAHPJ4ZW2PN2BHIMS7ID4T: jwt updated
              [ OK ] pushed to a total of 1 nats-server
+ LEAF_PID=115
+ sleep 1
+ nats-server -c leaf.conf
+ SERVICE_PID=125
+ sleep 1
+ nats --context main-user reply greet hello
12:12:21 Listening on "greet" in group "NATS-RPLY-22"
+ nats --context leaf-user request greet 
12:12:22 Sending request on "greet"


12:12:22 [#0] Received on subject "greet":
12:12:22 Received with rtt 861.64µs
hello

+ kill 125
+ kill 115
+ kill 99

Recording

Note, playback is half speed to make it a bit easier to follow.