Connect To Hazelcast Running on Kubernetes from Outside

See something wrong? Edit this page.

This is a complete example presenting how to use Hazelcast cluster deployed on Kubernetes with Hazelcast Client running outside of Kubernetes. This example assumes you have a running Kubernetes cluster and the kubectl tool installed and configured.

What You’ll Learn

In this guide you will learn how to deploy Hazelcast Kubernetes cluster and connect to it using a client outside Kubernetes.

Prerequisites

~15 minutes
Kubernetes cluster (Docker for Desktop or Minikube is good enough)
Git
JDK 1.8+
Apache Maven 3.2+

Code samples

All the code used in this guide is available in here. To download it, execute the following command.

git clone https://github.com/hazelcast-guides/kubernetes-external-client.git

Configure Hazelcast cluster on Kubernetes

Configure Service Account

Hazelcast uses Kubernetes API for the member discovery and therefore it requires granting permission to certain resources. To create ServiceAccount with minimal permissions, run the following command.

kubectl apply -f rbac.yaml

The Service Account 'hazelcast-service-account' was created, and you can use it in all further steps.

Install Hazelcast cluster

To install Hazelcast cluster, you need to include the Service-Per-Pod annotations into your StatefulSet (or Deployment) Hazelcast configuration. Then, deploy Hazelcast cluster into your Kubernetes environment.

kubectl apply -f hazelcast-cluster.yaml

To check the deployment you can run the following command:

$ kubectl get pods
NAME          READY   STATUS    RESTARTS   AGE
hazelcast-0   1/1     Running   0          2m
hazelcast-1   1/1     Running   0          2m
hazelcast-2   1/1     Running   0          2m

If you are using Minikube, you need to execute minikube tunnel now in order to get LoadBalancer External IPs assigned.

As the first step, you need to start Hazelcast cluster in such a way that each member is exposed with a separate public IP/port. In Kubernetes PODs can be accessed from outside only via services, so the configuration requires creating a separate service (LoadBalancer or NodePort) for each Hazelcast member POD. This can be done manually or using Metacontroller plugin with Service-Per-Pod Decorator Controller.

Manual
Metacontroller

To create a loadbalancer for each running Hazelcast pod you need to run the following command:

for pod in $(kubectl get pods -o jsonpath="{.items[*].metadata.name}"); do \
  kubectl create service loadbalancer ${pod} --tcp=5701 -o yaml --dry-run=client | kubectl set selector --local -f - "statefulset.kubernetes.io/pod-name=${pod}" -o yaml | kubectl create -f -; \
done

Check the services were created with by running the command:

$ kubectl get svc
NAME          TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)          AGE
hazelcast-0   LoadBalancer   10.175.248.67    35.195.77.97    5701:30915/TCP   11m
hazelcast-1   LoadBalancer   10.175.254.105   104.155.70.30   5701:30440/TCP   11m
hazelcast-2   LoadBalancer   10.175.253.232   34.78.182.50    5701:30305/TCP   11m

Install Metacontroller plugin

To install Metacontroller plugin, it’s enough to execute the following commands:

# Create metacontroller namespace.
kubectl create namespace metacontroller
# Create metacontroller service account and role/binding.
kubectl apply -f https://raw.githubusercontent.com/metacontroller/metacontroller/v1.4.2/manifests/production/metacontroller-rbac.yaml
# Create CRDs for Metacontroller APIs
kubectl apply -f https://raw.githubusercontent.com/metacontroller/metacontroller/v1.4.2/manifests/production/metacontroller-crds-v1.yaml
# Create Metacontroller StatefulSet.
kubectl apply -f https://raw.githubusercontent.com/metacontroller/metacontroller/v1.4.2/manifests/production/metacontroller.yaml

If you have any issues while creating Metacontroller, it may mean that you don’t have ClusterRole access to your cluster. Please check this for details.

Install Service-Per-Pod DecoratorController

To install Service-Per-Pod Decorator Controller, you need to execute the following commands.

kubectl create configmap service-per-pod-hooks -n metacontroller --from-file=hooks
kubectl apply -f service-per-pod.yaml

This Decorator Controller automatically creates a service for each POD marked with the following annotations.

annotations:
    service-per-pod-label: "statefulset.kubernetes.io/pod-name"
    service-per-pod-ports: "5701:5701"

You can check that for each Hazelcast Member POD there was a service created and that Hazelcast members formed a cluster.

$ kubectl get all
NAME              READY     STATUS    RESTARTS   AGE
pod/hazelcast-0   1/1       Running   0          2m
pod/hazelcast-1   1/1       Running   0          1m
pod/hazelcast-2   1/1       Running   0          1m

NAME                  TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)          AGE
service/hazelcast-0   LoadBalancer   10.19.241.253   35.188.83.111    5701:30597/TCP   2m
service/hazelcast-1   LoadBalancer   10.19.251.243   35.192.168.46    5701:32718/TCP   2m
service/hazelcast-2   LoadBalancer   10.19.254.0     35.193.248.247   5701:30267/TCP   2m
service/kubernetes    ClusterIP      10.19.240.1     <none>           443/TCP          1h

$ kubectl logs pod/hazelcast-2
...
Members {size:3, ver:3} [
        Member [10.16.1.10]:5701 - abab30fe-5a45-484d-bad5-e60c252572ca
        Member [10.16.2.7]:5701 - 9b948e91-0115-470f-850e-d5cbf2e3b0e1
        Member [10.16.0.8]:5701 - e68ce431-4000-467b-92c6-0072b2601d60 this
]
...

Configure Hazelcast Client outside Kubernetes

When we have a working Hazelcast cluster deployed on Kubernetes, we can connect to it with an external Hazelcast Client. You need first to fetch the credentials of the created Service Account and then use them to configure the client.

Check Kubernetes Master IP

To check the IP address of the Kubernetes Master, use the following command.

$ kubectl cluster-info
Kubernetes master is running at https://35.233.44.52

Check Access Token and CA Certificate

First, you need to find the name of the secret for the created Service Account.

$ kubectl get secret
NAME                                    TYPE                                  DATA   AGE
default-token-7xd6t                     kubernetes.io/service-account-token   3      102m
hazelcast-service-account-token-m8p82   kubernetes.io/service-account-token   3      35m

Then, to fetch Access Token, use the following command.

$ kubectl get secret hazelcast-service-account-token-m8p82 -o jsonpath={.data.token} | base64 --decode | xargs echo
eyJhbGciOiJSUzI1NiIsImtpZCI6InNfRVBNNHg1dFRlWHY1VmVNdVJtaHY3ZzZ6aEhQWWxZT2ZzNXQwcklSYzQifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImhhemVsY2FzdC1zZXJ2aWNlLWFjY291bnQtdG9rZW4tbThwODIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiaGF6ZWxjYXN0LXNlcnZpY2UtYWNjb3VudCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjhjNzJhYmU3LTgzMjYtNDk5ZC04Yzc2LWU2MzMzMDAwNzYzYyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmhhemVsY2FzdC1zZXJ2aWNlLWFjY291bnQifQ.af6PWcxOV3IN3C-SiSa6hbgW9TlernWC3b5K1dNLVplzBdnJKI-NjacgMfd03BcVnXNLpXHFxr5gbwwguv9ZjeOAz0FrrQvs7rLRnXcQaE0x7EcG1lyfQ8utwzUvSodEQN6kbzjBF0WWClGJXdtdQvVn9lcFqDjfC6Jugk7dDEaD9W_mDjIXZEaeEtUIR5tnivGwRxG0dEpYb0nj9xAC9fjq40BTHWlbuVoFBhVl3-kPZreIiYHQYurOx30wuszuu0Ba16MbxyPB4rvzufngb8ej5S-KtUllxHiuYA4Gp01zbjb8RIsnB-OM02fCfIgG-YyyQjjD0K-BEEyuihs-0g

To fetch CA Certificate, use the following command.

$ kubectl get secret hazelcast-service-account-token-6s94h -o jsonpath='{.data.ca\.crt}' | base64 --decode
-----BEGIN CERTIFICATE-----
MIIDKjCCAhKgAwIBAgIQHJPQGLRW659SCjtRVEF23TANBgkqhkiG9w0BAQsFADAv
MS0wKwYDVQQDEyQzMjU5YmE1ZC03MWVhLTRiNGYtODZmNi0yOGFmNzYzZTg0NWYw
HhcNMjAxMTA5MTE1MjU1WhcNMjUxMTA4MTI1MjU1WjAvMS0wKwYDVQQDEyQzMjU5
YmE1ZC03MWVhLTRiNGYtODZmNi0yOGFmNzYzZTg0NWYwggEiMA0GCSqGSIb3DQEB
AQUAA4IBDwAwggEKAoIBAQDCGVpermveJbIvrTcyHtC9zax+87cSPQCz7lfbI8gB
75JpeDZy/32RUwG7xRFZ58gynqvkMG4z/ZjguXJsp8g9dT8qMLf01CbqyQscoVnL
cyiicGNjnaMeQul8W3XN4uXlqTubFHYcA/quX07CJhRyES8JNliRdygQJdWF8aKs
OirX2A0Vncoq6eGsr3yCJYTvxZQLNM/3wxCvUJPLJcYbKgIZUNZbvC8ovFG7ChAk
G0V+Bw726eiiHGCMFAkz42f3/T+wxMdIAm2SKunT3OTyO9/6SCuJflPMB9/kPQRE
yxH2H5FdW2p+qC2o2T5pgslmPkaJ7qck5JsgwoOgvwzhAgMBAAGjQjBAMA4GA1Ud
DwEB/wQEAwICBDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRtxNC8UAR7mXjC
fXOV58kmMg7T8zANBgkqhkiG9w0BAQsFAAOCAQEAAJHer6jqoPGMEdjRSBxgW7yw
DPU210ffOmWWuqGAHCbuAhcVZG5dce9KjTzKc/sWqI0qR3zrZhMpP06cce2NIsHr
VbViVgPT0/K4oNP4l4BBaseLKNgfywOSUtmTtwyKbQJTHElwFwbMzBZF9kihZMah
l0zhm5JizXwmBsE38r3+soS9mIvjASGcGp7exTftPyMP8xMuBUvg28Z9w8wxfHv4
oZ/yor2Cp4bdGDGQvwKt/WqXfxIk4LW5jKUupoZ+2kkl8eYEE314PFIhayWcTrIf
bnDp6Qq8bRQ6acz6CI5AaZROUQmRYmN8Aaz3THJmkfGcejzvbYux/GiD7Nxlkw==
-----END CERTIFICATE-----

Configure Hazelcast Client

Modify src/main/resources/hazelcast-client.yaml to include your credentials and Kubernetes master IP.

hazelcast-client:
  network:
    kubernetes:
      enabled: true
      namespace: default
      use-public-ip: true
      kubernetes-master: https://35.233.44.52
      api-token: eyJhbGciOiJSUzI1NiIsImtpZCI6InNfRVBNNHg1dFRlWHY1VmVNdVJtaHY3ZzZ6aEhQWWxZT2ZzNXQwcklSYzQifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImhhemVsY2FzdC1zZXJ2aWNlLWFjY291bnQtdG9rZW4tbThwODIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiaGF6ZWxjYXN0LXNlcnZpY2UtYWNjb3VudCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjhjNzJhYmU3LTgzMjYtNDk5ZC04Yzc2LWU2MzMzMDAwNzYzYyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmhhemVsY2FzdC1zZXJ2aWNlLWFjY291bnQifQ.af6PWcxOV3IN3C-SwSa6hbgW9TlernWC3b5K1dNLVplzBdnJKI-NjacgMfd03BcVnXNLpXHFxr5gbwwguv9ZjeOAz0FrrQvs7rLRnXcQaE0x7EcG1lyfQ8utwzUvSodEQN6kbzjBF0WWClGJXdtdQvVn9lcFqDjfC6Jugk7dDEaD9W_mDjIXZEaeEtUIR5tnivGwRxG0dEpYb0nj9xAC9fjq40BTHWlbuVoFBhVl3-kPZreIiYHQYurOx30wuszuu0Ba16MbxyPB4rvzufngb8ej5S-KtUllxHiuYA4Gp01zbjb8RIsnB-OM02fCfIgG-YyyQjjD0K-BEEyuihs-0g
      ca-certificate: |
        -----BEGIN CERTIFICATE-----
        MIIDKjCCAhKgAwIBAgIQHJPQGLRW659SCjtRVEF23TANBgkqhkiG9w0BAQsFADAv
        MS0wKwYDVQQDEyQzMjU5YmE1ZC03MWVhLTRiNGYtODZmNi0yOGFmNzYzZTg0NWYw
        HhcNMjAxMTA5MTE1MjU1WhcNMjUxMTA4MTI1MjU1WjAvMS0wKwYDVQQDEyQzMjU5
        YmE1ZC03MWVhLTRiNGYtODZmNi0yOGFmNzYzZTg0NWYwggEiMA0GCSqGSIb3DQEB
        AQUAA4IBDwAwggEKAoIBAQDCGVpermveJbIvrTcyHtC9zax+87cSPQCz7lfbI8gB
        75JpeDZy/32RUwG7xRFZ58gynqvkMG4z/ZjguXJsp8g9dT8qMLf01CbqyQscoVnL
        cyiicGNjnaMeQul8W3XN4uXlqTubFHYcA/quX02CJhRyES8JNliRdygQJdWF8aKs
        OirX2A0Vncoq6eGsr3yCJYTvxZQLNM/3wxCvUJPLJcYbKgIZUNZbvC8ovFG7ChAk
        G0V+Bw726eiiHGCMFAkz42f3/T+wxMdIAm2SKunT3OTyO9/6SCuJflPMB9/kPQRE
        yxH2H5FdW2p+qC2o2T5pgslmPkaJ7qck5JsgwoOgvwzhAgMBAAGjQjBAMA4GA1Ud
        DwEB/wQEAwICBDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRtxNC8UAR7mXjC
        fXOV58kmMg7T8zANBgkqhkiG9w0BAQsFAAOCAQEAAJHer6jqoPGMEdjRSBxgW7yw
        DPU210ffOmWWuqGAHCbuAhcVZG5dce9KjTzKc/sWqI0qR3zrZhMpP06cce2NIsHr
        VbViVgPT0/K4oNP4l4BBaseLKNgfywOSUtmTtwyKbQJTHElwFwbMzBZF9kihZMah
        l0zhm5JizXwmBsE38r3+soS9mIvjASGcGp7exTftPyMP8xMuBUvg28Z9w8wxfHv4
        oZ/yor2Cp4bdGDGQvwKt/WqXfxIk4LW5jKUupoZ+2kkl8eYEE314PFIhayWcTrIf
        bnDp6Qq8bRQ6acz6CI5AaZROUQmRYmN8Aaz3THJmkfGcejzvbYux/GiD7Nxlkw==
        -----END CERTIFICATE-----

Run Hazelcast Client application

You can run the client application with the following command.

mvn spring-boot:run

Application is a web service that uses Hazelcast Client to connect to the Hazelcast cluster. To check it works correctly, you can execute the following commands:

$ curl "localhost:8080/put?key=some-key&value=some-value"
{"response":null}

$ curl "localhost:8080/get?key=some-key"
{"response":"some-value"}

You can also check the application logs to see:

Members [3] {
	Member [10.172.2.8]:5701 - 77fb7da6-79bd-4b5b-9c63-2c425a111c06
	Member [10.172.0.4]:5701 - d63bd9f6-7afd-4b9b-88d3-f3b6afb749f9
	Member [10.172.0.5]:5701 - 049fe137-9cfd-4be8-80a3-d1357a31f6f4
}