Connect To Hazelcast Running on Kubernetes from Outside
See something wrong? Edit this page.
This is a complete example presenting how to use Hazelcast cluster deployed on Kubernetes with Hazelcast Client running outside of Kubernetes. This example assumes you have a running Kubernetes cluster and the kubectl tool installed and configured.
What You’ll Learn
In this guide you will learn how to deploy Hazelcast Kubernetes cluster and connect to it using a client outside Kubernetes.
Prerequisites
-
~15 minutes
-
Kubernetes cluster (Docker for Desktop or Minikube is good enough)
-
JDK 1.8+
-
Apache Maven 3.2+
Code samples
All the code used in this guide is available in here. To download it, execute the following command.
git clone https://github.com/hazelcast-guides/kubernetes-external-client.git
Configure Hazelcast cluster on Kubernetes
Configure Service Account
Hazelcast uses Kubernetes API for the member discovery and therefore it requires granting permission to certain resources. To create ServiceAccount with minimal permissions, run the following command.
kubectl apply -f rbac.yaml
The Service Account 'hazelcast-service-account' was created, and you can use it in all further steps.
Install Hazelcast cluster
To install Hazelcast cluster, you need to include the Service-Per-Pod annotations into your StatefulSet (or Deployment) Hazelcast configuration. Then, deploy Hazelcast cluster into your Kubernetes environment.
kubectl apply -f hazelcast-cluster.yaml
To check the deployment you can run the following command:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
hazelcast-0 1/1 Running 0 2m
hazelcast-1 1/1 Running 0 2m
hazelcast-2 1/1 Running 0 2m
If you are using Minikube, you need to execute minikube tunnel now in order to get LoadBalancer External IPs assigned. |
As the first step, you need to start Hazelcast cluster in such a way that each member is exposed with a separate public IP/port. In Kubernetes PODs can be accessed from outside only via services, so the configuration requires creating a separate service (LoadBalancer or NodePort) for each Hazelcast member POD. This can be done manually or using Metacontroller plugin with Service-Per-Pod Decorator Controller.
To create a loadbalancer
for each running Hazelcast pod you need to run the following command:
for pod in $(kubectl get pods -o jsonpath="{.items[*].metadata.name}"); do \
kubectl create service loadbalancer ${pod} --tcp=5701 -o yaml --dry-run=client | kubectl set selector --local -f - "statefulset.kubernetes.io/pod-name=${pod}" -o yaml | kubectl create -f -; \
done
Check the services were created with by running the command:
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hazelcast-0 LoadBalancer 10.175.248.67 35.195.77.97 5701:30915/TCP 11m
hazelcast-1 LoadBalancer 10.175.254.105 104.155.70.30 5701:30440/TCP 11m
hazelcast-2 LoadBalancer 10.175.253.232 34.78.182.50 5701:30305/TCP 11m
-
Install Metacontroller plugin
To install Metacontroller plugin, it’s enough to execute the following commands:
# Create metacontroller namespace.
kubectl create namespace metacontroller
# Create metacontroller service account and role/binding.
kubectl apply -f https://raw.githubusercontent.com/metacontroller/metacontroller/v1.4.2/manifests/production/metacontroller-rbac.yaml
# Create CRDs for Metacontroller APIs
kubectl apply -f https://raw.githubusercontent.com/metacontroller/metacontroller/v1.4.2/manifests/production/metacontroller-crds-v1.yaml
# Create Metacontroller StatefulSet.
kubectl apply -f https://raw.githubusercontent.com/metacontroller/metacontroller/v1.4.2/manifests/production/metacontroller.yaml
If you have any issues while creating Metacontroller, it may mean that you don’t have ClusterRole access to your cluster. Please check this for details.
-
Install Service-Per-Pod DecoratorController
To install Service-Per-Pod Decorator Controller, you need to execute the following commands.
kubectl create configmap service-per-pod-hooks -n metacontroller --from-file=hooks
kubectl apply -f service-per-pod.yaml
This Decorator Controller automatically creates a service for each POD marked with the following annotations.
annotations:
service-per-pod-label: "statefulset.kubernetes.io/pod-name"
service-per-pod-ports: "5701:5701"
You can check that for each Hazelcast Member POD there was a service created and that Hazelcast members formed a cluster.
$ kubectl get all
NAME READY STATUS RESTARTS AGE
pod/hazelcast-0 1/1 Running 0 2m
pod/hazelcast-1 1/1 Running 0 1m
pod/hazelcast-2 1/1 Running 0 1m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/hazelcast-0 LoadBalancer 10.19.241.253 35.188.83.111 5701:30597/TCP 2m
service/hazelcast-1 LoadBalancer 10.19.251.243 35.192.168.46 5701:32718/TCP 2m
service/hazelcast-2 LoadBalancer 10.19.254.0 35.193.248.247 5701:30267/TCP 2m
service/kubernetes ClusterIP 10.19.240.1 <none> 443/TCP 1h
$ kubectl logs pod/hazelcast-2
...
Members {size:3, ver:3} [
Member [10.16.1.10]:5701 - abab30fe-5a45-484d-bad5-e60c252572ca
Member [10.16.2.7]:5701 - 9b948e91-0115-470f-850e-d5cbf2e3b0e1
Member [10.16.0.8]:5701 - e68ce431-4000-467b-92c6-0072b2601d60 this
]
...
Configure Hazelcast Client outside Kubernetes
When we have a working Hazelcast cluster deployed on Kubernetes, we can connect to it with an external Hazelcast Client. You need first to fetch the credentials of the created Service Account and then use them to configure the client.
Check Kubernetes Master IP
To check the IP address of the Kubernetes Master, use the following command.
$ kubectl cluster-info
Kubernetes master is running at https://35.233.44.52
Check Access Token and CA Certificate
First, you need to find the name of the secret for the created Service Account.
$ kubectl get secret
NAME TYPE DATA AGE
default-token-7xd6t kubernetes.io/service-account-token 3 102m
hazelcast-service-account-token-m8p82 kubernetes.io/service-account-token 3 35m
Then, to fetch Access Token, use the following command.
$ kubectl get secret hazelcast-service-account-token-m8p82 -o jsonpath={.data.token} | base64 --decode | xargs echo
eyJhbGciOiJSUzI1NiIsImtpZCI6InNfRVBNNHg1dFRlWHY1VmVNdVJtaHY3ZzZ6aEhQWWxZT2ZzNXQwcklSYzQifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImhhemVsY2FzdC1zZXJ2aWNlLWFjY291bnQtdG9rZW4tbThwODIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiaGF6ZWxjYXN0LXNlcnZpY2UtYWNjb3VudCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjhjNzJhYmU3LTgzMjYtNDk5ZC04Yzc2LWU2MzMzMDAwNzYzYyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmhhemVsY2FzdC1zZXJ2aWNlLWFjY291bnQifQ.af6PWcxOV3IN3C-SiSa6hbgW9TlernWC3b5K1dNLVplzBdnJKI-NjacgMfd03BcVnXNLpXHFxr5gbwwguv9ZjeOAz0FrrQvs7rLRnXcQaE0x7EcG1lyfQ8utwzUvSodEQN6kbzjBF0WWClGJXdtdQvVn9lcFqDjfC6Jugk7dDEaD9W_mDjIXZEaeEtUIR5tnivGwRxG0dEpYb0nj9xAC9fjq40BTHWlbuVoFBhVl3-kPZreIiYHQYurOx30wuszuu0Ba16MbxyPB4rvzufngb8ej5S-KtUllxHiuYA4Gp01zbjb8RIsnB-OM02fCfIgG-YyyQjjD0K-BEEyuihs-0g
To fetch CA Certificate, use the following command.
$ kubectl get secret hazelcast-service-account-token-6s94h -o jsonpath='{.data.ca\.crt}' | base64 --decode
-----BEGIN CERTIFICATE-----
MIIDKjCCAhKgAwIBAgIQHJPQGLRW659SCjtRVEF23TANBgkqhkiG9w0BAQsFADAv
MS0wKwYDVQQDEyQzMjU5YmE1ZC03MWVhLTRiNGYtODZmNi0yOGFmNzYzZTg0NWYw
HhcNMjAxMTA5MTE1MjU1WhcNMjUxMTA4MTI1MjU1WjAvMS0wKwYDVQQDEyQzMjU5
YmE1ZC03MWVhLTRiNGYtODZmNi0yOGFmNzYzZTg0NWYwggEiMA0GCSqGSIb3DQEB
AQUAA4IBDwAwggEKAoIBAQDCGVpermveJbIvrTcyHtC9zax+87cSPQCz7lfbI8gB
75JpeDZy/32RUwG7xRFZ58gynqvkMG4z/ZjguXJsp8g9dT8qMLf01CbqyQscoVnL
cyiicGNjnaMeQul8W3XN4uXlqTubFHYcA/quX07CJhRyES8JNliRdygQJdWF8aKs
OirX2A0Vncoq6eGsr3yCJYTvxZQLNM/3wxCvUJPLJcYbKgIZUNZbvC8ovFG7ChAk
G0V+Bw726eiiHGCMFAkz42f3/T+wxMdIAm2SKunT3OTyO9/6SCuJflPMB9/kPQRE
yxH2H5FdW2p+qC2o2T5pgslmPkaJ7qck5JsgwoOgvwzhAgMBAAGjQjBAMA4GA1Ud
DwEB/wQEAwICBDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRtxNC8UAR7mXjC
fXOV58kmMg7T8zANBgkqhkiG9w0BAQsFAAOCAQEAAJHer6jqoPGMEdjRSBxgW7yw
DPU210ffOmWWuqGAHCbuAhcVZG5dce9KjTzKc/sWqI0qR3zrZhMpP06cce2NIsHr
VbViVgPT0/K4oNP4l4BBaseLKNgfywOSUtmTtwyKbQJTHElwFwbMzBZF9kihZMah
l0zhm5JizXwmBsE38r3+soS9mIvjASGcGp7exTftPyMP8xMuBUvg28Z9w8wxfHv4
oZ/yor2Cp4bdGDGQvwKt/WqXfxIk4LW5jKUupoZ+2kkl8eYEE314PFIhayWcTrIf
bnDp6Qq8bRQ6acz6CI5AaZROUQmRYmN8Aaz3THJmkfGcejzvbYux/GiD7Nxlkw==
-----END CERTIFICATE-----
Configure Hazelcast Client
Modify src/main/resources/hazelcast-client.yaml to include your credentials and Kubernetes master IP.
hazelcast-client:
network:
kubernetes:
enabled: true
namespace: default
use-public-ip: true
kubernetes-master: https://35.233.44.52
api-token: eyJhbGciOiJSUzI1NiIsImtpZCI6InNfRVBNNHg1dFRlWHY1VmVNdVJtaHY3ZzZ6aEhQWWxZT2ZzNXQwcklSYzQifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImhhemVsY2FzdC1zZXJ2aWNlLWFjY291bnQtdG9rZW4tbThwODIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiaGF6ZWxjYXN0LXNlcnZpY2UtYWNjb3VudCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjhjNzJhYmU3LTgzMjYtNDk5ZC04Yzc2LWU2MzMzMDAwNzYzYyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmhhemVsY2FzdC1zZXJ2aWNlLWFjY291bnQifQ.af6PWcxOV3IN3C-SwSa6hbgW9TlernWC3b5K1dNLVplzBdnJKI-NjacgMfd03BcVnXNLpXHFxr5gbwwguv9ZjeOAz0FrrQvs7rLRnXcQaE0x7EcG1lyfQ8utwzUvSodEQN6kbzjBF0WWClGJXdtdQvVn9lcFqDjfC6Jugk7dDEaD9W_mDjIXZEaeEtUIR5tnivGwRxG0dEpYb0nj9xAC9fjq40BTHWlbuVoFBhVl3-kPZreIiYHQYurOx30wuszuu0Ba16MbxyPB4rvzufngb8ej5S-KtUllxHiuYA4Gp01zbjb8RIsnB-OM02fCfIgG-YyyQjjD0K-BEEyuihs-0g
ca-certificate: |
-----BEGIN CERTIFICATE-----
MIIDKjCCAhKgAwIBAgIQHJPQGLRW659SCjtRVEF23TANBgkqhkiG9w0BAQsFADAv
MS0wKwYDVQQDEyQzMjU5YmE1ZC03MWVhLTRiNGYtODZmNi0yOGFmNzYzZTg0NWYw
HhcNMjAxMTA5MTE1MjU1WhcNMjUxMTA4MTI1MjU1WjAvMS0wKwYDVQQDEyQzMjU5
YmE1ZC03MWVhLTRiNGYtODZmNi0yOGFmNzYzZTg0NWYwggEiMA0GCSqGSIb3DQEB
AQUAA4IBDwAwggEKAoIBAQDCGVpermveJbIvrTcyHtC9zax+87cSPQCz7lfbI8gB
75JpeDZy/32RUwG7xRFZ58gynqvkMG4z/ZjguXJsp8g9dT8qMLf01CbqyQscoVnL
cyiicGNjnaMeQul8W3XN4uXlqTubFHYcA/quX02CJhRyES8JNliRdygQJdWF8aKs
OirX2A0Vncoq6eGsr3yCJYTvxZQLNM/3wxCvUJPLJcYbKgIZUNZbvC8ovFG7ChAk
G0V+Bw726eiiHGCMFAkz42f3/T+wxMdIAm2SKunT3OTyO9/6SCuJflPMB9/kPQRE
yxH2H5FdW2p+qC2o2T5pgslmPkaJ7qck5JsgwoOgvwzhAgMBAAGjQjBAMA4GA1Ud
DwEB/wQEAwICBDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRtxNC8UAR7mXjC
fXOV58kmMg7T8zANBgkqhkiG9w0BAQsFAAOCAQEAAJHer6jqoPGMEdjRSBxgW7yw
DPU210ffOmWWuqGAHCbuAhcVZG5dce9KjTzKc/sWqI0qR3zrZhMpP06cce2NIsHr
VbViVgPT0/K4oNP4l4BBaseLKNgfywOSUtmTtwyKbQJTHElwFwbMzBZF9kihZMah
l0zhm5JizXwmBsE38r3+soS9mIvjASGcGp7exTftPyMP8xMuBUvg28Z9w8wxfHv4
oZ/yor2Cp4bdGDGQvwKt/WqXfxIk4LW5jKUupoZ+2kkl8eYEE314PFIhayWcTrIf
bnDp6Qq8bRQ6acz6CI5AaZROUQmRYmN8Aaz3THJmkfGcejzvbYux/GiD7Nxlkw==
-----END CERTIFICATE-----
Run Hazelcast Client application
You can run the client application with the following command.
mvn spring-boot:run
Application is a web service that uses Hazelcast Client to connect to the Hazelcast cluster. To check it works correctly, you can execute the following commands:
$ curl "localhost:8080/put?key=some-key&value=some-value"
{"response":null}
$ curl "localhost:8080/get?key=some-key"
{"response":"some-value"}
You can also check the application logs to see:
Members [3] {
Member [10.172.2.8]:5701 - 77fb7da6-79bd-4b5b-9c63-2c425a111c06
Member [10.172.0.4]:5701 - d63bd9f6-7afd-4b9b-88d3-f3b6afb749f9
Member [10.172.0.5]:5701 - 049fe137-9cfd-4be8-80a3-d1357a31f6f4
}