Creating a vRealize Log Insight 2.5 cluster with Integrated Load Balancing
vRealize Log Insight 2.5 improves on the clustering in previous versions with an Integrated Load Balancer (ILB) which allows you to distribute load across your cluster of Log Insight instances without actually needing an external load balancer. The advantage of this over an external load balancer is that the source IP is maintained which allows for easier analysis.
The minimum number of nodes in a cluster is three, the first node becomes the Master node and the other two become Worker nodes. The maximum number of nodes supported is six, though acording to Mr Log Insight himself, Steve Flanders, the hard limit is more:
@sammcgeown yes though hard limit in product is much higher. How big do you need?
— Steve Flanders (@smflanders) February 11, 2015
The Log Insight appliance comes in four sizes: an Extra Small, for use in labs or PoC, through to Large, which can consume a whopping 112.5GB of logs per day. Those figures scale in a linear fashion for clusters, so a 3-node cluster of large instances can consume 337.5GB per day, from 2,250 syslog connections at a rate of 22,500 events a second. See more on sizing vRealize Log Insight.
Load Balanced Cluster Pre-requisites
- Configure a minimum of three nodes in a Log Insight cluster.
- Verify that all Log Insight nodes and the specified Integrated Load Balancer IP address are on the same network.
- The Log Insight master and worker nodes must have the same certificates. Otherwise the Log Insight Agents configured to connect through SSL will reject the connection. When uploading a CA-signed certificate to Log Insight master and worker nodes, set the Common Name to ILB IP address during certificate generation request. See Generate a Certificate Signing Request.
- You must synchronize the time on the Log Insight Linux Agent virtual appliance with an NTP server. See Synchronize the Time on the Log Insight Virtual Appliance.
With those in mind, I will deploy 3 nodes, VRLI-01, VRLI-02 and VRLI-03, and load balance them under the URL “loginsight.definit.local”.
Generate a certificate for vRealize Log Insight
I am using a Microsoft Windows Server 2012 Certificate Authority, with the VMware certificate template created as per Creating a Microsoft Certificate Authority Template for SSL certificate creation in vSphere 5.x (2062108).
I also have OpenSSL installed to generate the CSR, and will use certreq.exe to submit and retrieve the certificate.
Create a config file for OpenSSL with the following settings. Replace the values in red with your own, ensuring that the Common Name is the IP address you will assign to the load balancer.
[ req ]
default_bits = 2048
default_keyfile = rui.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = DNS:loginsight.definit.local, DNS:vrli-01.definit.local, DNS:vrli-02.definit.local, DNS:vrli-03.definit.local
[ req_distinguished_name ]
countryName = UK
stateOrProvinceName = West Sussex
localityName = Horsham
0.organizationName = DefinIT
organizationalUnitName = Lab
commonName = 192.168.1.26
Use OpenSSL to generate a new private key file, and a Certificate Signing Request:
c:\OpenSSL\bin\openssl.exe req -new -nodes -out z:\Certificates\LogInsight\rui.csr -keyout z:\Certificates\LogInsight\rui.key -config z:\Certificates\LogInsight\loginsight.cfg
This command generates a CSR (rui.csr), a private key (rui.key) and takes the config file as an input (loginsight.cfg).
Next, submit the template using certreq:
Z:\Certificates\LogInsight>certreq -submit -attrib "CertificateTemplate:VMware" rui.csr
Select your Issuing Certificate Authority and then save the certificate that is returned (I called mine LogInsight.cer).
Now create a new text file and paste in, in order, the contents of the private key file (rui.key), the certificate (LogInsight.cer) and the Issuing CA certificate. Save the new file as LogInsight.pem.
This certificate will need to be added to all of the LogInsight instances later.
Deploy the Master vRealize Log Insight node
The OVF deploy is simple:
Once the node has completed it’s boot up and is at the screen below you can proceed.
Open your web browser of choice and go to the IP address of the Master node and you should see the startup wizard. For the Master node we start a new deployment:
Note: If you see “Failed to start new deployment” you can reboot the appliance and try again, however I found that it caused problems later on in the deployment (“Login failed” during the wizard) and required a re-deploy of the appliance. When I redeployed, I rebooted pre-emptively and still saw the “Failed to start new deployment” message. Rebooting again allowed me to proceed – this time without the “Login failed” messages. The login failed errors actually stop you from completing the wizard, so if you see those it’s time to redeploy.
After a reboot clicking the “Next” button takes you straight to the next step, which is to set the admin password (This is a different user to the “root” password that we set during the OVF deploy, that’s used for SSH access).
At this point I skipped the NTP and SMTP settings to configure them later (I have found the wizard to be unreliable, NTP would not pass tests and SMTP would not send a test email). Finish the wizard and you should be presented with the login page:
Configure the Master Node
Log in to the master node using “admin” and the password you set. In the top right hand side of the dashboard view, you can open the Administration page
Select “Time” under the “Configuration” heading. Time is always critical for clustering, as well as for accurate logging. I have a local NTP server set up on my network to which everything syncs – so I configured and tested that:
Similarly, go to the SMTP section and configure the SMTP settings:
And again, if required, select the Authentication page and configure Active Directory support
AD Groups can be added via the “Access Control” page
Finally, import the certificate via the SSL page and reboot
Deploy the Worker nodes
Deploy two more instances in the same way as the first – but this time use the “Join Existing Deployment” option, and then specify the first node (for me vrli-01.definit.local) and click “go”. You’ll then see a message to go and approve the Worker on the Master node (and you’ll get an email, if you set that up).
Click allow to join the worker to the cluster.
The Worker will then show a nice green tick, and the Master will have a whole load of information which can be summed up as “configure DNS and NTP, and think about load balancing” – all of which we have done, or will do.
Click OK on the Worker node and it will take you back to the login page. Log in using the admin credentials and you can see there are very few options available to us, just some general information and the SSL page.
Import the SSL certificate we generated before and restart the Worker
Enabling the Integrated Load Balancer
Now the cluster has been deployed and the two Worker nodes are registered with the Master, we can enable the Integrated Load Balancer (ILB). So long as the pre-requisites have been met, this is simply a case of ticking the box and entering an IP address for the load balancer:
Once saved, it will take a few seconds to configure and then the status will go green – “Available”
I’ve created a DNS record for the load balanced IP to use to direct clients, this means any future changes can be implemented easily.
My Log Insight cluster is now available, secured by SSL, on https://loginsight.definit.local
Post deployment configuration
Having a load balanced cluster is all well and good, but if the nodes are on the same physical host and that goes down, you could be faced with a loss of data while HA recovers them. Be sure to create an anti-affinity rule to keep them on separate hosts!