There are a couple of tweaks required in order to forward logs to vRealize Automation Log Insight Cloud. We need to use the HTTP output in the Fluent Bit configuration to forward the logs as a JSON payload to the Log Insight API. We also need to append an authentication header with an API key, and configure some tags so that the Tanzu Kubnernetes Grid Content Pack will pick up the logs.
To deploy the Fluent-bit package, first list the available versions and then install the package using a YAML configuration file. Fluent-bit will be installed in the tanzu-system-logging namespace by default.
1
2
3
4
5
6
7
8
9
$ tanzu package available list fluent-bit.tanzu.vmware.com -A
| Retrieving package versions for fluent-bit.tanzu.vmware.com...
NAME VERSION RELEASED-AT NAMESPACE
fluent-bit.tanzu.vmware.com 1.7.5+vmware.1-tkg.1 2021-05-13T18:00:00Z default
$ tanzu package install fluent-bit --package-name fluent-bit.tanzu.vmware.com \
--version 1.7.5+vmware.1-tkg.1 --values-file fluent-bit-data-values.yaml \
--create-namespace
Lines 15-23 configure the HTTP output with the Log Insight address, API key authentication and JSON formatting
Lines 68-69 add a filter to tag environment and log_type for the TKG content pack
Lines 70-72 add a tag for the tkg_cluster and tkg_instance
Once the package has been deployed you can check it’s sending logs and tagging them correctly by filtering on the tags log_type contains kubernetes, environemt contains tanzu_k8s_grid and the tkg_cluster contains the name of my TKG cluster:
Log Insight Cloud with filters for TKG content
As you can see, the logs are flowing! It took me a little bit of tweaking and re-trying to get this right - hopefully it will save you a moment or two!
namespace:"tanzu-system-logging"fluent_bit:config:service:| [Service]
Flush 1
Log_Level info
Daemon off
Parsers_File parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020outputs:| [OUTPUT]
Name http
Match *
Host data.mgmt.cloud.vmware.com
Port 443
URI /le-mans/v1/streams/ingestion-pipeline-stream
Header Authorization Bearer <My API Key>
Format json
tls On
tls.verify Oninputs:| [INPUT]
Name tail
Tag kube.*
Path /var/log/containers/*.log
Parser cri
DB /var/log/flb_kube.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
[INPUT]
Name systemd
Tag kube_systemd.*
Path /var/log/journal
DB /var/log/flb_kube_systemd.db
Systemd_Filter _SYSTEMD_UNIT=kubelet.service
Systemd_Filter _SYSTEMD_UNIT=containerd.service
Read_From_Tail On
Strip_Underscores On
[INPUT]
Name tail
Tag apiserver_audit.*
Path /var/log/kubernetes/audit.log
Parser json
DB /var/log/flb_kube_audit.db
Mem_Buf_Limit 50MB
Refresh_Interval 10
Skip_Long_Lines On
[INPUT]
Name tail
Tag audit.*
Path /var/log/audit/audit.log
Parser logfmt
DB /var/log/flb_system_audit.db
Mem_Buf_Limit 50MB
Refresh_Interval 10
Skip_Long_Lines Onfilters:| [FILTER]
Name record_modifier
Match *
Record environment tanzu_k8s_grid
Record log_type kubernetes
Record tkg_cluster <Cluster Name>
Record tkg_instance <vCenter Name>
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc:443
Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
Kube_Tag_Prefix kube.var.log.containers.
Merge_Log On
Merge_Log_Key log_processed
K8S-Logging.Parser On
K8S-Logging.Exclude Off
[FILTER]
Name modify
Match kube.*
Copy kubernetes k8s
[FILTER]
Name nest
Match kube.*
Operation lift
Nested_Under kubernetesparsers:| [PARSER]
Name apache
Format regex
Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name apache2
Format regex
Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name apache_error
Format regex
Regex ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$
[PARSER]
Name nginx
Format regex
Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name json
Format json
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
[PARSER]
Name docker-daemon
Format regex
Regex time="(?<time>[^ ]*)" level=(?<level>[^ ]*) msg="(?<msg>[^ ].*)"
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
[PARSER]
# http://rubular.com/r/tjUt3Awgg4
Name cri
Format regex
Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
[PARSER]
Name logfmt
Format logfmt
[PARSER]
Name syslog-rfc5424
Format regex
Regex ^\<(?<pri>[0-9]{1,5})\>1 (?<time>[^ ]+) (?<host>[^ ]+) (?<ident>[^ ]+) (?<pid>[-0-9]+) (?<msgid>[^ ]+) (?<extradata>(\[(.*)\]|-)) (?<message>.+)$
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
[PARSER]
Name syslog-rfc3164-local
Format regex
Regex ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
Time_Key time
Time_Format %b %d %H:%M:%S
Time_Keep On
[PARSER]
Name syslog-rfc3164
Format regex
Regex /^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$/
Time_Key time
Time_Format %b %d %H:%M:%S
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
[PARSER]
Name kube-custom
Format regex
Regex (?<tag>[^.]+)?\.?(?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<docker_id>[a-z0-9]{64})\.log$streams:""plugins:""daemonset:resources:{}podAnnotations:{}podLabels:{}
Sam McGeown
