Forwarding Tanzu Kubernetes Grid logs to vRealize Log Insight Cloud using the Fluent-bit Package

Written by Sam McGeown on 13/12/2021 · Read in about 5 min (940 words)
Published under Cloud-Native and VMware

In my previous post I walked through configuring kubernetes ingress with automatically generated SSL certificates and DNS registration using Tanzu Kubernetes Grid’s Packages. Another of the packaged applications available is Fluent-bit, which enables log forwarding from your Kubernetes cluster and workloads to a range of supported logging endpoints.

There are a couple of tweaks required in order to forward logs to vRealize Automation Log Insight Cloud. We need to use the HTTP output in the Fluent Bit configuration to forward the logs as a JSON payload to the Log Insight API. We also need to append an authentication header with an API key, and configure some tags so that the Tanzu Kubnernetes Grid Content Pack will pick up the logs.

I covered the pre-requisites for deploying Tanzu Packages in my previous post so I’ll assume you’ve got them covered.

To deploy the Fluent-bit package, first list the available versions and then install the package using a YAML configuration file. Fluent-bit will be installed in the tanzu-system-logging namespace by default.

1
2
3
4
5
6
7
8
9
$ tanzu package available list fluent-bit.tanzu.vmware.com -A

| Retrieving package versions for fluent-bit.tanzu.vmware.com...
  NAME                         VERSION               RELEASED-AT           NAMESPACE 
  fluent-bit.tanzu.vmware.com  1.7.5+vmware.1-tkg.1  2021-05-13T18:00:00Z  default

$ tanzu package install fluent-bit --package-name fluent-bit.tanzu.vmware.com \
      --version 1.7.5+vmware.1-tkg.1 --values-file fluent-bit-data-values.yaml \
      --create-namespace

The YAML configuration file I used (fluent-bit-data-values.yaml) is below, which is mostly standard apart from the tweaks for Log Insight Cloud.

  • Lines 15-23 configure the HTTP output with the Log Insight address, API key authentication and JSON formatting
  • Lines 68-69 add a filter to tag environment and log_type for the TKG content pack
  • Lines 70-72 add a tag for the tkg_cluster and tkg_instance

Once the package has been deployed you can check it’s sending logs and tagging them correctly by filtering on the tags log_type contains kubernetes, environemt contains tanzu_k8s_grid and the tkg_cluster contains the name of my TKG cluster:

Log Insight Cloud with filters for TKG content
Log Insight Cloud with filters for TKG content

As you can see, the logs are flowing! It took me a little bit of tweaking and re-trying to get this right - hopefully it will save you a moment or two!

fluent-bit-data-values.yaml

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
namespace: "tanzu-system-logging"
fluent_bit:
  config:
    service: |
      [Service]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020      
    outputs: |
      [OUTPUT]
        Name              http
        Match             *
        Host              data.mgmt.cloud.vmware.com
        Port              443
        URI               /le-mans/v1/streams/ingestion-pipeline-stream
        Header            Authorization Bearer <My API Key>
        Format            json
        tls               On
        tls.verify        On      
    inputs: |
      [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            cri
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10

      [INPUT]
        Name                systemd
        Tag                 kube_systemd.*
        Path                /var/log/journal
        DB                  /var/log/flb_kube_systemd.db
        Systemd_Filter      _SYSTEMD_UNIT=kubelet.service
        Systemd_Filter      _SYSTEMD_UNIT=containerd.service
        Read_From_Tail      On
        Strip_Underscores   On

      [INPUT]
        Name              tail
        Tag               apiserver_audit.*
        Path              /var/log/kubernetes/audit.log
        Parser            json
        DB                /var/log/flb_kube_audit.db
        Mem_Buf_Limit     50MB
        Refresh_Interval  10
        Skip_Long_Lines   On

      [INPUT]
        Name              tail
        Tag               audit.*
        Path              /var/log/audit/audit.log
        Parser            logfmt
        DB                /var/log/flb_system_audit.db
        Mem_Buf_Limit     50MB
        Refresh_Interval  10
        Skip_Long_Lines   On      
    filters: |
      [FILTER]
        Name                record_modifier
        Match               *
        Record environment  tanzu_k8s_grid
        Record log_type     kubernetes
        Record tkg_cluster  <Cluster Name>
        Record tkg_instance <vCenter Name>

      [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off

      [FILTER]
        Name                  modify
        Match                 kube.*
        Copy                  kubernetes k8s

      [FILTER]
        Name                  nest
        Match                 kube.*
        Operation             lift
        Nested_Under          kubernetes      
    parsers: |
      [PARSER]
          Name   apache
          Format regex
          Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
          Time_Key time
          Time_Format %d/%b/%Y:%H:%M:%S %z

      [PARSER]
          Name   apache2
          Format regex
          Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
          Time_Key time
          Time_Format %d/%b/%Y:%H:%M:%S %z

      [PARSER]
          Name   apache_error
          Format regex
          Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$

      [PARSER]
          Name   nginx
          Format regex
          Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
          Time_Key time
          Time_Format %d/%b/%Y:%H:%M:%S %z

      [PARSER]
          Name   json
          Format json
          Time_Key time
          Time_Format %d/%b/%Y:%H:%M:%S %z

      [PARSER]
          Name        docker
          Format      json
          Time_Key    time
          Time_Format %Y-%m-%dT%H:%M:%S.%L
          Time_Keep   On

      [PARSER]
          Name        docker-daemon
          Format      regex
          Regex       time="(?<time>[^ ]*)" level=(?<level>[^ ]*) msg="(?<msg>[^ ].*)"
          Time_Key    time
          Time_Format %Y-%m-%dT%H:%M:%S.%L
          Time_Keep   On

      [PARSER]
          # http://rubular.com/r/tjUt3Awgg4
          Name cri
          Format regex
          Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
          Time_Key    time
          Time_Format %Y-%m-%dT%H:%M:%S.%L%z

      [PARSER]
          Name        logfmt
          Format      logfmt

      [PARSER]
          Name        syslog-rfc5424
          Format      regex
          Regex       ^\<(?<pri>[0-9]{1,5})\>1 (?<time>[^ ]+) (?<host>[^ ]+) (?<ident>[^ ]+) (?<pid>[-0-9]+) (?<msgid>[^ ]+) (?<extradata>(\[(.*)\]|-)) (?<message>.+)$
          Time_Key    time
          Time_Format %Y-%m-%dT%H:%M:%S.%L
          Time_Keep   On

      [PARSER]
          Name        syslog-rfc3164-local
          Format      regex
          Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
          Time_Key    time
          Time_Format %b %d %H:%M:%S
          Time_Keep   On

      [PARSER]
          Name        syslog-rfc3164
          Format      regex
          Regex       /^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$/
          Time_Key    time
          Time_Format %b %d %H:%M:%S
          Time_Format %Y-%m-%dT%H:%M:%S.%L
          Time_Keep   On

      [PARSER]
          Name    kube-custom
          Format  regex
          Regex   (?<tag>[^.]+)?\.?(?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<docker_id>[a-z0-9]{64})\.log$      
    streams: ""
    plugins: ""
  daemonset:
    resources: { }
    podAnnotations: { }
    podLabels: { }
Share this post