Benthos plugins

1. Ouput to clickhouse

2. Processor to process geo (Using maxmind database)

3. Processor to parse user agent

Build

 go build -o cmd/benthos .

1. Benthos output plugin to clickhouse

• Using clickhouse go client

Example

• Create clickhouse table

CREATE TABLE IF NOT EXISTS
    sample(hitmiss String, client_ip String, status Int32, timestamp DateTime)
    engine=Memory;

• Using plugin

...
output:
  type: clickhouse
  plugin:
    connection_string: "tcp://localhost:9000"
    query: "insert into sample(hitmiss, client_ip, status, timestamp) values(?, ?, ?, ?)"
    columns:
      - hitmiss
      - client_ip
      - status$floatToInt32
      - timestamp$stringToDateOrNow$2006-01-02T15:04:05.000Z

• Conncetion string : The connection string

• Query : insert query

• Columns: Columns in jsonfields tobe inserted

  • Supported converter: stringToInt32, floatToInt32, floatToUInt32, floatToUInt8, stringToDateOrNow, unixToDateOrNow

2. Process geo location

...
    - type: geo
      plugin:
        file: GeoLite2-City.mmdb
        field: client_ip
...

• file: maxmind mmdb file
• field: field to process

3. Parse user agent

...
    - type: useragent
      plugin:
        file: regexes.yaml
        field: user_agent
...

• file: regex of useragent
• field: field to process