Logging

Zalando cluster will ship logs to Scalyr for all containers running on a cluster node. The logs will include extra attributes/tags/metadata depending on deployment manifests. Whenever a new container starts on a cluster node, its logs will be shipped.

Note

Logs are shipped per container and not per application. To view all logs from certain application you can use Scalyr UI https://www.scalyr.com/events and filter using Logs attributes.

One Scalyr account will be provisioned for each community, i.e. the same Scalyr account is used for both test and production clusters.

You need to make sure the minimum requirements are satisfied to start viewing logs on Scalyr.

Requirements

Logging output

Always make sure your application logs to stdout & stderr. This will allow cluster log shipper to follow application logs, and also allows you to follow logs via Kubernetes native logs command.

$ zkubectl logs -f my-pod-name my-container-name

Labels

In order for the container logs to be shipped, your deployment must include the following metadata labels:

  • application
  • version

Logs attributes

All logs are shipped with extra attributes that can help in filtering from Scalyr UI (or API). Usually those extra fields are extracted from deployment labels, or the Kubernetes cluster/API.

application
Application ID. Retrieved from metadata labels.
version
Application version. Retrieved from metadata labels.
release
Application release. Retrieved from metadata labels. [optional]
cluster
Cluster ID. Retrieved from Kubernetes cluster.
container
Container name. Retrieved from Kubernetes API.
node
Cluster node running this container. Retrieved from Kubernetes cluster.
pod
Pod name running the container. Retrieved from Kubernetes cluster.
namespace
Namespace running this deployment(pod). Retrieved from Kubernetes cluster.

Log parsing

The default parser for application logs is the json parser. In some cases however you might want to use a custom Scalyr parser for your application. This can be achieved via Pod annotations.

However, the json parser only parses the JSON generated from the Docker logs. If your application generates logs in JSON, the default parser will only see them as an escaped string of JSON. However, Scalyr provides a special parser escapedJson for that.

Scalyr’s default parser can even be configured to also make a pass with the escapedJson parser. That way there is no need to configure anything on a per application level to get properly parsed fields from JSON based application logs in Scalyr. Just edit the JSON parser to contain the following config.

// Parser for log files containing JSON records.
{
   attributes: {
     // Tag all events parsed with this parser so we can easily select them in queries.
     dataset: "json"
   },

   formats: [
     {format: "${parse=json}$", repeat: true},
     {format: "\\{\"log\":\"$log{parse=escapedJson}$", repeat: true}
   ]
 }

The following example shows how to annotate a pod to instruct the log watcher to use the custom parser json-java-parser for pod container my-app.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 3
  template:
    metadata:
      labels:
        application: my-app
      annotations:
        # specify scalyr log parser
        kubernetes-log-watcher/scalyr-parser: '[{"container": "my-app-container", "parser": "json-java-parser"}]'
    spec:
      containers:
      - name: my-app-container
        image: pierone.stups.zalan.do/myteam/my-app:cd53
        ports:
        - containerPort: 8080

The value of kubernetes-log-watcher/scalyr-parser annotation should be a JSON serialized list. If container value did not match, then it will fall back to the default parser (i.e. json).

Note

You need to specify the container in the parser annotation because you can have multiple containers in a pod which may use different log formats.