Datadog Integration
Account Setup
To set up an account with Datadog, open the Datadog website and click on Free Trial. Select the appropriate region for storing data. Then fill out the remaining details and follow their instructions.
To complete the sign-up process, you have to install one of their agents and have it send data. The agent isn’t required for the Observability Kit integration, so it can be removed immediately afterwards.
The last sign-up step provides instructions for installing the agent. An easy way to run the agent once is to use the Docker option. This allows you to create a container, send some data, and then stop and remove the container immediately afterwards.
OpenTelemetry Collector Setup
Datadog doesn’t support the OpenTelemetry APIs directly. Instead, the OpenTelemetry collector must be used as a proxy for converting data into the Datadog format and forwarding it to their API endpoints.
Start by creating a collector.yaml
configuration file with the following content:
receivers:
otlp:
protocols:
http:
grpc:
processors:
batch:
exporters:
datadog:
api:
site: DD_SITE
key: DD_API_KEY
service:
pipelines:
metrics:
receivers: [otlp]
processors: [batch]
exporters: [datadog]
traces:
receivers: [otlp]
processors: [batch]
exporters: [datadog]
Replace the following placeholders in the configuration:
-
DD_SITE
- The domain of the Datadog region for which you registered. You can get this from the browsers address bar when logged into the Datadog UI. For example, for the US1 region the value should bedatadoghq.com
; for the EU region,datadoghq.eu
. -
DD_API_KEY
- An API key for authenticating against the Datadog API. To create an API key, open the Datadog web UI, click onOrganization Setting
and thenAPI Keys
. Create a new key. Make sure to save it, and then insert it into the configuration here.
You can run the collector either by downloading and running the binary manually, or by using Docker.
Download the binary from the latest release on GitHub. Make sure to download the otelcol_contrib_
variant. It includes support for Datadog and the correct variant for your operating system.
Next, run the collector using the following command:
otelcontribcol_linux_amd64 --config collector.yaml
Logging Setup
Datadog doesn’t support collecting logs in the OpenTelemetry format. You can use the following work-arounds for collecting log data, and correlating them with the traces collected by Observability Kit.
Logback configuration
This method uses a Logback appender to send logs to the Datadog TCP logs API, while also adding the necessary Datadog metadata to correlate logs to traces.
First, add the following dependency to your Maven configuration to include a TCP appender for Logback:
<dependency>
<groupId>net.logstash.logback</groupId>
<artifactId>logstash-logback-encoder</artifactId>
<version>7.2</version>
</dependency>
To correlate logs to traces collected by Observability Kit, you need to customize the JSON output of Logback. To do that, create a DataDogContextProvider
class with the following content:
package com.example.application;
import ch.qos.logback.classic.spi.ILoggingEvent;
import com.fasterxml.jackson.core.JsonGenerator;
import net.logstash.logback.composite.AbstractJsonProvider;
import java.io.IOException;
import java.util.Map;
public class DataDogContextProvider extends AbstractJsonProvider<ILoggingEvent> {
@Override
public void writeTo(JsonGenerator generator, ILoggingEvent iLoggingEvent) throws IOException {
Map<String, String> mdcPropertyMap = iLoggingEvent.getMDCPropertyMap();
if (mdcPropertyMap.containsKey("trace_id")) {
String traceId = mdcPropertyMap.get("trace_id");
String traceIdHexString = traceId.substring(traceId.length() - 16 );
long datadogTraceId = Long.parseUnsignedLong(traceIdHexString, 16);
String datadogTraceIdString = Long.toUnsignedString(datadogTraceId);
generator.writeStringField("dd.trace_id", datadogTraceIdString);
System.out.println("dd.trace_id: " + datadogTraceIdString);
}
if (mdcPropertyMap.containsKey("span_id")) {
String spanId = mdcPropertyMap.get("span_id");
String spanIdHexString = spanId.substring(spanId.length() - 16 );
long datadogSpanId = Long.parseUnsignedLong(spanIdHexString, 16);
String datadogSpanIdString = Long.toUnsignedString(datadogSpanId);
generator.writeStringField("dd.span_id", datadogSpanIdString);
}
}
}
This class checks if the log event has the trace_id
and span_id
attributes. It writes corresponding Datadog trace and span IDs to the JSON output.
Next, create a custom Logback encoder that makes use of this JSON provider. The encoder extends from LogstashEncoder
, which is an encoder that generates JSON output that can be processed by the Datadog logging endpoint:
package com.example.application;
import ch.qos.logback.classic.spi.ILoggingEvent;
import net.logstash.logback.LogstashFormatter;
import net.logstash.logback.composite.AbstractCompositeJsonFormatter;
import net.logstash.logback.encoder.LogstashEncoder;
public class DataDogLogstashEncoder extends LogstashEncoder {
@Override
protected AbstractCompositeJsonFormatter<ILoggingEvent> createFormatter() {
AbstractCompositeJsonFormatter<ILoggingEvent> formatter = super.createFormatter();
((LogstashFormatter)formatter).addProvider(new DataDogContextProvider());
return formatter;
}
}
Next add a new TCP appender to your Logback configuration, using the encoder created above, like so:
<appender name="JsonTcp" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
<destination>tcp-intake.logs.${DD_SITE}:443</destination>
<keepAliveDuration>20 seconds</keepAliveDuration>
<encoder class="com.example.application.DataDogLogstashEncoder">
<prefix class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="ch.qos.logback.classic.PatternLayout">
<pattern>${DD_API_KEY} %mdc{keyThatDoesNotExist}</pattern>
</layout>
</prefix>
</encoder>
<ssl />
</appender>
The configuration has two variables that need to be passed as system variables when running the Java agent:
-
DD_SITE
- The domain of the Datadog region for which you registered. This should have the same value as in the collector configuration from earlier. -
DD_API_KEY
- An API key for authenticating against the Datadog API. This should have the same value as in the collector configuration from earlier.
For example, to pass these system variables when running the application, use this:
java -DDD_SITE=datadoghq.eu -DDD_API_KEY=1234ABCD ...
Finally, register the appender for the root logger to take it into effect. You would do that like so:
<root level="info">
<appender-ref ref="JsonTcp" />
<!-- ...other appenders -->
</root>
Observability Kit Configuration
To configure Observability Kit, create an agent.properties
file with the following content:
otel.service.name=vaadin
otel.traces.exporter=otlp
otel.metrics.exporter=otlp
In the configuration above, the service name is defined as vaadin
, which is also the default of the agent. The service name can be customized.
By default, the OpenTelemetry Protocol (OTLP) exporters assume that the OpenTelemetry collector is running locally. Therefore, you don’t need to configure endpoints. If the collector is running on a different system, you need to configure an endpoint for the exporter:
otel.exporter.otlp.endpoint=https://collector.my-domain.net:4317
Running the App
Run the application using the Java binary and pass the respective arguments for the Java agent and the agent configuration:
java -javaagent:PATH/TO/vaadin-opentelemetry-javaagent-VERSION.jar \
-Dotel.javaagent.configuration-file=PATH/TO/agent.properties \
-jar myapp.jar
Note
|
Replace placeholder paths and version
Remember to correct the path to the agent.properties file, as well as the path and version of the Observability Kit .jar file.
|
Viewing Data
Viewing Traces
In the Datadog web UI, select APM
and then Traces
from the menu on the left. The view provides graphs of the number of requests, failed requests and latency, as well as a list of the latest traces.
The search bar allows filtering for tags, which are attribute values of spans. To get an idea of what kind of tags can be filtered, click on a trace. This displays a detailed view with all tags recorded for the span. Clicking on a tag will present an option on which to filter that specific tag value.
Clicking on a trace ID will display a side panel that shows detailed information about a specific trace, such as nested spans, and attributes and related logs.
See Datadog documentation for Trace Explorer for more information on how to use this view.
Viewing Metrics
In the Datadog web UI, select Metrics
and then Explorer
from the menu on the left.
In the metrics input field, enter one of the metrics collected by Observability Kit. For example, use otel.process.runtime.jvm.memory.usage
to view the JVM memory usage.
See Datadog documentation for the Metrics Explorer for more information on how to use this view.
Viewing Logs
Logging integration is limited to the Logback logging framework.
In the Datadog web UI, select Logs
from the menu on the left. The view shows a list of recent log entries, which can be filtered by text or log level.
See Datadog documentation for the Logs Explorer for more information on how to use this view.