Troubleshooting Tools for Microservices Architecture

25 Mar 2019  Sergio Martin Rubio  6 mins read.

One of the biggest challenges when transitioning to a microservices architecture is troubleshooting and debugging. When the number of microservices grows, a simple HTTP request can hit dozens of applications, and in case something goes wrong or the performance is not as good as expected, it might be quite tricky to know where the issue is.

Logging and instrumentation are very important tools to understand what is going on in a microservices architecture.

One of the most famous tools to capture and process logs is Splunk. Basically, this tool allows us to centralize and manage application logs. It also provides metrics, generates reports and alerts for a particular search, and offers a friendly user interface.

On the other hand, for instrumentation we can use Zipkin, which provides a distributed tracing system, and will help you troubleshoot latency issues.

In order to understand how these tools can help us, we are going to build a Spring Boot application which will be fully integrated with Zipkin through Spring Cloud Sleuth.


This web application will only contain an endpoint which will log a simple string with Slf4j.

public class SpringBootZipkinSleuthSplunkApplication {
    public static void main(String[] args) {, args);
    public String helloWorld() {"Hello World Endpoint");
        return "Hello World!";

The file will contain the configuration for Sleuth and Zipkin.

Note: In order to explicitly provide a different service name for all spans coming from your application we can set to the desired name.

spring.sleuth.sampler.probability value is 1, which is 100% (default: 0.1, which is 10 percent)

localhost cannot be used in this example, since the Spring Boot application will run inside a Docker container, so we have to use the name specified for Zipkin on our docker compose file (http://zipkin:9411)

Sleuth sends its tracing data to Zipkin by default, if the following dependency is added to your project.


This library is also responsible for adding the trace IDs and store them locally in order to continue the trace.

A Logback configuration file is also necessary in order to visualize the traces ID in Splunk.

    <include resource="org/springframework/boot/logging/logback/base.xml"/><springProperty scope="context" name="appName" source=""/><appender name="logstash" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
        <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
                        "severity": "%level",
                        "service": "${appName:-}",
                        "trace": "%X{X-B3-TraceId:-}",
                        "span": "%X{X-B3-SpanId:-}",
                        "parent": "%X{X-B3-ParentSpanId:-}",
                        "thread": "%thread",
                        "class": "%logger{40}",
                        "message": "%message"
    </appender><root level="INFO">
        <appender-ref ref="logstash"/>

Zipkin defines three kind of ids:

  • Trace Id: ID shared by every span in a trace
  • Span Id: ID of a particular span which might be the same as the trace id if only one service is involved during the tracing.
  • Parent Id: ID present on child spans to know which was the previous span ID. If the span does not have a parent id is considered the root of the trace.

Finally, here is the docker-compose.yml that powers the whole setup.

version: '2'
    image: com.thedeveloperhive/spring-boot-zipkin-sleuth-splunk
      - LOGGING_FILE=/logs/app.log
      - '8080:8080'
      - log_volume:/logs
    image: splunk/splunk
    hostname: splunk
      - SPLUNK_START_ARGS=--accept-license
      - SPLUNK_PASSWORD=password
      - '8000:8000'
    image: splunk/universalforwarder:6.5.3-monitor
    hostname: splunkforwarder
      - SPLUNK_START_ARGS=--accept-license --answer-yes
      - SPLUNK_FORWARD_SERVER=splunk:9997
      - SPLUNK_ADD=monitor /logs
      - SPLUNK_PASSWORD=password
    restart: always
      - splunk
      - log_volume:/logs
    image: openzipkin/zipkin
      - '9411:9411'

Note: Splunk works on the client-server model. Splunk Forwarder is used to collect the machine generated data from client side and forward to Splunk server.

Running Application

Build application and run docker compose:

mvn clean install
docker-compose up

Note: A Docker image will be built during the Maven install stage, since the Spotify docker-maven-plugin was added on the pom file.

If we hit the web service at http://localhost:8080 a log entry will be generated and forwarded to Splunk and Zipkin.

app_1 | 2019-03-22 21:55:35.647  INFO [-,73d169f5b5e76599,73d169f5b5e76599,false] 1 --- [or-http-epoll-3] .SpringBootZipkinSleuthSplunkApplication : Hello World Endpoint

As we can see the trace ID is the same as the span ID, since only one service was involved during the tracing, and there is no parent ID.

Zipkin interface is available at http://localhost:9411

Zipkin UI.
Zipkin UI.

Now login into Splunk web console (http://localhost:8000) and search for “73d169f5b5e76599”.

Splunk UI
Splunk UI


Both logging and instrumentation are essential in any enterprise microservices architecture, and tools like Splunk and Zipkin can be excellent allies to act fast and precisely when issues arise.

Source Code