Structured Logging and Correlation

Overview

Structured logging is the practice of writing log events as machine-readable data instead of only plain text. In .NET, this usually means writing logs through ILogger<T> with named message-template placeholders such as {OrderId}, {UserId}, or {ElapsedMilliseconds}. The log message still has a readable template, but the values are also captured as separate properties that can be searched, filtered, grouped, and analyzed by log systems.

Correlation is the practice of connecting log events that belong to the same logical operation. In a simple application, this could mean adding a RequestId or CorrelationId to every log created while handling one HTTP request. In a distributed system, it often means propagating trace context across services so that logs, traces, and sometimes metrics can be connected using identifiers such as TraceId, SpanId, request IDs, or business transaction IDs.

This topic matters because production debugging is rarely about one isolated log line. Real incidents usually require answering questions such as:

• Which request caused this exception?
• Which user, order, tenant, or file transfer was involved?
• Which downstream service failed?
• Did the same request produce warnings in another service?
• Can we search all logs for this transaction without manually guessing timestamps?

For interviews, structured logging and correlation are important because they show whether a developer can build observable, supportable applications. A strong candidate should understand not only how to call LogInformation, but also how to design useful log messages, choose appropriate log levels, avoid sensitive data leaks, propagate correlation IDs, use scopes, work with distributed tracing, and make logs useful during real production incidents.

Core Concepts

Logging, Observability, and Diagnostics

Logging is one part of observability. Observability usually includes logs, metrics, and traces:

• Logs are timestamped events that describe something that happened.
• Metrics are numeric measurements aggregated over time, such as request count, error rate, or CPU usage.
• Traces show the path of a request or operation across components, often split into spans.

Structured logging improves logs by making them queryable. Correlation improves logs by connecting related events. Together, they allow developers and support teams to move from isolated messages to a full picture of what happened.

Example plain-text log:

Payment failed for order 12345 because gateway timeout

Example structured log template:

_logger.LogWarning(
    "Payment failed for order {OrderId}. Reason: {Reason}",
    orderId,
    reason);

The rendered message is readable, but the logging provider can also store OrderId and Reason as separate searchable fields.

ILogger<T> and Log Categories

The common .NET logging abstraction is Microsoft.Extensions.Logging.ILogger<T>. It is usually injected through dependency injection:

public sealed class OrderService
{
    private readonly ILogger<OrderService> _logger;

    public OrderService(ILogger<OrderService> logger)
    {
        _logger = logger;
    }

    public async Task SubmitAsync(Guid orderId, CancellationToken cancellationToken)
    {
        _logger.LogInformation("Submitting order {OrderId}", orderId);

        await Task.Delay(100, cancellationToken);

        _logger.LogInformation("Order {OrderId} submitted", orderId);
    }
}

ILogger<T> uses the type name as the log category. This helps filter logs by namespace or class, for example:

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning",
      "MyCompany.Payments": "Debug"
    }
  }
}

In interviews, it is useful to mention that the logger abstraction separates application code from the concrete logging provider. The application can use ILogger<T>, while the host can configure Console, Debug, Application Insights, OpenTelemetry, Serilog, NLog, or another provider.

Log Levels

Log levels describe event severity and importance. Common .NET log levels are:

Good logging is not about logging everything. It is about logging the right information at the right level.

Example:

_logger.LogDebug("Querying product cache for {ProductId}", productId);

_logger.LogInformation("Created order {OrderId} for customer {CustomerId}", orderId, customerId);

_logger.LogWarning("Inventory is low for product {ProductId}. Remaining: {RemainingQuantity}", productId, quantity);

_logger.LogError(exception, "Failed to create order {OrderId}", orderId);

Common mistakes:

• Logging normal expected validation errors as Error.
• Logging every step of a high-volume code path at Information.
• Using Critical for ordinary business failures.
• Writing logs without enough context to diagnose the issue.

Structured Logging and Message Templates

Structured logging in .NET usually uses message templates:

_logger.LogInformation(
    "Processed file {FileName} with {RecordCount} records in {ElapsedMilliseconds} ms",
    fileName,
    recordCount,
    elapsedMilliseconds);

The placeholders are not the same as string interpolation. They become named properties.

Prefer this:

_logger.LogInformation("User {UserId} logged in", userId);

Avoid this:

_logger.LogInformation($"User {userId} logged in");

The interpolated version creates a final string before the logger receives it. The logging system cannot reliably capture UserId as a separate structured property.

Good placeholder names should be stable, descriptive, and consistent:

_logger.LogInformation(
    "Payment authorized for order {OrderId} using provider {PaymentProvider}",
    orderId,
    paymentProvider);

Avoid inconsistent names:

_logger.LogInformation("Payment authorized for {Id}", orderId);
_logger.LogInformation("Payment captured for {Order}", orderId);
_logger.LogInformation("Payment refunded for {OrderNumber}", orderId);

If these all refer to the same concept, use the same property name, such as {OrderId}.

Structured Properties vs Rendered Messages

A log event has at least two audiences:

1. Humans reading the message.
2. Machines indexing fields.

For example:

_logger.LogWarning(
    "Customer {CustomerId} exceeded payment retry limit. AttemptCount: {AttemptCount}",
    customerId,
    attemptCount);

The rendered message helps a human understand the event quickly. The structured fields let a log platform answer questions such as:

• Show all logs for CustomerId = 42.
• Count warnings by AttemptCount.
• Find all payment retry-limit failures in the last hour.

This is one of the main interview points: structured logging is not just a style choice; it directly affects production support and incident response.

Correlation IDs

A correlation ID is an identifier used to connect related logs for one operation.

In an ASP.NET Core API, a correlation ID is commonly:

• Generated at the edge if the request does not provide one.
• Read from a header such as X-Correlation-ID if provided by a trusted caller.
• Added to response headers so clients can report it.
• Added to all logs created during the request.
• Passed to downstream HTTP calls, queues, or messages.

Simple middleware example:

public sealed class CorrelationIdMiddleware
{
    private const string HeaderName = "X-Correlation-ID";
    private readonly RequestDelegate _next;
    private readonly ILogger<CorrelationIdMiddleware> _logger;

    public CorrelationIdMiddleware(
        RequestDelegate next,
        ILogger<CorrelationIdMiddleware> logger)
    {
        _next = next;
        _logger = logger;
    }

    public async Task InvokeAsync(HttpContext context)
    {
        string correlationId = context.Request.Headers.TryGetValue(HeaderName, out var values)
            ? values.FirstOrDefault() ?? Guid.NewGuid().ToString("N")
            : Guid.NewGuid().ToString("N");

        context.Response.Headers[HeaderName] = correlationId;

        using (_logger.BeginScope(new Dictionary<string, object>
        {
            ["CorrelationId"] = correlationId
        }))
        {
            await _next(context);
        }
    }
}

Registering the middleware:

app.UseMiddleware<CorrelationIdMiddleware>();

This pattern makes every log inside the request scope include the same CorrelationId, as long as the provider supports scopes and is configured to include them.

BeginScope and Contextual Logging

ILogger.BeginScope attaches contextual properties to all logs within a logical operation.

Example:

using (_logger.BeginScope(new Dictionary<string, object>
{
    ["OrderId"] = orderId,
    ["CustomerId"] = customerId
}))
{
    _logger.LogInformation("Starting order checkout");
    await _paymentService.AuthorizeAsync(orderId, cancellationToken);
    _logger.LogInformation("Finished order checkout");
}

Instead of repeating OrderId and CustomerId in every message, the scope can enrich all log events produced inside it.

Scopes are useful for:

• Request IDs.
• Correlation IDs.
• Tenant IDs.
• User IDs.
• Job IDs.
• Message IDs.
• Business transaction IDs.

However, scopes should be used carefully. A scope that contains too much data can make every log event expensive or risky. A scope should contain small, stable identifiers rather than large objects or sensitive data.

Request ID, Correlation ID, Trace ID, and Span ID

These terms are related but not identical.

In simple systems, CorrelationId and TraceId may feel similar. In distributed tracing, the trace ID is part of a standardized trace context. A custom correlation ID can still be useful when the business wants a stable transaction ID that exists outside tracing tools.

Good practical approach:

• Use distributed tracing identifiers such as TraceId and SpanId for technical trace correlation.
• Use a business correlation ID such as OrderId, TransferId, MessageId, or CorrelationId when support teams and clients need a stable reference.
• Avoid inventing many unrelated IDs that duplicate each other without a clear purpose.

Distributed Tracing and Activity

In .NET, distributed tracing uses System.Diagnostics.Activity to represent units of work. ASP.NET Core and many libraries can create activities automatically for inbound and outbound calls.

During a request, logs can be connected to the current activity through values such as:

• TraceId
• SpanId
• ParentId

Example:

using System.Diagnostics;

_logger.LogInformation(
    "Current trace: {TraceId}, span: {SpanId}",
    Activity.Current?.TraceId,
    Activity.Current?.SpanId);

For most application code, you should not manually log the trace IDs in every message. Instead, configure the logging provider or observability pipeline to include activity tracking properties automatically.

W3C Trace Context

W3C Trace Context is a standard way to propagate tracing information between services, commonly through HTTP headers such as traceparent and tracestate.

For example, when Service A calls Service B:

1. Service A receives or creates a trace context.
2. Service A logs messages with the current trace context.
3. Service A sends an HTTP request to Service B with trace headers.
4. Service B reads the headers and continues the same trace.
5. Logs in both services can be connected by the same trace ID.

This is more reliable than manually passing only a custom header in modern distributed systems because many frameworks and observability tools understand trace context.

OpenTelemetry and Log Correlation

OpenTelemetry is a vendor-neutral observability standard and ecosystem. In .NET, OpenTelemetry can collect traces, metrics, and logs and export them to backends such as Azure Monitor, Grafana, Jaeger, Zipkin, Datadog, or other systems.

A common production design is:

Application code -> ILogger<T> -> logging provider/exporter -> observability backend
Application code -> Activity/OpenTelemetry tracing -> exporter -> observability backend

When log correlation is enabled, logs can include trace fields such as TraceId and SpanId. This allows a developer to move from a log event to the full trace, or from a trace span to related logs.

The important interview point is that OpenTelemetry does not remove the need for good log design. It can carry and correlate telemetry, but developers still need to choose useful log levels, message templates, properties, and redaction rules.

Logging in ASP.NET Core

ASP.NET Core has built-in logging integration. Common examples include:

• Request logs.
• Hosting lifetime logs.
• Middleware logs.
• Controller or endpoint logs.
• Entity Framework Core logs.
• Application service logs.

Typical usage in a controller or minimal API service:

app.MapPost("/orders/{orderId:guid}/submit", async (
    Guid orderId,
    OrderService orderService,
    ILogger<Program> logger,
    CancellationToken cancellationToken) =>
{
    logger.LogInformation("Submitting order {OrderId}", orderId);

    await orderService.SubmitAsync(orderId, cancellationToken);

    return Results.Accepted($"/orders/{orderId}");
});

For production APIs, logs are most useful when they include:

• Request path or endpoint name.
• Status code.
• Duration.
• Correlation or trace ID.
• Authenticated user or tenant identifier when safe and appropriate.
• Business identifiers such as order ID, payment ID, or file transfer ID.

HTTP Logging and Payload Logging

ASP.NET Core can log HTTP request and response information. This can be useful, but it must be used carefully.

HTTP logging may include:

• Request method.
• Path.
• Query string.
• Headers.
• Status code.
• Duration.
• Request or response body, if enabled.

Payload logging is risky because request and response bodies may contain:

• Passwords.
• Tokens.
• Personal information.
• Payment data.
• Health information.
• Confidential business data.

Best practices:

• Do not log request or response bodies by default.
• Prefer logging identifiers and outcomes instead of full payloads.
• Use allowlists for safe fields rather than relying only on blocklists.
• Redact or omit sensitive fields before logs leave the application.
• Be careful with headers such as Authorization, Cookie, and API keys.
• Test the performance impact of HTTP logging.

Example of safer application-level logging:

_logger.LogInformation(
    "Create customer request received for tenant {TenantId}. ExternalReference: {ExternalReference}",
    tenantId,
    request.ExternalReference);

Avoid logging the entire request object:

// Avoid this in production unless the object is explicitly safe and redacted.
_logger.LogInformation("Create customer request: {@Request}", request);

Sensitive Data and Redaction

Logs often live longer and are accessible to more people than application databases. Therefore, logs must be treated as a data security boundary.

Do not log:

• Passwords.
• Access tokens or refresh tokens.
• API keys.
• Full credit card numbers.
• Sensitive personal data.
• Full request bodies unless explicitly approved and redacted.
• Large domain objects containing unknown fields.

Prefer:

_logger.LogInformation(
    "Password reset requested for user {UserId}",
    userId);

Avoid:

_logger.LogInformation(
    "Password reset requested for email {Email} with token {Token}",
    email,
    resetToken);

Even if the email is acceptable in some systems, the token should not be logged.

Logging Exceptions Correctly

When logging exceptions, pass the exception object to the logging method:

try
{
    await paymentGateway.AuthorizeAsync(command, cancellationToken);
}
catch (PaymentGatewayException ex)
{
    _logger.LogError(
        ex,
        "Payment authorization failed for order {OrderId} using provider {PaymentProvider}",
        command.OrderId,
        command.Provider);

    throw;
}

Avoid logging only the exception message:

_logger.LogError("Payment failed: {Message}", ex.Message);

Passing the exception object preserves stack trace and exception details for the logging provider.

Avoid double logging the same exception at every layer. Usually, log at the boundary where the exception is handled or where important context exists. If a lower layer logs and rethrows, and an upper layer also logs the same exception, logs can become noisy and misleading.

Correlation Across HTTP Calls

For service-to-service calls, correlation must leave the current process.

With distributed tracing, modern HTTP clients and ASP.NET Core can propagate trace context. For a custom correlation header, you may still need a delegating handler.

Example using a custom correlation context:

public interface ICorrelationContext
{
    string? CorrelationId { get; }
}

public sealed class CorrelationHeaderHandler : DelegatingHandler
{
    private readonly ICorrelationContext _correlationContext;

    public CorrelationHeaderHandler(ICorrelationContext correlationContext)
    {
        _correlationContext = correlationContext;
    }

    protected override Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request,
        CancellationToken cancellationToken)
    {
        if (!string.IsNullOrWhiteSpace(_correlationContext.CorrelationId))
        {
            request.Headers.TryAddWithoutValidation(
                "X-Correlation-ID",
                _correlationContext.CorrelationId);
        }

        return base.SendAsync(request, cancellationToken);
    }
}

Registration:

builder.Services.AddTransient<CorrelationHeaderHandler>();

builder.Services.AddHttpClient<PaymentClient>()
    .AddHttpMessageHandler<CorrelationHeaderHandler>();

In real systems, prefer standard trace context propagation where possible, and use custom correlation headers for business or legacy needs.

Correlation Across Queues and Background Jobs

Correlation can break when work moves to a queue, background service, or scheduled job. HTTP trace context flows automatically in many cases, but queue messages require explicit propagation.

When publishing a message, include correlation metadata:

public sealed record OrderSubmittedMessage(
    Guid OrderId,
    string CorrelationId,
    string? TraceParent);

Publishing example:

var message = new OrderSubmittedMessage(
    OrderId: orderId,
    CorrelationId: correlationId,
    TraceParent: Activity.Current?.Id);

await messageBus.PublishAsync(message, cancellationToken);

Consuming example:

using (_logger.BeginScope(new Dictionary<string, object>
{
    ["CorrelationId"] = message.CorrelationId,
    ["OrderId"] = message.OrderId
}))
{
    _logger.LogInformation("Processing submitted order message");
    await orderProcessor.ProcessAsync(message.OrderId, cancellationToken);
}

For advanced tracing, use a propagator or message header approach compatible with your messaging library and observability tooling.

Business Correlation vs Technical Correlation

Technical correlation connects logs by infrastructure context, such as trace ID or request ID. Business correlation connects logs by domain identifiers, such as:

• OrderId
• CustomerId
• TenantId
• PaymentId
• FileTransferId
• BatchId
• InvoiceId

For production support, business identifiers are often more useful than raw trace IDs because users and support teams can understand them.

Example:

_logger.LogInformation(
    "Generated invoice {InvoiceId} for tenant {TenantId} and customer {CustomerId}",
    invoiceId,
    tenantId,
    customerId);

A good logging strategy uses both:

• Trace/correlation IDs for technical request flow.
• Business IDs for domain-level troubleshooting.

Event IDs

EventId can give important log messages a stable identifier independent of the text message.

Example:

public static class LogEvents
{
    public static readonly EventId PaymentAuthorized = new(1001, nameof(PaymentAuthorized));
    public static readonly EventId PaymentFailed = new(1002, nameof(PaymentFailed));
}

_logger.LogInformation(
    LogEvents.PaymentAuthorized,
    "Payment authorized for order {OrderId}",
    orderId);

Event IDs are useful when:

• Log message text changes but dashboards should remain stable.
• Alerts are based on specific application events.
• Large systems need consistent event classification.

Not every log needs a custom event ID, but important business or operational events often benefit from one.

High-Performance Logging

For normal application logs, ILogger extension methods are usually sufficient. For hot paths or high-volume logs, consider source-generated logging with LoggerMessageAttribute.

Example:

public static partial class OrderLog
{
    [LoggerMessage(
        EventId = 1001,
        Level = LogLevel.Information,
        Message = "Order {OrderId} processed in {ElapsedMilliseconds} ms")]
    public static partial void OrderProcessed(
        ILogger logger,
        Guid orderId,
        double elapsedMilliseconds);
}

Usage:

OrderLog.OrderProcessed(_logger, orderId, elapsedMilliseconds);

Source-generated logging can reduce allocations and avoid repeatedly parsing message templates at runtime. This matters most for performance-sensitive code paths.

For expensive log arguments, check whether the level is enabled:

if (_logger.IsEnabled(LogLevel.Debug))
{
    var diagnosticDetails = BuildExpensiveDiagnosticDetails(order);
    _logger.LogDebug("Order diagnostic details: {Details}", diagnosticDetails);
}

Logging Configuration

Logging should be configurable by environment. A common setup is:

• Development: more verbose logs.
• Production: Information or Warning by default, with selected categories enabled when needed.
• Incident response: temporarily increase logging for specific categories.

Example:

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning",
      "Microsoft.EntityFrameworkCore.Database.Command": "Warning",
      "MyCompany.Payments": "Debug"
    }
  }
}

Avoid enabling very verbose logs globally in production for long periods. It can increase cost, reduce performance, and make important events harder to find.

Logging Providers and Sinks

A logging provider controls where logs go and how they are formatted. Common destinations include:

• Console output.
• Files.
• Azure Application Insights or Azure Monitor.
• OpenTelemetry exporters.
• Elasticsearch or OpenSearch.
• Seq.
• Datadog.
• Splunk.
• SQL-based sinks.

Application code should usually depend on ILogger<T>, not on a specific provider API. Provider-specific features can be useful, but coupling all application code to one logging framework makes future changes harder.

Good Log Message Design

A useful log message should answer:

• What happened?
• Where did it happen?
• Which operation was involved?
• Which identifiers allow us to search related events?
• Was it normal, unexpected, failed, or critical?

Good example:

_logger.LogWarning(
    "Failed to send invoice email {InvoiceId} to customer {CustomerId}. RetryAttempt: {RetryAttempt}",
    invoiceId,
    customerId,
    retryAttempt);

Poor example:

_logger.LogWarning("Email failed");

The poor example may be readable, but it is not actionable in production.

Common Mistakes

Common mistakes include:

• Using string interpolation instead of message templates.
• Logging sensitive data.
• Logging entire request or domain objects without redaction.
• Missing correlation IDs in background jobs or queued messages.
• Logging exceptions without passing the exception object.
• Logging and rethrowing the same exception in every layer.
• Using inconsistent property names such as {User}, {UserId}, {Customer}, and {CustomerId} for the same concept.
• Logging too much at Information and creating noise.
• Not configuring scopes or trace IDs in the production logging provider.
• Depending only on logs and ignoring metrics/traces.

Best Practices

Practical best practices:

• Use ILogger<T> through dependency injection.
• Use structured message templates with stable property names.
• Include business identifiers that help production support.
• Use scopes for request-wide or operation-wide context.
• Use distributed tracing and trace context for service-to-service correlation.
• Propagate correlation across HTTP, queues, and background jobs.
• Keep log levels meaningful and consistent.
• Avoid sensitive data and use redaction for approved sensitive fields.
• Pass exception objects to logging methods.
• Avoid logging huge objects or payloads by default.
• Use source-generated logging for hot paths.
• Configure log levels by environment and category.
• Test that logs contain the fields needed during real incident scenarios.