Skip to main content

Workflow: Service Auto-Instrumentation

This guided workflow walks platform engineers and AI assistants through zero-touch OpenTelemetry auto-instrumentation on Kubernetes using the TalkOps OpenTelemetry MCP Server.

Step 1: Detect Workload Runtime & Language​

Before injecting instrumentation, inspect the target namespace to determine the runtime language and current telemetry status:

# MCP Tool Invocation
otel_list_instrumented_services(namespace="production")

The tool executes a 4-tier inspection strategy:

  1. Pod Annotations: Checks for existing instrumentation.opentelemetry.io/inject-* tags.
  2. Container Image Inspection: Evaluates image names and registries (e.g. python:3.12-slim, eclipse-temurin:21-jre, node:20-alpine).
  3. Naming Conventions: Matches deployment/container names against known service patterns.
  4. Environment Variables: Inspects runtime variables (JAVA_HOME, PYTHONPATH, NODE_VERSION).

Step 2: Validate Framework & SDK Compatibility​

Confirm that the application's framework is supported and retrieve the recommended propagator configuration:

otel_lookup_instrumentation(language="python", framework="fastapi")

Output Returned:

  • Injection Annotation: instrumentation.opentelemetry.io/inject-python: "true"
  • Supported Exporters: OTLP/gRPC (:4317) and OTLP/HTTP (:4318)
  • Default Propagators: tracecontext, baggage, b3
  • Supported Auto-Instrumented Libraries: FastAPI, Starlette, HTTPX, Requests, SQLAlchemy, Redis, Celery

Step 3: Ensure Instrumentation CRD Exists​

Ensure an Instrumentation custom resource is present in the target namespace, configured to send telemetry to your collector:

otel_patch_instrumentation(
  name="production-instrumentation",
  namespace="production",
  exporter_endpoint="http://otel-collector.monitoring.svc:4317",
  sampler_type="parentbased_traceidratio",
  sampler_argument="0.20",
  dry_run=false
)

[!TIP] Always use parentbased_traceidratio rather than raw traceidratio. Parent-based sampling respects the sampling decision made by upstream services, preventing broken or orphaned distributed trace fragments.

Step 4: Annotate the Deployment​

Apply the injection annotation to the Deployment pod template with automatic environment variable conflict detection:

otel_annotate_deployment(
  deployment_name="order-service",
  namespace="production",
  language="python",
  instrumentation_name="production-instrumentation",
  dry_run=false
)

[!IMPORTANT] The tool automatically checks for hardcoded OTEL_EXPORTER_OTLP_ENDPOINT or OTEL_TRACES_EXPORTER environment variables in container specifications. If hardcoded endpoints exist, they silently override the Operator's injected settings; the tool warns you and provides remediation steps before applying changes.

Step 5: Verify Pod Rollout & Telemetry Flow​

Verify that the rolling restart succeeded and the init container injected the OpenTelemetry agent:

  1. The OpenTelemetry Operator injects an init container (opentelemetry-auto-instrumentation-python) that copies the agent libraries into a shared volume.
  2. The application container starts with modified PYTHONPATH or JAVA_TOOL_OPTIONS.
  3. Outbound spans and metrics begin streaming to http://otel-collector.monitoring.svc:4317.