feat(otel): advanced observability phase 2 - data layer and privacy hardening#3797
Open
jaegeral wants to merge 16 commits intogoogle:masterfrom
Open
feat(otel): advanced observability phase 2 - data layer and privacy hardening#3797jaegeral wants to merge 16 commits intogoogle:masterfrom
jaegeral wants to merge 16 commits intogoogle:masterfrom
Conversation
Collaborator
Author
|
/gemini review |
jaegeral
commented
May 6, 2026
| - CHOKIDAR_USEPOLLING=true | ||
| - PROMETHEUS_MULTIPROC_DIR=/tmp/ | ||
| - ENABLE_STRUCTURED_LOGGING=true | ||
| # - ENABLE_STRUCTURED_LOGGING=true |
jaegeral
commented
May 6, 2026
jaegeral
commented
May 6, 2026
| from typing import Generator, List, Dict, Optional, Any, Union | ||
|
|
||
| from dateutil import parser, relativedelta | ||
| from flask import abort |
Collaborator
Author
There was a problem hiding this comment.
just changing the order of imports to make lint happy
jaegeral
changed the title
Collaborator
Author
|
Had |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This Pull Request implements Phase 2 of the OpenTelemetry instrumentation for Timesketch. The focus of this phase is extending distributed tracing into the data layer (OpenSearch and PostgreSQL)
Data Layer Visibility
OpenSearch: Implemented manual instrumentation for
search()andcount()methods. This captures targeted indices and internal processing latency (took_ms) without exposing the query content.SQLAlchemy (PostgreSQL): Integrated automatic tracing for all database operations, providing visibility into connection health and statement execution time.
ACL Layer: Wrapped
has_permission()checks in logical spans. This groups the many individual database metadata lookups (often 40+ per request) into a single, understandable context in the trace waterfall.Content Exclusion: Raw search query strings and DSL structures are now completely excluded from span attributes. This ensures that no sensitive case data (PII, hostnames, identifiers) ever leaves the application via telemetry.
Sensitive Data Scrubber: A global
SpanProcessorautomatically redacts credentials (passwords, tokens, sessions) from any span attribute across the entire system.Analyst Attribution: Authenticated
user.idanduser.nameare captured on all API spans. This allows correlating performance bottlenecks or errors back to a specific investigator without seeing their search terms.-Developer Experience & Resilience
@safe_telemetry_calldecorator that ensures telemetry logic (like JSON serialization) is "best-effort" and never crashes the primary business logic.Infrastructure & Documentation
otel-collector-config.yamlto the centraldata/directory for better deployment readiness.docs/OpenTelemetry.mdwith current architecture and configuration details.