hello
I use grafana 9.4.9
and tempo 2.1.1
I use grafana agent as opentelmetry collector
and config automatic_logging
use Package xk6-client-tracing · GitHub as source
when I search in loki, log is comming very realtime
but in tempo search is not realtime
if I use log’s tid go to tempo use TraceQL to search
it’s is can find trace
is it’s bug here ?
or it’s normal ?
Hi. It can take a few seconds since a trace being ingested, until it shows up in search. Search is not supported in live traces (default 10s since last received batch), whereas ID lookups are.
but I was delay at least 5 minutes
here is my config
multitenancy_enabled: true
server:
http_listen_port: 8097
grpc_listen_port: 9097
http_server_read_timeout: 300s
http_server_write_timeout: 300s
grpc_server_max_recv_msg_size: 104857600
grpc_server_max_send_msg_size: 104857600
log_level: info
distributor:
receivers: # this configuration will listen on all ports and protocols that tempo is capable of.
jaeger: # the receives all come from the OpenTelemetry collector. more configuration information can
protocols: # be found there: https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver
thrift_http: #
grpc: # for a production deployment you should only enable the receivers you need!
thrift_binary:
thrift_compact:
zipkin:
otlp:
protocols:
http:
grpc:
max_recv_msg_size_mib: 104857600
opencensus:
ingester:
lifecycler:
ring:
# number of replicas of each span to make while pushing to the backend
replication_factor: 3
trace_idle_period: 10s # The length of time after a trace has not received spans to consider it complete and flush it.
max_block_duration: 30m
# duration to keep blocks in the ingester after they have been flushed
complete_block_timeout: 15m
compactor:
ring:
kvstore:
store: memberlist
compaction:
compacted_block_retention: 1h
block_retention: 4320h #180d
compaction_cycle: 10s
v2_in_buffer_bytes: 100
v2_out_buffer_bytes: 100
v2_prefetch_traces_count: 100000
storage:
trace:
backend: s3
wal:
path: /tmp/tempo/wal # where to store the the wal locally
s3:
bucket: tempo
endpoint: minio:9000
access_key: minioadmin
secret_key: minioadmin
insecure: true
forcepathstyle: true
cache: memcached
cache_max_block_age: 48h
memcached:
addresses: memcached:11211
timeout: 100ms
update_interval: 1m
# Pool used for finding trace IDs.
pool:
max_workers: 400 # Worker pool determines the number of parallel requests to the object store backend.
queue_depth: 20000 # Maximum depth for the querier queue jobs. A job is required for each block searched.
block:
bloom_filter_false_positive: 0.01 # Bloom filter false positive rate. lower values create larger filters but fewer false positives.
querier:
frontend_worker:
frontend_address: "tempo-query-frontend-1:9097"
max_concurrent_queries: 20
query_relevant_ingesters: false
trace_by_id:
query_timeout: 300s
search:
query_timeout: 300s
query_frontend:
# number of times to retry a request sent to a querier
max_retries: 2
# number of block queries that are tolerated to error before considering the entire query as failed
# numbers greater than 0 make possible for a read to return partial results
tolerate_failed_blocks: 0
# Maximum number of outstanding requests per tenant per frontend; requests beyond this error with HTTP 429.
max_outstanding_per_tenant: 10000
search:
# The number of concurrent jobs to execute when searching the backend.
concurrent_jobs: 1000
# The target number of bytes for each job to handle when performing a backend search.
target_bytes_per_job: 104857600
# Limit used for search requests if none is set by the caller
default_result_limit: 200
# The maximum allowed value of the limit parameter on search requests. If the search request limit parameter
# exceeds the value configured here it will be set to the value configured here.
# The default value of 0 disables this limit.
max_result_limit: 0
# The maximum allowed time range for a search.
# 0 disables this limit.
max_duration: 168h
# query_backend_after and query_ingesters_until together control where the query-frontend searches for traces.
# Time ranges before query_ingesters_until will be searched in the ingesters only.
# Time ranges after query_backend_after will be searched in the backend/object storage only.
# Time ranges from query_backend_after through query_ingesters_until will be queried from both locations.
# query_backend_after must be less than or equal to query_ingesters_until.
query_backend_after: 15m
query_ingesters_until: 30m
# If set to a non-zero value, it's value will be used to decide if query is within SLO or not.
# Query is within SLO if it returned 200 within duration_slo seconds OR processed throughput_slo bytes/s data.
# NOTE: `duration_slo` and `throughput_bytes_slo` both must be configured for it to work
# If set to a non-zero value, it's value will be used to decide if query is within SLO or not.
# Query is within SLO if it returned 200 within duration_slo seconds OR processed throughput_slo bytes/s data.
duration_slo: 10s
throughput_bytes_slo: 0
# Trace by ID lookup configuration
trace_by_id:
# The number of shards to split a trace by id query into.
# (default: 50)
query_shards: 50
# If set to a non-zero value, a second request will be issued at the provided duration.
# Recommended to be set to p99 of search requests to reduce long-tail latency.
hedge_requests_at: 2s
# The maximum number of requests to execute when hedging.
# Requires hedge_requests_at to be set. Must be greater than 0.
hedge_requests_up_to: 2
# If set to a non-zero value, it's value will be used to decide if query is within SLO or not.
# Query is within SLO if it returned 200 within duration_slo seconds.
duration_slo: 0s
memberlist:
join_members: ["tempo-distributor-1"]
overrides:
ingestion_rate_strategy: global
ingestion_burst_size_bytes: 104857600
ingestion_rate_limit_bytes: 104857600
max_bytes_per_trace: 5000000
max_traces_per_user: 0
max_bytes_per_tag_values_query: 5000000
metrics_generator_processors:
- service-graphs
- span-metrics
metrics_generator:
# Specifies which processors to use.
processor:
# Span metrics create metrics based on span type, duration, name and service.
span_metrics:
# Configure extra dimensions to add as metric labels.
dimensions:
- http.method
- http.target
- http.status_code
- service.version
# Service graph metrics create node and edge metrics for determinng service interactions.
service_graphs:
# Configure extra dimensions to add as metric labels.
dimensions:
- http.method
- http.target
- http.status_code
- service.version
# The registry configuration determines how to process metrics.
registry:
collection_interval: 5s # Create new metrics every 5s.
storage:
path: /tmp/tempo/generator/wal
remote_write:
- url: https://172.18.254.252:8095/api/v1/push
send_exemplars: true
tls_config:
insecure_skip_verify: true
basic_auth:
username: "mimirwrite"
password: "mimirwrite"
``Tempo does not necessarily return the most recent traces in the requested time range. It simply returns the first matching traces it finds. You’ve searched with no parameters so it will return 20 (or so) random traces in the time period.
okay…
does any config can change default 20 limit ??
There is a configuration setting for Tempo’s query_frontend:
query_frontend:
search:
default_result_limit: 20
However, this limit is only used for search requests if none is set by the caller. It does not influence the limit in the query editor UI. To my knowledge there it is not possible to change the default in the query editor.
Is there a way to return traces of a time range, like between 20 minutes to 15 minutes ago?
This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.