Contrib.social

Sometimes produces invalid profiles crashing pyroscope

Open on Drupal.org →
Created on 11 March 2025, about 2 months ago

Problem/Motivation

Pyroscope trace:

ts=2025-03-11T17:12:23.844305453Z caller=interceptor.go:66 level=info msg="request parameters" route=/querier.v1.QuerierService/SelectMergeSpanProfile tenant=anonymous traceID=7cc3141b96101391 duration=74.878425ms
panic: runtime error: index out of range [7] with length 7

goroutine 14449 [running]:
github.com/grafana/pyroscope/pkg/ingester/otlp.(*profileBuilder).convertSampleBack(0xc003dab380, 0xc002eedda0)
	github.com/grafana/pyroscope/pkg/ingester/otlp/convert.go:214 +0x28d
github.com/grafana/pyroscope/pkg/ingester/otlp.ConvertOtelToGoogle(0xc000d9eea0)
	github.com/grafana/pyroscope/pkg/ingester/otlp/convert.go:23 +0xa5
github.com/grafana/pyroscope/pkg/ingester/otlp.(*ingestHandler).Export(0xc00115b800, {0x3889378?, 0xc001213230?}, 0xc001062ed0)
	github.com/grafana/pyroscope/pkg/ingester/otlp/ingest_handler.go:100 +0x493
github.com/grafana/pyroscope/api/otlp/collector/profiles/v1development._ProfilesService_Export_Handler({0x23f32a0, 0xc00115b800}, {0x3889378, 0xc001213230}, 0xc000fea400, 0x0)
	github.com/grafana/pyroscope/api/otlp/collector/profiles/v1development/profiles_service.pb.go:303 +0x1a6
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000fff600, {0x3889378, 0xc001155320}, {0x38a3500, 0xc000199040}, 0xc0015d17a0, 0xc001155b60, 0x4e89980, 0x0)
	google.golang.org/grpc@v1.65.0/server.go:1379 +0xdf8
google.golang.org/grpc.(*Server).handleStream(0xc000fff600, {0x38a3500, 0xc000199040}, 0xc0015d17a0)
	google.golang.org/grpc@v1.65.0/server.go:1790 +0xe8b
google.golang.org/grpc.(*Server).serveStreams.func2.1()
	google.golang.org/grpc@v1.65.0/server.go:1029 +0x8b
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 14426
	google.golang.org/grpc@v1.65.0/server.go:1040 +0x125

Steps to reproduce

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes

🐛 Bug report
Status

Active

Version

1.0

Component

Code

Created by

🇨🇭Switzerland znerol

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @znerol
  • Comment about 2 months ago →
    🇨🇭Switzerland znerol

    My hunch is this is due to a mismatch between LocationTable and LocationIndices.

    https://github.com/grafana/pyroscope/blob/466aaa8194e06aab452b00f56efd5d...

    		gs.LocationId = append(gs.LocationId, p.convertLocationBack(p.src.LocationTable[p.src.LocationIndices[i]]))
  • Comment about 2 months ago →
    🇨🇭Switzerland znerol

    Possibly related warning:

    Warning: Undefined array key 33 in Drupal\Component\OTSpanProfile\ExcimerAdapter\LocationIndices->set() (line 49 of vendor/drupal/otspanprofile/ExcimerAdapter/LocationIndices.php)

    #0 web/core/includes/bootstrap.inc(108): _drupal_error_handler_real()
#1 vendor/drupal/otspanprofile/ExcimerAdapter/LocationIndices.php(49): _drupal_error_handler()
#2 vendor/drupal/otspanprofile/ExcimerAdapter/Samples.php(52): Drupal\Component\OTSpanProfile\ExcimerAdapter\LocationIndices->set()
#3 vendor/drupal/otspanprofile/ExcimerAdapter.php(105): Drupal\Component\OTSpanProfile\ExcimerAdapter\Samples->add()
#4 vendor/drupal/otspanprofile/SpanProfiler.php(104): Drupal\Component\OTSpanProfile\ExcimerAdapter->getProfile()
#5 vendor/drupal/otspanprofile/ProfileSpanProcessor.php(58): Drupal\Component\OTSpanProfile\SpanProfiler->send()
#6 vendor/drupal/otspanprofile/ProfileSpanProcessor.php(67): Drupal\Component\OTSpanProfile\ProfileSpanProcessor->forceFlush()
#7 vendor/open-telemetry/sdk/Trace/SpanProcessor/MultiSpanProcessor.php(62): Drupal\Component\OTSpanProfile\ProfileSpanProcessor->shutdown()
#8 vendor/open-telemetry/sdk/Trace/TracerSharedState.php(71): OpenTelemetry\SDK\Trace\SpanProcessor\MultiSpanProcessor->shutdown()
#9 vendor/open-telemetry/sdk/Trace/TracerProvider.php(98): OpenTelemetry\SDK\Trace\TracerSharedState->shutdown()
#10 vendor/open-telemetry/sdk/Common/Util/functions.php(36): OpenTelemetry\SDK\Trace\TracerProvider->shutdown()
#11 [internal function]: OpenTelemetry\SDK\Trace\TracerProvider->OpenTelemetry\SDK\Common\Util\{closure}()
#12 vendor/open-telemetry/sdk/Common/Util/functions.php(53): Closure->call()
#13 vendor/open-telemetry/sdk/Common/Util/ShutdownHandler.php(76): OpenTelemetry\SDK\Common\Util\{closure}()
#14 [internal function]: OpenTelemetry\SDK\Common\Util\ShutdownHandler::OpenTelemetry\SDK\Common\Util\{closure}()
#15 {main}

    This indicates that LocationIndices algorithm might fail in some scenarios.

  • Merge request !2Issue #3512288: Prevent out-of-bounds array access → (Merged) created by znerol
  • Pipeline finished with Skipped
    about 2 months ago
    #446879
  • Comment about 2 months ago →
    System Message
    • znerol → committed 49dd6dc6 on 1.x 
      Issue #3512288: Sometimes produces invalid profiles crashing pyroscope
  • Comment about 2 months ago →
    🇨🇭Switzerland znerol
  • Comment about 1 month ago →
    System Message

    Automatically closed - issue fixed for 2 weeks with no activity.

contrib.social Blog FAQ Discussions
Production build 0.71.5 2024