spanmetrics cannot work with numerical default value in dimensions since version v0.109.0

[timbeemster]

Component(s)

No response

What happened?

Description

In our setup we use the VERSION environment variable to set a dimension on metrics, so we can query our metrics based on the version of the application.
The version is sourced in the environment variables and consists of a combination of numbers and letters.
It has come to our attention that in a newer version (v109) the spanmetrics connector does not allow a default of only digits to be set.
Similarly, if there are only digits and the letter e, it will try to decode the value as a float.

Perhaps something in the decoding of the yaml config changed?

Steps to Reproduce

I've created a minimal setup in a public repo: https://github.com/timbeemster/otelcol-custom/blob/main/README
Build the docker image and run the setup.

Expected Result

A running otel collector

Actual Result

Error message on startup (see log output section)

Let me know if I can assist in debugging the problem.

Collector version

v0.109.0

Environment information

Environment

OS: Golang base image for building + Debian12 (distroless variant for running)
Compiler(if manually compiled): golang:1.23.2

(see: https://github.com/timbeemster/otelcol-custom/blob/main/Dockerfile )

OpenTelemetry Collector configuration

builder config: https://github.com/timbeemster/otelcol-custom/blob/main/otel-builder-config.yml
collector config: https://github.com/timbeemster/otelcol-custom/blob/main/otel-collector-config.yml

Log output

Error: failed to get config: cannot unmarshal the configuration: decoding failed due to the following error(s):

error decoding 'connectors': error reading configuration for "spanmetrics": decoding failed due to the following error(s):

'dimensions[0].default' expected type 'string', got unconvertible type 'float64', value: '1.23e+06'
2024/12/11 15:45:40 collector server run finished with error: failed to get config: cannot unmarshal the configuration: decoding failed due to the following error(s):

error decoding 'connectors': error reading configuration for "spanmetrics": decoding failed due to the following error(s):

'dimensions[0].default' expected type 'string', got unconvertible type 'float64', value: '1.23e+06'

Additional context

No response

[bacherfl]

Hi @timbeemster! If you enclose the environment variable for the version in quotes, it should be parsed properly:

connectors:
  spanmetrics:
    dimensions:
      - name: gitHash
        default: "${VERSION}"

[timbeemster]

Hi @bacherfl ,
Thanks for reaching out. Unfortunately it does not fix the problem.
I already tested it earlier, because I thought it could solve the problem.
However, the same output is given.

Patch on the shared public repo to reproduce the problem:

diff --git a/otel-collector-config.yml b/otel-collector-config.yml
index df1789f..b29c00e 100644
--- a/otel-collector-config.yml
+++ b/otel-collector-config.yml
@@ -7,7 +7,7 @@ connectors:
   spanmetrics:
     dimensions:
       - name: gitHash
-        default: ${VERSION}
+        default: "${VERSION}"
 
 exporters:
   otlp:

rebuild docker image + run:
docker run -e VERSION=123e4 <hash>

output:

Error: failed to get config: cannot unmarshal the configuration: decoding failed due to the following error(s):

error decoding 'connectors': error reading configuration for "spanmetrics": decoding failed due to the following error(s):

'dimensions[0].default' expected type 'string', got unconvertible type 'float64', value: '1.23e+06'
2024/12/12 09:53:12 collector server run finished with error: failed to get config: cannot unmarshal the configuration: decoding failed due to the following error(s):

error decoding 'connectors': error reading configuration for "spanmetrics": decoding failed due to the following error(s):

'dimensions[0].default' expected type 'string', got unconvertible type 'float64', value: '1.23e+06'

[bacherfl]

ok now i can also reproduce this - I was trying to add the value directly into the config map earlier, but when using an env var this issue starts to appear. I will check if there have been some changes to that recently

[bacherfl]

It seems like in v0.109.0 a feature flag that controls whether the mapstructure library used for decoding should use the option WeaklyTypedInput has been removed, and therefore this option is now always set to false - some more background information including the rationaly behind this decision can be found in this issue: open-telemetry/opentelemetry-collector#10552

[timbeemster]

Hi @bacherfl, thanks for your investigation.
The related issue seems relevant indeed.
However, I can still not come to a point of a working config.
I tried to use:
default: !!str "${VERSION}" but it did not work.

I don't see other positives comments on the issue with a working solution/workaround

[bacherfl]

Would it be a possible solution to add a prefix to the commit hash? i.e. something like

connectors:
  spanmetrics:
    dimensions:
      - name: gitHash
        default: "v-${env:VERSION}"

This is the only workaround I can think of at the moment

[bacherfl]

also, can you open an issue for the confmap component in the core repository about this? This should be the place where this can be fixed, as the original string value gets converted into a float somewhere in this component (i'm currently trying to find out where exactly)

[timbeemster]

Hi @bacherfl,

I've opened up a bug report in the Core project.
Unfortunately prepending a v is not possible, as the collector is not the only component reasoning about that version number in our landscape.

Thanks for helping out thus far!

[bacherfl]

Thanks for creating the issue @timbeemster!

One more idea regarding prepending a prefix to the version env var until the issue is fixed in core: You could, as a workaround, also use the transform processor to remove the prefix again, so the format of the gitHash property stays compatible with other components in your environment - for example:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

connectors:
  spanmetrics:
    dimensions:
      - name: gitHash
        default: "sha256:${env:VERSION}"

exporters:
  debug:
    verbosity: detailed

processors:
  transform:
    metric_statements:
      - context: datapoint
        statements:
          - replace_pattern(attributes["gitHash"], "sha256:([1-9a-zA-Z]+)", "$1")

extensions:

service:
  pipelines:
    traces/spanmetrics:
      receivers: [ otlp ]
      exporters: [ spanmetrics ]
    metrics:
      receivers: [ otlp, spanmetrics ]
      processors: [ transform ]
      exporters: [ debug ]

[bacherfl]

Closing this one as the issue for the confmap component has been created in core: open-telemetry/opentelemetry-collector#11879