Extension randomly fails to deliver logs

[aaliakseyenka]

Hi,

Just 2 days ago we started getting random errors from lambda, which seem to be related to the extension. No changes or deployment were done, just errors appeared at some point and now randomly continue to popup. Our lambda uses the following configuration
Memory 256Mb
Runtime Amazon Linux 2023
Architecture arm64

Extension layer arn:aws:lambda:eu-central-1:464622532012:layer:Datadog-Extension-ARM:68

DD_ENV prod
DD_LOGS_CONFIG_PROCESSING_RULES [{"type": "exclude_at_match","name": "exclude_lambda_metadata","pattern": "^(START|END|INIT|REPORT).*"}]
DD_SERVERLESS_LOGS_ENABLED true
DD_SERVICE service
DD_SITE datadoghq.eu

The errors that appear:

DD_EXTENSION | ERROR | Failed to send logs to datadog: error sending request for url (https://http-intake.logs.datadoghq.eu/api/v2/logs)
Exiting: timeout, deadline: 1737666472995

The lambda has timeout of 5 seconds, but its runtime duration is only 2ms. Base only CloudWatch Logs we see that for ex. lambda finished at

2025-01-23T12:00:43.030Z

and then the error log from extension is

2025-01-23T12:00:48.058Z
DD_EXTENSION | ERROR | Failed to send logs to datadog: error sending request for url (https://http-intake.logs.datadoghq.eu/api/v2/logs)

This doesn't happen during cold starts, based on CloudWatch Logs it happens when lambda is already warm.

Does someone have any idea what could go wrong.

[astuyve]

Hi @aaliakseyenka - thanks for the report.
It's really hard to tell from incomplete logs, but it looks like the extension is repeatedly trying to send data to datadog and then failing for some reason, and then the last request is aborted because the function times out as there is 5s between your posted timestamp and the end.

Can you open a support ticket so that we can share the full debug logs in a secure place? You can email [email protected] to get started, then ask them to escalate it to AJ so that I can find it.

Thanks!
AJ

[aaliakseyenka]

thanks for the answer @astuyve. What is the best way to capture more logs from DD in order to provide them to you? because in cloudwatch we have only 2 entries right now:

• app's result log
• DD error I shared

[astuyve]

You'll want to set DD_LOG_LEVEL: debug, the support team will walk you through anything additional we may need.