How to unlock OpenTelemetry observability when using Node.js code bundled with webpack – without giving up on webpack optimizations to your Lambda functions.
OpenTelemetry (OTel) is an emerging industry standard that dev teams use to instrument, generate, collect, and export telemetry to better understand software performance and behavior. At Helios, we promote OTel observability – How? We leverage OTel to provide developers with actionable insights into their code within distributed systems. We give them visibility into how data flows through their applications, enabling them to quickly identify, reproduce and debug issues in their flows.
While OTel is a useful observability framework, there are cases where it doesn’t mesh well with certain tools. In our day-to-day we work with customers who use webpack to bundle their Node.js Lambda functions. This is a common method to reduce Lambda artifact size and optimize cold start times. However, we’ve seen that after OTel is installed, distributed tracing data is partial – instrumentation doesn’t work on the bundled code as expected.
In this blog, I’m going to take you deeper into where the collision between OTel and webpack is. Then I’ll share how we solved the problem at Helios, by auto-starting instrumentations and patching bundled modules explicitly.
Why does OTel clash with webpack?
When OTel is used to instrument dynamic languages like Javascript there’s a collision with static code bundling tools such as webpack, because of the dynamic patching nature of OTel.
The gist of the problem is that OTel uses the require-in-the-middle module, but webpack switches the require
to a __webpack_require
. Because OTel cannot find the require it’s looking for, instrumentation is not executed. This issue can escalate even further with tree-shaking of OTel code.
webpack is a static code bundler which transpiles your source code on build time by using static code analysis. When webpack processes your application, it combines every required module that your project uses into one (or more) bundles. Since modules are put inline in the bundle, and required using webpack’s internal require
function called __webpack_require
, hooking the require
on runtime doesn’t work. Also, when it transpiles dynamic imports like import(foo)
webpack does not resolve the actual module as foo
because foo
can be any path to any file in your system or project.
This combination causes OTel instrumentation patching not to work on runtime, and it can potentially be tree-shaken after webpack is unable to detect its dynamic imports.
How to solve the root cause of OTel clashing with serverless webpack
TL:DR: A summary of what I did to solve the problem
- Instrument Lambda internal dependencies before the Lambda runtime loads to make sure OTel is properly applied on your native modules that run outside of the bundle.
- Instrument bundled dependencies properly by either:
- Excluding those dependencies from the webpack bundle
- 👍 Pros
Simple: The webpack bundling won’t affect OTel as your instrumentednode_modules
are not bundled at all.
Standard: Most OTel SDK vendors suggest using this approach. - 🙅🏻 Cons
Limited performance optimization: webpack tree-shaking won’t affect your instrumented dependencies code, the bundle may be bigger, and Lambda cold start may be slower.
- 👍 Pros
- Including those dependencies in the bundle but explicitly requiring and patching them
- 👍 Pros
Optimized performance: webpack will be able to tree-shake the instrumented library. - 🙅🏻 Cons
Less stable: Using OTel internalpatch
method is not an external API so this solution is not future-proof and needs to be tested extensively.
Maintenance: Having an explicit import – need to make sure you remove it if the module is not in use in your runtime code.
- 👍 Pros
- Excluding those dependencies from the webpack bundle
For the purpose of this solution, I’ve used serverless-webpack, but these options can be adapted with minor tweaks to any code implementation. Also, this example specifically refers to AWS, but applies to other cloud providers as well. Below, I go into the solution step-by-step.
Pre-reqs for implementing the solution
- Install serverless-webpack.
If you’re starting from scratch, you can find detailed info on starting a new serverless-webpack project on the README file.
npm install serverless-webpack --save-dev
- Create a custom webpack configuration.
Make sure that modules that are loaded in the Lambda runtime are excluded from your bundle.
The AWS Lambda execution environment contains a number of libraries such as the AWS SDK for the Node.js runtime (the full list can be found here).# serverless.yml custom: webpack: webpackConfig: ./webpack.config.js
// webpack.config.js module.exports = { .. externals: [/^aws-sdk$/] .. }
1. Initializing OTel instrumentations on modules that are loaded in the Lambda runtime
Below you can find a basic example for instrumenting Node.js and aws-sdk:
// lambdaOtelInit.js const { NodeTracerProvider } = require("@opentelemetry/sdk-trace-node"); const { registerInstrumentations } = require("@opentelemetry/instrumentation"); const { AwsInstrumentation } = require("@opentelemetry/instrumentation-aws-sdk"); const provider = new NodeTracerProvider(); provider.register() registerInstrumentations({ instrumentations: [ new AwsInstrumentation(), ], });
In order to work properly, OTel instrumentation must be initialized before other libraries are imported into the runtime. You can do this by providing the NODE_OPTIONS
configuration to your Lambda:
NODE_OPTIONS="--require lambdaOtelInit.js"
At this point, OTel should be applied properly on the Lambda internal dependencies.
2. Taking care of instrumenting the bundled dependencies properly
There are a couple of different options to accomplish this:
- Exclude all other instrumented
node_modules
from the webpack build
By excluding your instrumented modules from webpack, they will be required regularly at runtime, so instrumentation should work properly.// webpack.config.js module.exports = { .. externals: [/^aws-sdk$/, /^pg$/] // or you can exclude all modules with nodeExternals() .. }
This is the commonly used approach to solve the OTel/webpack clash.
- Auto start instrumentations by patching them explicitly in the bundle
In the code below, we explicitly require the Node.js Postgres module and patch it, allowing webpack to recognize that the instrumentation patch is being called.
// OTelAutoInit.js const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node'); const { registerInstrumentations } = require('@opentelemetry/instrumentation'); const { PgInstrumentation } = require('@opentelemetry/instrumentation-pg'); const tracerProvider = new NodeTracerProvider({ plugins: { pg: { path: '@opentelemetry/instrumentation-pg' } } }); const pgInstrumentation = new PgInstrumentation(); registerInstrumentations({ tracerProvider, instrumentations: [pgInstrumentation] }); // this is where the magic happens function autoStartPgInstrumentation() { try { const pg = require('pg') pgInstrumentation.patch(pg); } catch (error) { // error initializing aws-sdk instrumentation; } } autoStartPgInstrumentation();
Make sure the init code runs before the handler code. This can be done by adding another entry point prior to the handler entry point.
// webpack.config.js module.exports = { .. entry: ['./OTelAutoInit.js', './handler.js'], .. };
This approach allows you to benefit from webpack optimizations (including tree-shaking) – while having OTel instrumentation work.
Conclusion
Although it appears that webpack & OTel do not play nice, there are ways to get them to work together and achieve OTel observability. Above I described how developers can ensure their bundled dependencies are instrumented properly by 1/ Initializing OTel instrumentations on modules that are loaded in the Lambda runtime; and 2/ Either (A) Excluding those dependencies from the webpack bundles; OR (B) Including those dependencies in the bundle but explicitly patching them. With this solution, Helios has helped our customers who use webpack benefit from OTel and observability across their application.