How can I initialize `duckdb-wasm` within NextJS?

Question

I'm working on a NextJS project that leverages a wasm package via npm; specifically this is duckdb-wasm.

duckdb-wasm needs to initialize from a set of bundles (e.g. based on browser capability). this can be done with JSDelivr or by specifying the bundles in your code (see here: https://www.npmjs.com/package/@duckdb/duckdb-wasm#Instantiation).

JSDelivr runs into a CORS issue when deployed and so I'm trying to get the second option (i.e. webpack via manual specification) working with NextJS. Unfortunately, this seems to run into problems: TypeError: url.replace is not a function.

Here's the _app.tsx for reference:

import "../styles/globals.css";

import * as duckdb from "@duckdb/duckdb-wasm";
import duckdb_wasm_coi from "@duckdb/duckdb-wasm/dist/duckdb-coi.wasm";
import duckdb_wasm_eh from "@duckdb/duckdb-wasm/dist/duckdb-eh.wasm";
import duckdb_wasm from "@duckdb/duckdb-wasm/dist/duckdb-mvp.wasm";
import {
  DuckDBConnectionProvider,
  DuckDBPlatform,
  DuckDBProvider,
} from "@duckdb/react-duckdb";
import type { AppProps } from "next/app";

let manualBundles: duckdb.DuckDBBundles;
const logger = new duckdb.ConsoleLogger(duckdb.LogLevel.WARNING);

manualBundles = {
  mvp: {
    mainModule: duckdb_wasm,
    mainWorker: new URL(
      "@duckdb/duckdb-wasm/dist/duckdb-browser-mvp.worker.js",
      import.meta.url
    ).toString(),
  },

  eh: {
    mainModule: duckdb_wasm_eh,
    mainWorker: new URL(
      "@duckdb/duckdb-wasm/dist/duckdb-browser-eh.worker.js",
      import.meta.url
    ).toString(),
  },

  coi: {
    mainModule: duckdb_wasm_coi,
    mainWorker: new URL(
      "@duckdb/duckdb-wasm/dist/duckdb-browser-coi.worker.js",
      import.meta.url
    ).toString(),
    pthreadWorker: new URL(
      "@duckdb/duckdb-wasm/dist/duckdb-browser-coi.pthread.worker.js",
      import.meta.url
    ).toString(),
  },
};

function App({ Component, pageProps }: AppProps) {
  return (
    <DuckDBPlatform logger={logger} bundles={manualBundles}>
      <DuckDBProvider>
        <DuckDBConnectionProvider>
          <Component {...pageProps} />
        </DuckDBConnectionProvider>
      </DuckDBProvider>
    </DuckDBPlatform>
  );
}

export default App;

And the nextjs.config.js:

const withTM = require("next-transpile-modules")(["@duckdb/react-duckdb"]);

/** @type {import('next').NextConfig} */
const nextConfig = {
  reactStrictMode: true,
  //swcMinify: true,
  webpack: (config, options) => {
    config.output.webassemblyModuleFilename = "static/wasm/[modulehash].wasm";

    config.module.rules.push({
      test: /.*\.wasm$/,
      type: "asset/resource",
      generator: {
        filename: "static/wasm/[name].[contenthash][ext]",
      },
    });

    config.experiments = {
      asyncWebAssembly: true,
      ...config.experiments,
    };

    return config;
  },
};

module.exports = withTM(nextConfig);

One way to work around this is to bring the definition of the bundles into a useEffect but this isn't desirable for a number of reasons, for example the rendering of the child components needs to block on the value of bundles.

I am experiencing the same problem. Have you been able to find a solution? — Ilya Boyandin, Nov 04 '22 at 13:47
@IlyaBoyandin unfortunately, no. I'm not actively working on this at the moment, but it would be nice nonetheless. — maxcountryman, Nov 05 '22 at 14:29

rvanlaarhoven · Answer 1 · 2022-07-18T18:47:13.300

1

I haven't worked with duckdb, but in general you should be able to alter the webpack config to include wasm files.

Have a look at the asyncWebAssembly experiments property in Webpack 5 as documented here.

An example of how you could use it in next.config.js:

module.exports = {
  webpack: (config) => {
    const experiments = config.experiments || {}
    config.experiments = { ...experiments, asyncWebAssembly: true }
    config.output.assetModuleFilename = 'static/[hash][ext]'
    config.output.publicPath = '/_next/'
    config.module.rules.push({
      test: /\.wasm/,
      type: 'asset/resource'
    })
    return config
  }
}

In addition to this, you might need an async import to import the modules:

async function loadDependencies() {
  const duckdb_wasm = await import('@duckdb/duckdb-wasm/dist/duckdb-mvp.wasm')
}

edited Jul 18 '22 at 18:47

answered Jul 18 '22 at 18:34

rvanlaarhoven

591
6
16

The specific problem seems to be related to the `new URL(...)` invocation; Webpack is already configured to with the experiment flag and like I mentioned this does work if moved into a `useEffect` hook, however this isn't a good solution. – maxcountryman Jul 18 '22 at 19:03
Right, I see you updated the question. Since you mention "duckdb-wasm needs to initialize from a set of bundles (e.g. based on browser capability)", the problem may be caused by instantiating the `manualBundles` outside the component, which is happening during build time where it is not aware of any browser? That probably explains that a `useEffect` does work correctly. – rvanlaarhoven Jul 18 '22 at 20:10
I believe it should be possible. For example, take a look at this React app: https://github.com/duckdb/duckdb-wasm/blob/f8ef472d603c3868b078b2bbfe4dfd43fa1781b9/packages/duckdb-wasm-app/src/app.tsx#L20-L37 – maxcountryman Jul 18 '22 at 20:59
That looks like a different situation though. It looks like it's client-side rendering the app, while Next.js renders the app on the server or statically during build time. – rvanlaarhoven Jul 18 '22 at 21:47
True but JSDelivr works here (assuming the URL import experimental flag is enabled). Is there no way to configure webpack to do the rewrite of the @duckdb/duckdb-wasm paths? – maxcountryman Jul 18 '22 at 22:25

How can I initialize `duckdb-wasm` within NextJS?

1 Answers1