Getting consistent data from API in my node server without breaking down

Question

I am using a node server to get trades data from binance. There are over a thousand pairs against which trades need to be fetched. The function takes time to run completely. I need the function to restart whenever it is finished running so I keep getting new data while my server is live and running. However, after my server has been running for 10-15 minutes, the following error occurs:

I want the server to run permanently in the background and for this function to keep fetching trades from API and storing those trades in my DB. I have another GET method defined that then fetches the trades from my DB. The function that I am trying to run permanently lies in my main server.js file:

const getTrades = async () => {
  let page = 1;
  let coins = [];
  const results = await db.query("SELECT * FROM pairs;");
  const pairs = results.rows;
  const latest = await db.query("SELECT MAX(trade_time) FROM trades");
  const latestTrade = latest.rows[0].max;

  while (page < 55) {
    gecko = await axios(
      `https://api.coingecko.com/api/v3/coins/markets?vs_currency=USD&order=market_cap_desc&per_page=250&page=${page}`
    );
    coins.push(gecko.data);
    page++;
  }
  console.log("Loop over");
  coins = coins.flat();
  for (const pair of pairs) {
    let biggestTrade = [];
    response = await axios.get(
      `https://api.binance.com/api/v3/trades?symbol=${pair.pair}`
    );
    let filtered = response.data;
    filtered = filtered.filter((trade) => trade.time > latestTrade);

    let sells = filtered.filter((trade) => trade.isBuyerMaker === true);
    let buys = filtered.filter((trade) => trade.isBuyerMaker === false);
    if (sells.length > 0) {
      biggestTrade.push(
        sells.reduce(function (prev, current) {
          return prev.quoteQty > current.quoteQty ? prev : current;
        })
      );
    }
    if (buys.length > 0) {
      biggestTrade.push(
        buys.reduce(function (prev, current) {
          return prev.quoteQty > current.quoteQty ? prev : current;
        })
      );
    }

    biggestTrade = biggestTrade.flat();
    for (const trade of biggestTrade) {
      let priceUSD = 0;
      let baseAssetIcon = "";
      for (const coin of coins) {
        if (coin.symbol.toUpperCase() === pair.quote_asset) {
          priceUSD = coin.current_price;
        }
        if (coin.symbol.toUpperCase() === pair.base_asset) {
          baseAssetIcon = coin.image;
        }
        if (priceUSD > 0 && baseAssetIcon.length > 0) {
          break;
        }
      }
      if (trade.quoteQty * priceUSD > 50000) {
        const results = db.query(
          "INSERT INTO trades (exchange_name, exchange_icon_url, trade_time, price_in_quote_asset,price_in_usd, trade_value, base_asset_icon,  qty, quoteQty, is_buyer_maker, pair, base_asset_trade, quote_asset_trade) VALUES($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12, $13)",
          [
            "Binance",
            "https://assets.coingecko.com/markets/images/52/small/binance.jpg?1519353250",
            trade.time,
            trade.price,
            priceUSD,
            trade.quoteQty * priceUSD,
            baseAssetIcon,
            trade.qty,
            trade.quoteQty,
            trade.isBuyerMaker,
            pair.pair,
            pair.base_asset,
            pair.quote_asset,
          ]
        );
        console.log("Trade Added");
      }
    }
  }
  console.log("PAIRS ARE OVER");
};

How can I make it so that the function runs repeatedly after a specified time period and the server does not break.

score 0 · Answer 1 · answered Mar 31 '22 at 00:25

If you make continuous multiple calls to external third party API's without interval between calls, you are asking for being disconnected because API's have security policies that will prevents this kind of clients. Imagine if the entire world making 55 connections at once in a server. The server certainly will crash.

I see in your code you are making 55 calls at once. I recommend you put a delay between each call.

const delay = time => new Promise(res => setTimeout(res,time));

await delay(1000); // 1 second delay

There are other optimization that can prevent issues with connections in axios, like sharing httpAgent:

import http from "http"
import https from "https"
import axios from "axios"

const httpAgent = new http.Agent({ keepAlive: true })
const httpsAgent = new https.Agent({ keepAlive: true })

const api = axios.create({
  baseURL: "http://google.com",
  httpAgent,
  httpsAgent,
})


//now you will reuse the axios instance:
while(page < 55) {
 await delay(500);

 gecko = await api(
      `https://api.coingecko.com/api/v3/coins/markets?vs_currency=USD&order=market_cap_desc&per_page=250&page=${page}`
    );
}

Getting consistent data from API in my node server without breaking down

1 Answers1