Bug: Synchronous File Reads Block the Event Loop During Leaderboard Sync

[TanCodeX]

Summary

The scripts/sync-leaderboard.js script uses synchronous file operations (fs.readFileSync) to load large JSON files (e.g., users.json and inactive-users.json). As the dataset grows, these operations block the Node.js event loop and consume significant memory, resulting in poor scalability and slower execution.

Affected File

• scripts/sync-leaderboard.js

Description

The script reads entire JSON files into memory using synchronous file operations:

const users = JSON.parse(fs.readFileSync('users.json', 'utf8'));
const inactiveUsers = JSON.parse(fs.readFileSync('inactive-users.json', 'utf8'));

fs.readFileSync() blocks the Node.js event loop until the file has been completely read. For large datasets, this can:

• Delay execution of other tasks.
• Increase memory usage by loading entire files into memory.
• Reduce overall application performance and scalability.

Although this is a standalone script, synchronous I/O becomes increasingly inefficient as file sizes grow.

Steps to Reproduce

1. Populate users.json and inactive-users.json with a large number of records.
2. Execute scripts/sync-leaderboard.js.
3. Observe increased execution time and memory consumption.
4. Monitor CPU and memory usage during file loading.

Expected Behavior

Large datasets should be processed using asynchronous or streaming file operations to minimize blocking and memory consumption.

Actual Behavior

The script synchronously reads entire JSON files into memory, blocking execution until the read operation completes.

Performance Impact

• Blocks the Node.js event loop during file reads.
• High memory usage for large JSON files.
• Slower execution as dataset size increases.
• Poor scalability with growing data volumes.

Recommendation

Replace synchronous file reads with asynchronous APIs such as fs.promises.readFile() where appropriate. For very large JSON files, consider using streaming parsers (e.g., stream-json) or storing the data in a database instead of loading entire files into memory.

Example using asynchronous file reads:

const fs = require('fs/promises');

const users = JSON.parse(
  await fs.readFile('users.json', 'utf8')
);

const inactiveUsers = JSON.parse(
  await fs.readFile('inactive-users.json', 'utf8')
);

For datasets that may become very large, use a streaming parser:

• Read files as streams.
• Process records incrementally instead of loading the entire file.
• Reduce peak memory usage and improve scalability.

[TanCodeX]

@jagdish-15 Please assign me this under gssoc.

[github-actions]

Thank you for opening an issue.

Please review our CONTRIBUTING.md guidelines before getting started.

A maintainer will review this issue shortly. If you would like to work on the issue yourself, please leave a comment requesting assignment.

Issue Assignment Policy

• The issue creator is given the first priority to work on their issue.
• If the creator declines or does not request assignment, the issue will be assigned to others in the order of requests received.
• Please do not submit PRs or begin work until you are officially assigned.

Note: This project is currently maintained by a solo maintainer, so reviews, assignments, and responses may sometimes take a little time. Thanks for your patience.

[jagdish-15]

I don't think this is necessary for our use case. This is a standalone sync script rather than a long-running HTTP server, so synchronous file reads aren't a concern here. The files involved are currently only a few KB in size, and even with significant growth, they're unlikely to become large enough for this to be a bottleneck. Switching to asynchronous or streaming I/O would add complexity without providing any meaningful benefit.