Parsing Large Files in Node.js

by | 19 Dec 2017 | Node.js, Programming | 0 comments

JavaScript heap out of memory.

This is the kind of error you will be getting when you parsed a large file (sized around 80mb and above) in Node.js, and this happens only when you are trying to load the entire file into memory at one go.

The solution to this problem is to parse the file part by part. In this article, we will be using a library called line-by-line to illustrate the concept. Processing the file line by line will not load the entire file into memory, hence you will not get any memory exception error.

With the line-by-line npm library, you can either process a file synchronously or asynchronously, depending on your requirements.

The next few code snippets will give you a better idea on how it works.

var LineByLineReader = require('line-by-line'),
    lr = new LineByLineReader('big_file.txt');

lr.on('error', function (err) {
	// 'err' contains error object
});

lr.on('line', function (line) {
	// pause emitting of lines...
	lr.pause();

	// ...do your asynchronous line processing..
	setTimeout(function () {

		// ...and continue emitting lines.
		lr.resume();
	}, 100);
});

lr.on('end', function () {
	// All lines are read, file is closed now.
});

 

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *

Related Posts

BrowserGoogle ChromeProgrammingTechnology
[Google Chrome Developer Ti] Prevent Warning "Your connection is not private" from appearing 5
[Google Chrome Developer Tip] Prevent Warning “Your connection is not private” from appearing

[Google Chrome Developer Tip] Prevent Warning “Your connection is not private” from appearing

This is probably more applicable to web developers. Many times when we try to run a localhost site with https in our local environment to emulate the production environment as close as possible, we will always run into a Google Chrome warning message "Your...

AndroidAppleBrowserGoogle ChromeInternet ExplorerLinuxMicrosoft EdgeMobile PhonesMozilla FirefoxOSSafariTechnology
How to do a hard refresh for Chrome, Safari, Firefox, Internet Explorer or Edge in Windows & Mac?

How to do a hard refresh for Chrome, Safari, Firefox, Internet Explorer or Edge in Windows & Mac?

Browser Hard Refresh in Google Chrome, Safari, Mozilla Firefox, Internet Explorer and Microsoft Edge Sometimes just by refreshing your browser alone doesn't help in letting you see the latest changes of a website. Any web developers will know this best. So below...

Amazon Web Services (AWS)Elastic BeanstalkServer
Unable to delete AWS Elastic Beanstalk Environment "resource XX-XXXXXXX has a dependent object" - MervCodes
Unable to delete AWS Elastic Beanstalk Environment “resource XX-XXXXXXX has a dependent object”

Unable to delete AWS Elastic Beanstalk Environment “resource XX-XXXXXXX has a dependent object”

Unable to delete AWS Elastic Beanstalk Environment "resource XX-XXXXXXX has a dependent object" Problem You will usually get this error when you are trying to delete an Elastic Beanstalk environment. Solution As the message in the Events log states, the security...

DatabaseMagentoMySQLOpen SourcePHPPhpMyAdminTechnology
Magento 1.9 Fix for Sending Double Emails or Sending Email to Wrong Recipients

Magento 1.9 Fix for Sending Double Emails or Sending Email to Wrong Recipients

As we know, Magento uses a cron job system to send out email from core_email_queue table. There is an additional table called core_email_queue_recipients, which as the name states, will store all recipients for emails there. But these records do not get removed at...

ApacheCentOSLinuxOSServerTechnology
How to Setup Let's Encrypt FREE SSL Certificate on Centos 6 Apache using Certbot? | MervCodes
How to Setup Let’s Encrypt FREE SSL Certificate on Centos 6 Apache using Certbot?

How to Setup Let’s Encrypt FREE SSL Certificate on Centos 6 Apache using Certbot?

Having SSL on your website these days is important especially if you want your website to rank better on Google search, and that's when Let's Encrypt come into the picture. Let's Encrypt is a free, automated, and open certificate authority (CA), run for the...

Amazon Web Services (AWS)ApacheCentOSDatabaseLinuxMySQLOSProgrammingServerTechnology
8 Simple Steps to Setup a Web Server with Apache Virtual Hosts & MySQL in Centos

8 Simple Steps to Setup a Web Server with Apache Virtual Hosts & MySQL in Centos

This tutorial will be guiding you in setting up a web server with pre-installed Centos. In summary, you will be installing Apache and MySQL, and learning how to configure virtual hosts in Apache. Note: before proceeding with this tutorial. Make sure you figure out...