Parsing Large Files in Node.js

by | 19 Dec 2017 | Node.js, Programming | 0 comments

JavaScript heap out of memory.

This is the kind of error you will be getting when you parsed a large file (sized around 80mb and above) in Node.js, and this happens only when you are trying to load the entire file into memory at one go.

The solution to this problem is to parse the file part by part. In this article, we will be using a library called line-by-line to illustrate the concept. Processing the file line by line will not load the entire file into memory, hence you will not get any memory exception error.

With the line-by-line npm library, you can either process a file synchronously or asynchronously, depending on your requirements.

The next few code snippets will give you a better idea on how it works.

var LineByLineReader = require('line-by-line'),
    lr = new LineByLineReader('big_file.txt');

lr.on('error', function (err) {
	// 'err' contains error object
});

lr.on('line', function (line) {
	// pause emitting of lines...
	lr.pause();

	// ...do your asynchronous line processing..
	setTimeout(function () {

		// ...and continue emitting lines.
		lr.resume();
	}, 100);
});

lr.on('end', function () {
	// All lines are read, file is closed now.
});

 

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *

Related Posts

APCCentOSFedoraLinuxOSPHPProgrammingServerTechnology
Install APC (Alternative PHP Cache) in CentOS 5/6/7 and Fedora 20/21

Install APC (Alternative PHP Cache) in CentOS 5/6/7 and Fedora 20/21

APC (Alternative PHP Cache) is a free and open source tool to cache PHP codes. 1. Install Dependency Packages for APC yum install php-pear php-devel httpd-devel pcre-devel gcc make -y 2. Install APC using PECL (PHP Extension Community Library) pecl install apc...