Parsing Large Files in Node.js

by | 19 Dec 2017 | Node.js, Programming | 0 comments

JavaScript heap out of memory.

This is the kind of error you will be getting when you parsed a large file (sized around 80mb and above) in Node.js, and this happens only when you are trying to load the entire file into memory at one go.

The solution to this problem is to parse the file part by part. In this article, we will be using a library called line-by-line to illustrate the concept. Processing the file line by line will not load the entire file into memory, hence you will not get any memory exception error.

With the line-by-line npm library, you can either process a file synchronously or asynchronously, depending on your requirements.

The next few code snippets will give you a better idea on how it works.

var LineByLineReader = require('line-by-line'),
    lr = new LineByLineReader('big_file.txt');

lr.on('error', function (err) {
	// 'err' contains error object

lr.on('line', function (line) {
	// pause emitting of lines...

	// your asynchronous line processing..
	setTimeout(function () {

		// ...and continue emitting lines.
	}, 100);

lr.on('end', function () {
	// All lines are read, file is closed now.



Submit a Comment

Your email address will not be published. Required fields are marked *

Related Posts

AppleiOSMobile PhonesObjective-CProgrammingSwiftTechnology
How to install Cocoapods for Objective-C/Swift development on Mac OS?

How to install Cocoapods for Objective-C/Swift development on Mac OS?

Steps to install Cocoapods Cocoapods is used in Xcode IDE for development from iOS apps using Objective-C/Swift. Cocoapods helps developers add external libraries to their iOS project easily. The following tutorial guides you on the steps to install Cocoapods on...

AndroidAndroidAppleiOSJavaMobile PhonesObjective-CProgramming
iPhone Camera
Complete overview of the mobile app development process (Infographic)

Complete overview of the mobile app development process (Infographic)

Clients always have the misperception that developing a mobile app is as easy as developing a mobile application would be a walk in a park without much effort involved. The following infographic pretty much sums up the total cost and effort for mobile app...