How to implement a stream.Transform for gulp

Gulp tasks make heavy use of node.js object Transform stream API (https://nodejs.org/api/stream.html#stream_class_stream_transform) to process files. More accurately, to process Vinyl (https://gulpjs.com/docs/en/api/vinyl) objects that represent files. There are gulp plugins that provide ready made node stream.Transform objects for performing many kinds of processing.

To write a gulp plugin is to implement a stream.Transform. Check “Implementing a Transform Stream” in the node.js stream API documentation (https://nodejs.org/api/stream.html#stream_implementing_a_transform_stream). The codes below is a gulp task that reads files under ./src/**.* and concatenates the contents to the corresponding files under ./dst, e.g. “./src/a.txt” concatenates to “./dst/a.txt”. Like most gulp tasks, it starts with gulp.src() to return an object stream (a stream of Vinyl objects), then pipes with a custom Transform.

The custom implemented stream.Transform uses the “new stream.Transform([options])” API. Interesting fields of the options object for implementing a Transform for gulp include:

  • readableObjectMode & writeableObjectMode: must both set to true in almost all cases
  • transform: function implementation for the stream._transform() method. In this case, the first parameter of the function “chunk” will be a Vinyl object, the second parameter “encoding” is not used, and “callback” is used to indicate finish of processing
  • flush: function implementation for the stream._flush() method. This is optional (not provided in code below). Only when the Transform want to do something before flushing the stream (for example, pushing some tail objects into the steam before closing), this function needs to be provided.

Another interesting thing of the above code is that the Transform actually uses another gulp plugin, i.e. Transform, the ‘gulp-append-prepend’ (https://www.npmjs.com/package/gulp-append-prepend) to do the actual work. Here, “gapStream” is an instance of the Transform object obtained by calling “gap.prependFile()” method (gap = require(‘gulp-append-prepend’)). In the transform implementation, the code first outputted the Vinyl object for debug, then obtained a Transform from gulp-append-prepend plugin that will prepend the corresponding file in the ‘./dst/’ directory (the corresponding file’s path is ‘./dst/’ concatenated with the ‘relative’ property of the Vinyl instance).

Then, the transform implementation uses the obtained Transform to do the work by using “write()” to write the accepted chunk into the obtained Transform, then using “read()” to obtain the product after processing, and finally using “this.push()” to push the product down the stream. Here “this.push()” is actually “readable.push()” since a Transform is a Duplex stream, i.e. both a readable and a writable.

The gulp documentation provides an example of using “through2” (https://github.com/rvagg/through2) to implement an inline plugin. “through2” is a tiny wrapper around the node stream.Transform to make the code above just slightly simpler.

In the above code, through2.obj is called with an implementation of transform function. Also, it can still use “this.push(productChunk)” to queue a product chunk, but can also just give the chunk to the callback function as shown above.

Developer, paddler

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store