Automatic layout testing

in Front-end | February 13, 2019

Hi, developer!

A frequent problem of a layout designer is that he corrects styles in one place, but they change in several places. And, as a rule, the layout designer notices it too late. In this post, I will talk about regression testing. The main point of the method is that you take screenshots of the site (not manually of course), and then, after making any edits, you make new screenshots and compare them with the previous ones. If there are any differences, the tests say about it and show it. You can take screenshots both of sites and individual blocks.

For this method we need:

Gulp – build system
puppeteer – node.js a library that allows you to control the Chromium browser without a user interface
pixelmatch – a pixel-level image comparison library created for comparing screenshots in tests
pngjs – a simple PNG coder/decoder for node.js
fs – module provides API for interacting with the file system
path – provides utilities for working with file and directory paths

Also, for the convenience of displaying all the results on one page in a browser, we will need additional packages:

Installation

Install gulp:

npm install gulp -g

Initialize npm at the root of your project:

npm init

Install the remaining packages:

npm i chrome-launcher fs http node-static path pixelmatch pngjs puppeteer

Installation may take some time as the chromium browser installation is included in the puppeteer package

gulp.js configuration

Write interactions for the installed packages

const puppeteer = require('puppeteer');
const fs = require('fs');
const PNG = require('pngjs').PNG;
const pixelmatch = require('pixelmatch');
const chromeLauncher = require('chrome-launcher');
const http = require('http');
const staticN = require('node-static');
const path = require('path');

Configuration for a specific project:

Set the width of the page when testing

var initialPageWidth = 1920;

Also, add the list of tested pages to the pageList array :

var pageList = [
  'index',
  'gallery',
  'faq'
];

The future structure of our test result

└── test/                 
   ├── before/           
   │   ├── index.png
   │   ├── ...
   │   └── pageXX.png  
   ├── after/  
   │   ├── index.png
   │   ├── ...
   │   └── pageXX.png  
   ├── difference/
   │	├── index.png
   │	├── ...
   │    └── pageXX.png  
   └── index_test.html

All results will be stored in the test folder, which will be divided into another 3 subdirectories:

before – to store reference screenshots
after – to store new screenshots, which will later be compared
difference – to store the result of comparing new screenshots with reference ones

Create locals to save the paths:

var beforeDir = 'test/before/',
    afterDir = 'test/after/',
    diffDir = 'test/difference/';

Creating Tasks

1. The task of creating reference screenshots

Initially, using the fs module, we create directories where the test results are stored if they are missing. We also clear out old screenshots, if there are any.

if (!fs.existsSync('test')){
  fs.mkdirSync('test');
}

if (!fs.existsSync(beforeDir)){
  fs.mkdirSync(beforeDir);
}

if (fs.existsSync(beforeDir)){
  fs.readdir(beforeDir, (err, files) => {
    for (const file of files) {
      fs.unlink(path.join(beforeDir, file), err => {});
    }
  });
}

After, we go through the array of pageList pages, which we have filled earlier. In page.goto we declare the address of the pages, in this case, it ishttp://localhost:1337/ On page.screenshot – the directory to save screenshots and also declare fullPage: true to save the whole page, not a specific height

pageList.map(async function(element, index) {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  await page.setViewport({ width: initialPageWidth, height: 0 });
  
  await page.goto('http://localhost:1337/' + element + '.html');

  await page.screenshot({path: beforeDir + element + '.png', fullPage: true});
  console.log(element + ' page +');

  await browser.close();	
})

Complete task of creating reference screenshots looks like this:

gulp.task('test-init', function() {
  if (!fs.existsSync('test')){
    fs.mkdirSync('test');
  }

  if (!fs.existsSync(beforeDir)){
   fs.mkdirSync(beforeDir);
  }

  if (fs.existsSync(beforeDir)){
    fs.readdir(beforeDir, (err, files) => {
      for (const file of files) {
	fs.unlink(path.join(beforeDir, file), err => {});
      }
    });
}

  pageList.map(async function(element, index) {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();
	
    await page.setViewport({ width: initialPageWidth, height: 0 
  });
	
    await page.goto('http://localhost:1337/' + element + '.html');

    await page.screenshot({path: beforeDir + element + '.png', fullPage: true});
    console.log(element + ' page +');
	
    await browser.close();	
  })
})

The task is started with the command:

gulp test-init

2. The task of creating new screenshots and comparing them with the reference ones

As in the previous task, we create the necessary directories or clear them out from old files:

var clearDir = [diffDir, afterDir, 'test/']

if (!fs.existsSync(afterDir)){
  fs.mkdirSync(afterDir);
}

if (!fs.existsSync(diffDir)){
  fs.mkdirSync(diffDir);
}

clearDir.map(function(element, index) {
  if (fs.existsSync(element)){
    fs.readdir(element, (err, files) => {
      for (const file of files) {
        fs.unlink(path.join(element, file), err => {});
      }
    });
  }
});

Then we pass again an array of pages to create new pages, but now comparing them with the reference ones.

pageList.map(async function(element, index) {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  await page.setViewport({ width: initialPageWidth, height: 0 });

  await page.goto('http://localhost:1337/' + element + '.html');

  await page.screenshot({path: afterDir + element + '.png', fullPage: true});

  await browser.close();

  pageName = element;
  img1[index] = await fs.createReadStream(afterDir + element + '.png').pipe(new PNG()).on('parsed', function() { parse2(element, index)});
})

The code above is similar to creating reference screenshots except additional adding of each taken screenshot to the img1 array. Next, using parse2 function, we add reference screenshots to the img2 array. After that, using doneReading function, log the result to the required directory:

function parse2(element, index, pageName) {
  img2[index] = fs.createReadStream(beforeDir + element + '.png').pipe(new PNG()).on('parsed', function() { doneReading(img1[index], img2[index], element)});
}

function doneReading(img1, img2, pageName) {
  var diff = new PNG({width: img1.width, height: img1.height});

  pixelmatch(img1.data, img2.data, diff.data, img1.width, img1.height, {threshold: 0.5});


  diff.pack().pipe(fs.createWriteStream(diffDir + pageName + timeMod + '.png'));
  console.log(pageName + ' ---- page compared');
}

For additional convenience, we display all the images of the compared pages on one page and open it in the browser.

Let’s create a variable that will be used as a modifier in the names of images and pages to avoid browser caching:

var timeMod = new Date().getTime();

Next, we create a list of all pages with the name and image

var imgList = pageList.map(function(file, i) {
  return '<li style="width: 49%; display: inline-block; list-style: none; background-color: #888;"><h2 style="font: 3vw sans-serif; margin: 0; padding: 1em; text-align: center;">' + pageList[i] + '</h2><img style="width: 100%; display: block;" src="difference/' + file + timeMod + '.png"/></li>'
})

And add the list created above to the html file

fs.writeFile('test/index_test' + timeMod + '.html', imgList, function (err) {});

Finally, create a local server with this page and open it in Google Chrome browser:

var fileServer = new staticN.Server();

http.createServer(function (req, res) {
  req.addListener('end', function () {
    fileServer.serve(req, res);
  }).resume();
}).listen(8080);

chromeLauncher.launch({
  startingUrl: 'http://localhost:8080/test/index_test' + timeMod + '.html',
  userDataDir: false 
}).then(chrome => {
  console.log(`Chrome debugging port running on ${chrome.port}`);
});

Complete task of creating new screenshots and comparing them with the reference ones looks like this:

gulp.task('test-compare', function() {
  var timeMod = new Date().getTime();
  var clearDir = [diffDir, afterDir, 'test/']

  if (!fs.existsSync(afterDir)){
    fs.mkdirSync(afterDir);
  }
	
  if (!fs.existsSync(diffDir)){
    fs.mkdirSync(diffDir);
  }
	
  clearDir.map(function(element, index) {
    if (fs.existsSync(element)){
      fs.readdir(element, (err, files) => {
        for (const file of files) {
          fs.unlink(path.join(element, file), err => {});
	}
      });
    }
  });

  function doneReading(img1, img2, pageName) {
    var diff = new PNG({width: img1.width, height: img1.height});
	pixelmatch(img1.data, img2.data, diff.data, img1.width, img1.height, {threshold: 0.5});
	diff.pack().pipe(fs.createWriteStream(diffDir + pageName + timeMod + '.png'));
    console.log(pageName + ' ---- page compared');
  }

  function parse2(element, index, pageName) {
    img2[index] = fs.createReadStream(beforeDir + element + '.png').pipe(new PNG()).on('parsed', function() { doneReading(img1[index], img2[index], element)});
  }

  pageList.map(async function(element, index) {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();
    await page.setViewport({ width: initialPageWidth, height: 0 });
    await page.goto('http://localhost:1337/' + element + '.html');
    await page.screenshot({path: afterDir + element + '.png', fullPage: true});
    await browser.close();
    pageName = element;
    img1[index] = await fs.createReadStream(afterDir + element + '.png').pipe(new PNG()).on('parsed', function() { parse2(element, index)});
  })

	var imgList = pageList.map(function(file, i) {
		return '<li style="width: 49%; display: inline-block; list-style: none; background-color: #888;"><h2 style="font: 3vw sans-serif; margin: 0; padding: 1em; text-align: center;">' + pageList[i] + '</h2><img style="width: 100%; display: block;" src="difference/' + file + timeMod + '.png"/></li>'
	})

	fs.writeFile('test/index_test' + timeMod + '.html', imgList, function (err) {});

	var fileServer = new staticN.Server();

	http.createServer(function (req, res) {
		req.addListener('end', function () {
				fileServer.serve(req, res);
		}).resume();
	}).listen(8080);

	chromeLauncher.launch({
		startingUrl: 'http://localhost:8080/test/index_test' + timeMod + '.html',
		userDataDir: false 
	}).then(chrome => {
		console.log(`Chrome debugging port running on ${chrome.port}`);
	});
})

The task is started with the command:

gulp test-init

Review of results

If our new screenshots coincide with the reference ones, the result will look like this:

If there is a mismatching between the images (test in the elements on the Catalog page), the difference will be highlighted in red:

You can also change the tolerance of image comparison using the threshold parameter, which is set when activating pixelmatch; it can have levels from 0 to 1, where a smaller one corresponds to a greater strictness of comparison. In this example, it is activated inside the doneReading function with the level threshold: 0.5