HTML5 Canvas – Rendering web video to tiles

September 2013

Software & Technologies

HTML5, Canvas, JavaScript, Web Video, Video Effects

Article

Recent experiments with HTML5 video and canvas have proven very interesting. The HTML5 canvas can sample pixels from images and video files. You can create color effects, and tile effects, etc., by using multiple canvas elements in unison.

In general, be careful when rendering video to the canvas, as it is processor intensive. That said, there are effects that can only be created by sampling video frames, and rendering them as pixels on the canvas. Once such effect is rendering a video to multiple tiles (see Figure 1).

video_tiles

Figure 1: HTML5 video rendered as tiles on a canvas

This post will walk through the theory and code structure related to rendering video to the HTML5 canvas as a tile effect. I’ll cover creating the basic tiles, then making them draggable.

You can download working sample files from this link to use as reference.

The big picture

The first thing to understand is that there are several elements required to render formatted video pixels to a canvas. The pieces to the puzzle include:

  • A <video> tag: Used to render the video to the page
  • A <canvas> tag (copy): Used to copy the pixels from the video element
  • A second <canvas> tag (draw): Used to copy formatted pixels from the first canvas

The video tag and the first canvas tag (copy), are hidden from the screen using CSS. The video tag downloads the video. The first canvas is used to copy the pixels of the entire video frame to the canvas. The second canvas (draw) is used to copy pixels from the first canvas, and apply them to itself as the visible canvas element. This last step is where color transformations are applied, or in this case, tiles of video pixels are sampled as desired.

The code

Once you’re familiar with the elements at play, it’s simply a matter of setting up the HTML code, and a bit of JavaScript to manipulate the canvas elements.

The basic HTML code template looks like this:

<body>
<div id="video-wrapper">
    <video id="video-source" loop autoplay width="600" height="337">
        <source src="video/jellys.mp4" type="video/mp4">
        <source src="video/jellys.webm" type="video/webm">
        <source src="video/jellys.ogv" type="video/ogg">
    </video>
    <canvas id="video-copy" width="600" height="337"></canvas>
    <canvas id="video-output" width="600" height="337"></canvas>
</div>
<script>
	function buildTiles{}
	function drawTiles{}
</script>
</body>

The page loads the jQuery library in the head tag, and then loads the video tag and two canvas tags in the body tag. At the bottom of the body tag, a script tag contains two empty functions; buildTiles and drawTiles. The buildTiles function is called once, at the start, to calculate the coordinate space of the tiles. The drawTiles function is called on an interval, and handles the job of copying pixels from the video element, and then pasting them as tiles into the visible canvas. From here, all the following code samples appear in the script tag.

Setting up the variables

You need a handful of variables to keep track of things.

The variables look like this:

    var video = $("#video-source")[0],
        canvasToCopy = $("#video-copy")[0],
        copy = canvasToCopy.getContext("2d"),
        canvasToDraw = $("#video-output")[0],
        draw = canvasToDraw.getContext("2d"),
        rows = 3,
        cols = 5,
        space = 4,
        tileWidth = Math.round(canvasToCopy.width/cols),
        tileHeight = Math.round(canvasToCopy.height/rows),
        tileCenterX = tileWidth/ 2,
        tileCenterY = tileHeight/ 2,
        tiles = [];

The first five variables define references to the video element, and the canvas elements. The copy and draw variables are references to each canvas’s Context2D object. This is important because they contain the drawing API for the canvas.

The rows, cols, and space variables define how many rows and columns appear in the tile grid, along with the spacing between tiles. The tileWidth, tileHeight, tileCenterX, and tileCenterY variables are used to calculate the size and position of each tile. This information is saved in the tiles list, which defines the default state of the tile layout.

Setting up the buildTiles function

The buildTiles function accomplishes two things; it creates a list of coordinates for each tile defining the tile’s position in the unaltered video, as well as the tile’s position on the canvas. It also hides the video and “copy” canvas elements.

The buildTiles code looks like this:

    function buildTiles(){

        var tileX = 0;
        var tileY = 0;
        var row = 0;
        var col = 0;

        for(var i=0; i < rows; i++){
            for(var j=0; j < cols; j++){

                // Save a tile object containing the location of the
                // tile in the original video frame, and the location
                // it should be drawn to on the output canvas
                var tile = {
                    videoX: tileX,
                    videoY: tileY,
                    x: (tileCenterX+tileX)+(space*col),
                    y: (tileCenterY+tileY)+(space*row)
                };
                tiles.push(tile);
                tileX += tileWidth;
                col++;
            }
            row++;
            col = 0;
            tileX = 0;
            tileY += tileHeight;
        }

        // Hide video and copy canvas
        $(video).css("display", "none");
        $(canvasToCopy).css("display", "none");
    }

Notice that the function creates a loop that cycles through the number of columns and rows to create a list of tile coordinates. The drawTiles function uses these coordinates to render the video to the visible canvas.

Setting up the drawTiles function

The drawTiles function is in charge of copying the copy each frame from the video element, and reconstructing it in tile format on the visible canvas. The process includes copying the video frame to the first canvas, clearing the pixels on the second canvas, then copying the tiles one by one to the second canvas for display. This function is called repeatedly on an interval to create the effect.

The code for the drawTiles function looks like this:

    function drawTiles() {
        if (video.paused || video.ended) {
            return;
        }
        // Draw the current video frame into the invisible canvas
        copy.drawImage(video, 0, 0, canvasToCopy.width, canvasToCopy.height);

        // Clear the output canvas
        draw.clearRect(0, 0, canvasToDraw.width, canvasToDraw.height);

        // Copy tiles from invisible canvas and draw them to the output canvas
        var len = tiles.length;
        for(var n=0; n < len; n++){
            var tile = tiles[n];
            draw.save();
            draw.translate(tile.x, tile.y);
            draw.drawImage(canvasToCopy, tile.videoX, tile.videoY, tileWidth, tileHeight, -tileCenterX, -tileCenterY, tileWidth, tileHeight);
            draw.restore();
        }
    }

Notice that the drawTiles function loops through the tiles list after the second canvas is cleared.

Starting the rendering

At the very bottom of the script, there are two lines of code that start rendering the effects.

The last bit of code looks like this:

    // Initialize
    buildTiles();

    // Render video on interval
    setInterval(drawTiles, 1000/30);

Here I’m calling the buildTiles function to calculate the coordinate space, then I’m starting an interval that calls the drawTiles function at 30 frames per second. If you look in the drawTiles code, you’ll notice that it only executes if the video is playing.

Check out the example_videotiles.html file in the supplied samples for a working file.

Making the tiles draggable

The next level of working with the effect, is to make the tiles draggable. Ultimately this is just an illusion as you’re really just dragging the coordinate space and then altering the tile’s data in the tiles list. If that seems a little confusing, don’t worry. The thing is that the canvas is a flat 2-dimensional object – you’re not really dragging the tiles, you’re just altering the coordinate space and depth order that then render in.

To accomplish the effect, you’ll add a few new variables followed by four new functions:

  • startDrag:
  • drag:
  • endDrag:
  • resetTiles:

Setting up the variables

You need to add four new variables at the bottom of the variables declaration.

The new variables look like this:

var  dragging = false,
        dragPoint = {x:0, y:0},
        selectedTile = null,
        topIndex = 0;

The dragging variable keeps track of whether you’re dragging a tile or now. The dragPoint variable is referenced by each tiles to determine its drag offset. The selectedTiles variable keeps track of which tile is currently in focus. The topIndex variable is used maintain the stacking order of overlapping tiles.

Setting up the functions

The buildTiles function needs to be updated with references to origin position, drag point, and zindex. I’m not showing the updates here, so compare the supplied files to see where the changes are.

The startDrag, drag, and endDrag functions respond to the mousedown, mousemove, and mouseup events respectively. Essentially, these functions are updating the coordinate space values saved in the tiles list with the offsets related to dragging and dropping a tile. When the drawTiles function renders the canvas, it uses these offsets to draw the tiles in the correct location and stacking order.

The startDrag function looks like this:

    function startDrag(e){

        var tilesUnderPoint = [];
        var len = tiles.length;
        var offset = $(canvasToDraw).offset();
        dragPoint = {x: e.pageX-offset.left, y: e.pageY-offset.top};

        // Save all tiles under mouse point for evaluation
        for(var n=0; n < len; n++){
            var tile = tiles[n];
            if((tile.x-tileCenterX <= dragPoint.x && (tile.x+tileCenterX) >= dragPoint.x) &&
                    (tile.y-tileCenterY <= dragPoint.y && (tile.y+tileCenterY) >= dragPoint.y)){
                tilesUnderPoint.push(tile);
            }
        }
        // If one or more tiles are selected, find the tile at the
        // top zindex, then set the order of the array based on zindex.
        // Placing the selected tile at the end of the drawing order
        // ensures that it will be drawn above the other tiles while
        // dragging.
        if( tilesUnderPoint.length > 0 ){
            function sortZIndex(a, b){
                return a.zindex- b.zindex;
            }
            dragging = true;
            topIndex++;
            tilesUnderPoint.sort(sortZIndex);
            selectedTile = tilesUnderPoint.pop();
            selectedTile.dragPoint = {x:selectedTile.x, y:selectedTile.y};
            selectedTile.zindex = topIndex;
            tiles.sort(sortZIndex);
        }
    }
    $(canvasToDraw).on("mousedown", startDrag);

Notice that the function loops through the tiles and creates a list of tiles that are underneath the point of the click. From there, it determines which tile is in the top depth in stacking order, and marks that tiles as the selected tile.

The drag function looks like this:

    function drag(e){
        if (dragging) {
            var offset = $(canvasToDraw).offset();
            selectedTile.x = selectedTile.dragPoint.x + ((e.pageX-offset.left) - dragPoint.x);
            selectedTile.y = selectedTile.dragPoint.y + ((e.pageY-offset.top) - dragPoint.y);
        }
    }
    $(canvasToDraw).on("mousemove", drag);

This function specifically updates the x and y tile coordinates of the selected tile. The coordinate offset is mapped to the saved tile data, then rendered to the screen in the drawTiles function.

The endDrag function looks like this:

    function endDrag(e){
        dragging = false;
        selectedTile = null;
    }
    $(canvasToDraw).on("mouseup", endDrag);

Not much happens here except a bit of cleanup.

And, finally, as a nicety I included the resetTiles function which resets all the tiles to their original coordinate space when you double-click on the canvas.

The resetTiles function looks like this:

    function resetTiles(e){
        var len = tiles.length;
        for(var n=0; n < len; n++){
            var tile = tiles[n];
            tile.x = tile.originX;
            tile.y = tile.originY;
        }
    }
    $(canvasToDraw).on("dblclick ", resetTiles);

And that’s about it.

Check out the example_videotilesdrag.html file in the supplied samples for a working file.