I explain a little bit behind how pointerpointer.com was built.
Thx to @markwithasee to showing the site to me
Just to be clear about the canvas. it's offset in the page because its used as a buffer canvas, mouse positions within the picture frame are sent over to it, and the offsets are determined relative to the canvas. So when I turn on the canvas, it looks like its in the wrong place but technically it doesn't need to be.
Lastly, the colors used in the canvas are there because in order to find which shape encloses your pointer, we can just just look up the pixel data and see what color is under the cursor, and thus the connection from color to shape to image. (thx rhys!)