My goal is to write a program which recognizes the chess position in an image of a digital game. I'm not trying to process actual photos of a game in real life, just images like the own below enter image description here

This seems as simple of an image recognition task as could ever be hoped for, but my naivety of image processing and recognition techniques is making this a challenge for me.

The first step, of splitting the board into its 64 squares, seems like it should have a pretty straightforward solution, but I'm not sure what it is. My current approach is to convert the image to grayscale, threshold it, and then scan the rows and columns for alternating stretches of white and black pixels where the stretches are the same length (+/- a couple pixels to account for aliasing). Once I find such an area, I know the height of each square and the location of one of the edges, so I can start exploring up and to the left and down and to the right until I find a corner which doesn't seem to be in the middle of four squares, in which case I've found the upper left and bottom right corners.

The above solution, with an unfortunate amount of tweaking, does at least get me to the point where I have the coordinates of each of the squares. From there I need to recognize the piece type and to get its color. For the piece type, I considered a neural network, but that seems like overkill and I don't even know what training data I could gather for it since there's only one actual image for each piece. I tried both Histogram of Oriented Gradients and Hu Moments to classify the pieces, but both did a poor job of differentiating pawns and bishops. This may just be an issue with the parameters I used for those algorithms, but I did try a decent amount of different combinations.

I'm also having trouble finding the color of the pieces. Simply counting the number of dark vs light pixels in the outline of the piece doesn't work too well because the inner details of the pieces actually makes the count of light vs dark pixels pretty close, especially after preprocessing the image to try to smooth out the texture of the square backgrounds. My current approach is, again, pretty brutish - to have an expected ratio of light to dark pixels for a given piece type on a given background square color.

Overall, it feels like I'm greatly over-complicating what should be a simple task. I would appreciate any advice on:

  1. Getting the location of the corners of the board and the length of each of the squares
  2. Determining which piece is in a square
  3. Determining the color of the piece

I'd like to end up with a solution which can handle multiple piece types and square colors. I wouldn't need to be able to handle new variations of board types on the fly, but would like to be able to update the program to handle them. For example, if I had something working for the first image, I'd want to be able to adjust it to also work on the one below: enter image description here

This is for a project I'm picking back up after not working on it for some time, so I don't have the specifics of the parameters I used for HoG or Hu, or the processed images.


1 Answer 1


If you have no/ limited variability, then train a multinomial regression or neural network on each piece, centred.then you just sweep/place the network over each square. ( Standardized to a fixed size)... You might want to Google, correlation filters too. ( Ie you should do template matching, not hog etc)..

Apart from standardising the size/location, you should standardize the colour...ie black on brown square, black on white square etc .. perhaps by doing Laplacian filter first?

  • $\begingroup$ Thanks for this, I hadn't considered doing multiple passes over the square, but that does make a lot of sense $\endgroup$
    – mowwwalker
    Commented Jan 20, 2019 at 15:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.