As undergraduate participants of the Indiana University Computer Vision Lab, we were tasked with discovering ways to combat the problem identifying counterfeit integrated circuits (ICs). Counterfeit ICs are essentially electronics that are made to look and act like regular ICs, but may be less expensive to produce or even contain malicious components. One of the ways to find counterfeit ICs it to simply count the number of ICs on a circuit board and compare that number to what it should be. However, this is extremely time consuming to do manually with the human eye.
In research conducted by Cazamias and Marek at Stanford University, the authors attempted to count the number of empty parking spaces in images of parking lots. They did this in two ways, the second of which would be the basis for my attempt at counting cars instead; here they trained a CNN on 70%/20%/10% train/test/validation split of 12,417 images of parking lots to yield a model achieving an accuracy of about 81%. The images of the parking lots consisted of three different camera angles of two different parking lots. Each was labeled with the number of cars present.
The CNN, which I reconstructed from scratch in TensorFlow/Keras, was composed of 3 convolutional layers of depths 10, 20, and 30 nodes respectively. A 5 x 5 kernel was used at each convolutional layer, and each was followed with batch normalization, a ReLU activation function, then max pooling of size 2 x 2. After these three layers, the input was flattened and fed to three fully connected layers of size 30, 20, and 10 respectively with ReLU activations after each. A softmax layer was used to classify a "count" of the number of empty spaces or cars. in their research they classified between 0 and 20 empty spaces, saying that anything over 20 empty spaces would matter in application since it would be easy to find a parking spot; however, my research used classes between 0 and 100 since there were no images in which the number of cars exceeded 100. This, I'll reason, could be why my accuracy was much lower than theirs on their problem.
My CNN would ultimately achieve an accuracy of about 60%, though given the differences in difficulty between our problems (as I reason above my random benchmark would be 1/100 while theirs was 1/20), it's fair that my accuracy be lower than their 81%. I also surpassed their accuracy, achieving 84% counting empty spaces, by implementing a technique on the data that I used to increase the accuracy on my problem.
I tried several things to increase an accuracy of ~55% initially. The first was using a continuous predictor instead of discrete classifier. This produced similar, but worse results. I also tried using color instead of grayscaled images, as is often done to speed up training time. This did help a bit, but results weren't much better. Finally, the most prroductive of my efforts were in manipulating the images before training. I had noticed that some images labeled as having 0 cars, actually often contained some. So, looking at details of the dataset, I found that only cars within certain coordinates of the parking lots were counted. Therefore, I cropped the pictures to bound only those coordinates. I would argue this to be legal, because for counting ICs on an integrated circuit, one would be taking many, practically identical pictures, and could crop the images to what they knew could contain ICs. This is what ultimately lead to my highest accuracy of 60% and how I achieved 84% accuracy on the previous reachers' task, compared to their 81%.
I used ten iterations of Monte Carlo Cross Validation to validate these results. The standard deviation among trials was low.
With the cropped photo method detailed above, I graphed the following frequencies of car count labels and car count predictions using the trained model in order to visualize their distributions. I'd say they are fairly similar, enough to conclude that the model wasn't just guessing and getting lucky.
There are several things that can be done to improve this model. Since there seems to be high bias, further meta-analysis of the model and its results could be done to see if the bias-variance trade-off is biased towards bias (haha). If this is the case the architecture could be made deeper or regularization parameters relaxed, since high bias is indicative of underfitting. Another interesting approach would be to use transfer learning, perhaps with a model trained on ImageNet, that might help the algorithm identify cars.
However, if I do choose to proceed with this project, I'll probably try to bring it back to the original problem of counting ICs. Perhaps after some adjustments, transfer learning from this model to a model trained on an IC/circuit board dataset (During this research I helped our lab label ICs on about 1000 images of circuit boards.) Regardless, it was a fun project, and I'm glad I got to test my machine learning knowledge and learn a lot more in the process.