Yes I had to make a trailer for it!

Practical Machine Vision for Impractical Curiosity: An ATM Surveillance Deep Dive

You know when you just wanted to try something quick to see if it will work and you end up going down a rabbit hole for a couple of weeks? Well this is how this project started and this blog is how it ended. I wanted to see how computer vision can be used where people use equipment unattended. At ATMs, petrol stations, shopping tills or even cafeterias, the list goes on. Well I chose ATMs since I’ve been in banking longer than I would care to admit. 

So lets get some things cleared up first. I needed some dataset to test out all my theories on. Turns out, well from looking for 10min online its a bit hard to get available footage of ATMs online for obvious reasons. However I happen to find this one VERY low resolution (230×250, what is that 0.0000003k or something) video on you tube of all places. And the reason the video is so low resolution is because is 11 years old! So this video as been online for 11 YEARS! So did not feel to bad using this for testing. However not trying to contribute to sharing details of people who I suspect never opted in to have their faced shown publicly (even if you can not really make out faces in this small scale video) I opted to blur everyone before creating anything that will be shown in this blog. The video does show a crime happening where card theft is taking place, but I opted to blur everyone none the less. 

I would also like to say all of the POC’s below is just proof of concepts. They are intended to be run in real time, if they actually can, well that is like sales making promises the coders need to keep. I worked on the POC ideas not trying to implement this in a live system. However like I said most of these should be able to run real time and might not even need a GPU to run. 
On the accuracy of the POC’s I would like to say, well, its POC so the concept works fine, however in a life PROD environment one might have different results in some cases, in other cases you will most defiantly have even better results as I did not spend much time fine tuning each setup to the best it can be, again POC! 

Below are some of the use cases I came up with, with some being more practical than others. 

Detecting stationary objects

I wanted to see if it was possible to detect if someone might have left something behind, on purpose or by accident. I will let the reasons for purposefully leaving behind objects to your own imagination. The original video clip did not have anything that was stationary for long enough to use so I added two objects manually in the video. I did not bother putting the objects behind the people as it was sufficient for my needs.  However in a real life situation its possible to detect the same objects over time even if its occluded for short periods of time. This way should something be left alone for a while, not only will it be detected it will also be identified and possibly be treaded with a greater criticality should the item be identified as a weapon for example. 

Below is the full video clip, feel free to skip ahead to see the objects being detected in real time. 

 

Removing humans from the picture

Say you needed to know where people are and what they are doing, but can not show the people, even to security personnel, well then all you need to do is replace the humans with their skeletons, no really. Using basic slight of hand you can have security cameras watching people without letting known who those people are. Each person can be switched to a representation of that person and even be used for additional extraction of information. For example the angle of each body part, where the head is pointing or even hand poses. This clip was to low resolution to be able to extract head or hand details, however with any camera from say 1080p upwards this would be very easy todo.  Below you can see the full video clip play out, but with no humans anywhere to be seen… whoooooooo… spoookkyyy! 

 

Heat map generation based on population location.

I presume you have heard the saying from some research paper saying 20% of the floor space is being user 80% of the time. That is a thing if you think about it. Look at any office space or anywhere in your house 80/20 will be there! However proving it is another issue, I mean if you really feel the need to prove such a thing on a weekend instead of going outside or something. This method I created is not very complex, but works rather well and I’m sure you can figure out the “formula” for yourself so I wont go into the technical details (see bottom of the page for that). Its remarkable to see, even with just a four minute video clip patterns start to emerge where people stand the most and where they do not or just walk past. In this case you can clearly see spots at the ATMS where people tend to stand. This method can also be user to see where lines form and how they from for perhaps queue management or product placement. The nice thing is you can also choose the time frame, daily weekly even per second, it all depends on your use case. 

 

Counting machine usage

 Ever wanted to know how may people use what machine, be it a washing machine, ATM or hotdog cooker? Well now you can. By combining a couple of machine vision models and some basic logic you can count every person using any machine or stand at a teller or anything your imagination can come up with. This POC is not the most accurate as you would need to track one person for a long time and if the tracking goes, even for one frame the AI will see that person as a new client and count that person again. But for a POC it worked rather well I must say. Obviously you can then feed the number of users to any manner of downstream systems for any manner of use cases. 

 

Detecting loitering at ATMs

Now this use case can be used at ATMs specifically, however I’m sure seeing if one person is hanging around for a long time in any space where they should not be would be beneficial. I’ve heard of people hanging around ATMs waiting to their “assist” someone in withdrawing money or just trying to look over peoples shoulders all the while hanging around a place way longer that any other person normally would. By combining the speed and idle duration you can quickly find those who has not moved for a while, either from a single spot as in the case below, or some a designated area like an hotel lobby for example. And once a preset trigger has gone off you can choose how to resolve the situation. From locking doors, ATM’s or even just turning off the lights on the turning on the sprinkler system.  

 

3D Data and position extraction from 2D video

Now this POC was more of a see if I can than having a real world use case for it, but it was fun building the custom code for this. Turn out most of the python libraries can do something right, but not other so I gave up and just created a new algorithm to extract depth from a 2d image and then project it into space. If I spent a little bit more time on this I would have been able to place the exact spot of each person on a top down view, however I would have needed to add some additional trigonometry and obtained some actual measurements form the physical environment fore re projecting the people’s locations from say a top down view, however simply by only using the tiny video file I was able to extract a decent estimate at least who was standing in front who who and when. I went from point cloud to mesh then extracting only the people and their relative position in 3d. Something you would see in some CSI series or something. I just say I think this is the only POC that would take some time, but not impossible to run in real time, however with a decent GPU anything is possible. Even so this version you see below runs at 30 FPS, this is the video frames, and the render speed was about the same if you can get your head around that. 

 

Behind the scenes

I did not want this blog to be a technical blog where I would loose 90% of those who where not uber nerds like me. (Yes I’m proud to be one! Call me if you want to chat about particle physics over a braai) However I would like to share a combined view of about 50% of the things that happens in the background of the POC’s above. If you still do not get what is happening at least its cool to look at and might make a interesting techno art music video. 

That being said for those of you who wish to deep dive into the code and look under the hood here is your chance simply click the link below: