AI Predictions with Simple Statistical Methods

Greg Surma

Apr 22, 2019·6 min read

In today’s article, we are going to build a prediction system with simple statistical methods. We are going to show that Machine Learning applications aren’t limited to neural networks only and we can achieve a decent predictive behavior with Computational Statistics. By the end of this article, you will be able to implement a very basic, yet impressive prediction system.

Inspiration

Caged Brain project was heavily inspired by Scott Aaronson’s Oracle:

In a class I taught at Berkeley, I did an experiment where I wrote a simple little program that would let people type either “f” or “d” and would predict which key they were going to push next. It’s actually very easy to write a program that will make the right prediction about 70% of the time. Most people don’t really know how to type randomly. They’ll have too many alternations and so on. There will be all sorts of patterns, so you just have to build some sort of probabilistic model. Even a very crude one will do well. I couldn’t even beat my own program, knowing exactly how it worked. I challenged people to try this and the program was getting between 70% and 80% prediction rates. Then, we found one student that the program predicted exactly 50% of the time. We asked him what his secret was and he responded that he “just used his free will.”

Scott Aaronson, Quantum Computing Since Democritus.

Caged Brain

We are going to build a predictive system encapsulated in a very simple game. While the user, is going to pick either ‘LEFT’ or ‘RIGHT’, the computer is going to analyze user’s moves and predict the next one. User’s goal is to be unpredictable and make computer fail in its predictions (green color). The lower the computer’s prediction % the better!

I recommend you play the game yourself and explore its mechanics on your own.

Oracle

Caged Brain’s prediction system (oracle) has ~70% accuracy. How can it be achieved with just a crude probabilistic analysis of user inputs?

First of all, let’s look at what oracle sees. It’s a sequence of ‘Left’ and ‘Right’ taps. For the sake of simplicity, let’s store it with an array of Bool values, where ‘Right’ is true and ‘Left’ is false.

Let’s consider a scenario where the user already tapped 10 times. An example of oracle’s memory can look like the following.

[false, true, true, false, true, false, true, false, true, false]

It may not look like a lot of data to process but actually it’s enough to make a decent prediction what the next move will be!

Before we proceed, we need to make ourselves familiar with the concept of n-gram.

N-Gram

Given a sequence s of some values, n-gram of it is a continuous sequence of n values from sequence s.

Let’s compute all 5-grams from our oracle’s memory.

Why 5?

It’s just an arbitrarily selected value that allows detecting patterns in human’s actions in this specific example. Feel free to experiment with different values!

Let’s go back to our scenario and compute all 5-grams.

These are all 5-grams (lines 3–7) from our oracle’s memory of 10 actions (line 1) formatted in a way that allows better understanding of how they were derived.

How can we make any sense out of them?

Statistical Evaluation

In order to predict user’s next move, we have to take a look into the history of his moves once again. To predict the user’s next action, we need to take n-1 last moves and extend them with available possibilities.

trueGram = [false, true, false, true, false] + [true]
falseGram = [false, true, false, true, false] + [false]

With that operation, we are going to have two possible n-grams reflecting user’s next move.

Our next step will be to verify these two possibilities according to the likelihood of their occurrence in the next move, given the history of the previous moves.

Assuming that humans usually don’t act completely randomly and are likely to follow patterns, we can predict that user’s next move will be the one that is associated with the n-gram that appeared more frequently in the past.

In order to do it, we are going to iterate over all gathered n-grams and for every n-gram verify whether it matches true/falseGram. To keep track of this, we are going to keep a balance and increment it for every trueGram match and decrement for every falseGram one.

If the balance is over 0, it means that in our history trueGram happened more often than falseGram (similarily with falseGram when below 0).

We can achieve the above logic with the following code.

Going back to our example - what would be the oracle’s prediction? Let’s take a look at its analysis consisting of iterating over all gathered n-grams and incrementing/decrementing balance accordingly.

We got a trueGram match (line 13) and no falseGram ones.

Positive balance implies that it should predict true which was a correct prediction as a user also tapped Right (true)!

Summary

Caged Brain achieves on average about 70% accuracy which implies that humans usually aren’t able to act randomly. Truly random behavior wouldn’t allow the oracle to learn any patterns, thus its accuracy would be around 50% in a long term.

Oracle performs even better when dealing with very basic and predictable strategies like tapping only left, only right or alternating.

What’s Next?

With the Caged Brain project, we’ve shown that decent predictive performance can be achieved with very simple computational statistics methods. What’s more, it turned out that we humans have a very hard time generating truly random behavior. If you think otherwise, I dare you to go below 50% in Caged Brain!

Don’t forget to check the project’s iOS game.

Questions? Comments? Feel free to leave your feedback in the comments section or contact me directly at https://gsurma.github.io.

And don’t forget to 👏 if you enjoyed this article 🙂.