Today I wanted to write an article to talk about a topic of growing popularity in the business intelligence market: machine learning. Why do we need machine learning? What are the kinds of things we use it for? What are some examples of what machine learning can do?
The reason we need machine learning is that there are some problems where it's very hard to write software program to solve. Take recognizing a three-dimensional object, for example. When it's from a novel viewpoint in new lighting conditions in a cluttered scene, it is very hard to do for a non-human system. We didn't know what program to write because we didn't know how it's done in our brain. And even if we did know what program to write it might be that it was a horrendously complicated program.
Another example is detecting a fraudulent credit card transaction where there may not be any nice simple rules that will tell you it's fraudulent. You would need to combine a very large number of not very reliable rules, and also those rules change over time because people change the tricks they use for fraud. So we need a complicated program that combines unreliable rules, and those rules need to be changed easily.
How the Machine Learning Approach Works
With the machine learning approach, instead of writing a program by hand for each specific task, for a particular task we collect a lot of examples that specify the correct output for giving an input. The machine learning algorithm then takes these examples and produces a program that does the job. The program produced by the learning algorithm may look very different from a typical handwritten program.
For example, it might contain millions of numbers about how you weigh different kinds of evidence. If we do it right, the program should work for new cases as well as the ones it is trained on. And if the data changes we should be able to change the program relatively easily by retraining it on the new data. The beauty of this approach is that masterminds of computation are cheaper than paying someone to write a program for a specific task. Therefore we can afford big complicated machine learning programs to produce these task-specific systems for us.
An example of the things that are best done by using the learning algorithm approach is recognizing patterns. For example, this could be recognizing objects in real-life scenes, or the identities or expressions on people's faces, or spoken words.
ML Can Recognize Anomalies
Another example is recognizing anomalies. For instance, an unusual sequence of credit card transactions will be an anomaly. Another example of an anomaly would be an unusual pattern of sensor readings in a nuclear power plant, and you wouldn't really want to have to deal with those anomaly detection analyses by doing supervised learning, which is where you look at the cases that led to an accident and see what caused them to blow up. You'd really like to recognize that something funny is happening without having any supervision signal. You want to know when the system is just not behaving in its normal way.
Another category of machine learning applications is for prediction. Good examples are predicting future stock prices or currency exchange rates. Or it could be predicting which movie a person will like from knowing which other movies they like and which movies a lot of other people liked. All of these examples make for good data visualizations in order to quickly spot the anomalies or trends or clusters.