In
Supervised machine learning, input data is labeled with the correct answer.
The input data is split into two parts namely training data and test data. However,
imagine a situation when the data is not labeled with a correct answer. Suppose
you are given the Income Tax Returns of huge numbers of users and you wish to find
the anomalous return filers or outliers. Or let say you wish to map the users
into different clusters as per their demographic attributes. In such a situation,
data scientists opt for Unsupervised machine learning.
Unsupervised machine learning (UML) helps in
discovering hidden patterns in the data. For instance, let’s assume you have images
of a set of animals. UML model can help you in partitioning and clustering
different animals in different buckets without being explicitly trained about
their attributes and features. Let us understand how this magic happens.
Whenever a huge amount of unlabeled data is passed
to the model. The model finds certain common features in the data. On basis of these
common features, all the data points are clustered, partitioned, and categorized
into different sets. For example, suppose you have 1000 points with X and Y
coordinates and you wish to divide them into two clusters. In a primitive
setup, one can take the following 5 steps:
1. Initialize two hypothetical (X, Y) tuples m1 and m2
as the centroid of two empty buckets C1, C2.
2. For each point, find its distance from the two centroids.
3. If the distance to C1 is less than the distance to C2 then
put the point in bucket C1, else put the point in bucket C2.
4. Recalculate the mean points of the two buckets C1
and C2 and take them as new centroids m1 and m2.
5. Repeat step 2-3 till the time values in the buckets
are not stabilized.
In the end, you will have all the points in the data
space categorized in the two buckets C1 and C2.
A fine reader can spot the devils in the details in the above five steps. There are four devils sitting in these five steps.
1. How do you decide the value of K? The number of
clusters/buckets could be 2, or 3, or even N.
2. How do you define the distance function to calculate
distance between data points in a complex scenario?
3. When do you say that C1 and C2 are stabilized? How
do you calculate loss or distortion which has to be minimized for accurate
estimation of stabilization?
4. Who is going to validate the results?
One
should be careful in applying unsupervised machine learning to a problem space
due to the following challenges:
1. As the data is unlabeled, one can never be sure
about the accuracy or precision of the outcome.
2. There is no definitive outcome. For instance, if you
use the K value as 2, you get two clusters. If you use the k value as 5, you can get
five clusters. Every algorithm listed above has its own share of pitfalls.
3. It requires a better domain knowledge (expert) so
that one should be able to see the outcome and validate the results of
unsupervised machine learning.
Despite the above challenges, Unsupervised machine
learning has proved to be pathbreaking and revolutionary in the field of Data
sciences due to its widespread application. It can be useful in many scenarios.
1. Clustering: It is useful in clustering the data into
different clusters. For instance, looking at the ITR profiles of people, one
can cluster taxpayers into obedient and non-obedient taxpayers.
2. Anomaly detection: It can identify outliers in the
data. Any anomalous input can be spotted using this. For instance, any unusual
hike in refund claims or exemption claims can be spotted. Or it could be used
to detect signals from an alien planet or intrusion detection in the network
3. Association mining: It can be used to identify a set
of items that occur together in the dataset. For example, if people have a
tendency of buying bread-butter more frequently than bread-spices so a
shopkeeper can decide whether to put bread with butter or with spices.
4. Dimensionality-reduction: If a data point contains
1000s of attribute then it is difficult to analyze and study it. Dimensionality
reduction helps in reducing the dimensions of analysis. For instance, out of
1000 attributes/features, 10 important features could be drawn and studied.
Some
of the common algorithms/ models for achieving these objectives are listed
below.
Clustering:
A common approach for clustering is
K-MEANS clustering. The 5-steps process explained above is an example of a K-means
clustering algorithm where the value of K is 2. Other prominent approaches are
DBSCAN and OPTICS.
Anomaly
detection: Isolation forest, Local
outlier factors are commonly used for discovering anomalous patterns.
Association
mining: Apriori algorithm, and FP-growth
algorithm is prominently used in mining patterns from the unlabeled data.
Dimensionality
reduction: Principal component analysis
and Singular value decomposition are the most commonly used dimensionality
reduction approaches.
In recent years, unsupervised learning has been used vigorously in neural
networks. This has revolutionized the research and application of Machine
Learning and Artificial Intelligence. However, this requires a dedicated
article on its own.
