University | University of London (UOL) |
Subject | DSM010 Big data analysis Course Work |
DSM010 Big data analysis Course Work, UOL, Singapore: Implement the K-Means clustering algorithm with Euclidean and Manhattan Distance Measures
Q2) Cluster Analysis using Apache Mahout.
For this question, you can optionally use the data (a set of text files that are placed in a folder) provided with Topic 4 for the K-Means algorithm. You are welcome to use your dataset for this question. If you choose to do so, please provide a link to the data in your report.
As we discussed in text clustering (Topic 4), the terms of the documents are considered features in text clustering. The vector space model is an algebraic model that maps the terms in a document into n-dimensional linear space. However, we need to represent textual information (terms) as a numerical representation and create feature vectors using the numerical values to evaluate the similarity between data points.
Use Apache Mahout and perform the standard steps for the cluster analysis,
1) create sequence files from the raw text,
2) create a sparse (efficient) representation of the vectors, initialization approximate centroids for K-Means,
3) run the K-Means algorithm,
4) get the final iteration’s clustering solution
5) evaluate the final solution
You need to consider the following points in the analysis:
• Implement the K-Means clustering algorithm with Euclidean and Manhattan Distance Measures.
• Find the optimum number (K) of clusters for the K-Mean clustering for the above distance measures.
• Implement K-Mean clustering algorithm with Cosine Distance Measure and
verify the relation between the average distance to the centroid and the K value.
• Plot the elbow graph for K-Mean clustering with Cosine Measure. Try to
smooth the graph so that you can explain the value for K as the best.
• Compare the different clusters you obtained with different distance measures and discuss what is the best setting for K-Means clustering for this dataset.
Hire a Professional Essay & Assignment Writer for completing your Academic Assessments
You need to include the following in your coursework submission:
(a) For Q1 submit the pseudo code and Python code for the mappers and reducers implementation for all of the descriptive statistics, along with some comments so that a layperson can implement. Anyone should be able to run your code and reproduce your results with the instructions that you have provided.
(b) For Q2, write a brief summary of the impact of parameter changes on the
performance of the K-Means algorithm. For example, you may: 1) compare different distance measures in the K-Means algorithm discuss the merits and demerits and 2) present a table that shows the performance of the K-Means algorithm for different K values.
(c) Submit a report on the experiments. This report will be a detailed explanation (Max 1500 words, excluding code and references) of what you explored, the results you obtained, and some discussion points on the limitations of MapReduce methodology and Hadoop’s MapReduce computing engine.
Credit will be given to:
• The depth and breadth of your investigation.
• The technical skills you demonstrate in your write-up.
• Good use of the Hadoop cluster.
• Critical evaluation of your work.
Buy Custom Answer of This Assessment & Raise Your Grades
Tags:-
- Imagine that you are currently working for a precision medicine startup: Machine Learning Paper Review in Precision Medicine, Written Assignment 1, NUS, Singapore
- Go Business offers PSG solutions for enterprises in Singapore: Collective Intelligence and Entrepreneurship, Assignment 1, JCU, Singapore
- Design an ontology based on- Accidents can be categorised as chemical, electrical, fire, kinetic or liquid: Collective Intelligence and Entrepreneurship, Assignment 1, JCU, Singapore
- Project Control Monitoring, Assignment, HU, Singapore: Deliberate Project MONITORING AND CONTROL or PMC Identification
- DSM500: Final Project Report, Coursework 2, UOL, Singapore
- DSM080: Financial Markets, Assignment, UOL, Singapore: A grapefruit juice futures contract is for 15,000 pounds of frozen grapefruit juice
- DSM080: Financial Markets, Assignment, UOL, Singapore: The share price of a certain stock today is $42.50, and five-month European style call options with a strike price of $45 currently sell for $4.25.
- DSM080: Financial Markets, Assignment, UOL, Singapore: A trader who is working in the gold markets is able to borrow money at the interest rate of 7% per annum
- DSM080 Financial Markets, Assignment, UOL, Singapore: A dollar-based American corporation has decided that it will have to pay 6 million UK pounds in three months
- Principle of Finance Assignment, UCD, Singapore: National Gaming Inc. (National) operates the weekly lottery in the country
UP TO 15 % DISCOUNT