➩ Supervised learning (data is labeled): use the data to figure out the relationship between the features and labels of the items, and apply the relationship to predict the label of a new item.

➩ If the labels are discrete (categories): classification.

➩ If the labels are continuous: regression.

➩ Unsupervised learning (data is unlabeled): use the data to find patterns and put items into groups.

➩ If the groups are discrete: clustering

➩ If the groups are continuous (lower dimensional representation): dimensionality reduction

➩ Items need to be represented by vector of numbers, item \(i\) is \(\left(x_{i1}, x_{i2}, ..., x_{im}\right)\).

➩ Some features are already represented by real numbers, sometimes need to be rescaled to \(\left[0, 1\right]\).

➩ Some features are categories, could be converted using one-hot encoding.

➩ Images features and text features require additional preprocessing.

➩ A text document needs to be split into words.

➩ Split the string by space and punctuation, sometimes regular expression rules can be used.

➩ Remove stop words: "the", "of", "a", "with", ...

➩ Stemming and lemmatization: "looks", "looked", "looking" to "look".

➩ nltk has functions to do these: Link, for example, nltk.corpus.stopwords.words("english") to get a list of stop words in English, and nltk.stem.WordNetLemmatizer().lemmatize(word) to lemmatize the token word.

➩ A word token is an occurrence of a word.

➩ A word type is a unique word as a dictionary entry.

➩ A vocabulary is a list of word types, typically 10000 or more word types. Sometimes <s> (start of sentence), </s> (end of sentence), and <unk> (out of vocabulary words) are included in the vocabulary.

➩ A corpus is a larger collection of text (like a DataFrame).

➩ A document is a unit of text item (like a row in a DataFrame).

➩ Bag of words features represent documents as an unordered collection of words: Link.

➩ Each document is represented by a row containing the number of occurrences of each word type in the vocabulary.

➩ For word type \(j\) and document \(i\), the feature is \(c\left(i, j\right)\), the number of times word \(j\) appears in document \(i\).

➩ TF-IDF or Term Frequency Inverse Document Frequency features adjust for the fact that some words appear more frequently in all documents: Link.

➩ The term frequency of word type \(j\) in document \(i\) is defined as \(\text{tf} \left(i, j\right) = \dfrac{c\left(i, j\right)}{\displaystyle\sum_{j'} c\left(i, j'\right)}\) where \(c\left(i, j\right)\) is the number of times word \(j\) appears in document \(j\), and \(\displaystyle\sum_{j'} c\left(i, j'\right)\) is the total number of words in document \(i\).

➩ The inverse document frequency of word type \(j\) in document \(i\) is defined as \(\text{idf} \left(i, j\right) = \log \left(\dfrac{n + 1}{n\left(j\right) + 1}\right)\) where \(n\left(j\right)\) is the number of documents containing word \(j\), and \(n\) is the total number of documents.

➩ For word type \(j\) and document \(i\), the feature is \(\text{tf} \left(i, j\right) \cdot \text{idf} \left(i, j\right)\).

➩ sklearn is the Python package for machine learning: Link.

➩ sklearn.feature_extraction.text.CountVectorizer can be used to convert text documents to a bag of words matrix: Doc.

➩ sklearn.feature_extraction.text.TfidfVectorizer can be used to convert text documents to a bag of words matrix: Doc.