Adds NLTK & Spacy to datascience-notebook
811
title: Text Mining for Social Sciences author: Nandan Rao date: April, 2019 ...
When solving a problem of interest, do not solve a more general problem as an intermediate step. Try to get the answer that you really need, but not a more general one. \hfill (Vladimir Vapnik)
Statistical Modeling: The Two Cultures
\url{https://projecteuclid.org/download/pdf_1/euclid.ss/1009213726}
Let's say you are inventing search. Imagine someone searching for the term "People who see ghosts". How could you pick between the following?
Let's try again with the term: "People who see ghosts"
"I don't believe people who see ghosts", said Mannie, before spitting into the wind and riding his bike down the street at top speed. He then went home and ate peanut-butter and jelly sandwiches all day. Mannie really liked peanut-butter and jelly sandwiches. He ate them so much that his poor mother had to purchase a new jar of peanut butter every afternoon.
We have collected a report of every resident in our community that has seen a ghost. Each resident was asked "how many ghosts have you seen?", "describe the last ghost you saw", and "tell us about your mother." Afterwards, we compared the ghost reports between the different individuals, and assessed whether or not they had actually seen these apparitions.
Let's try again with the term: "People who see ghosts"
"I don't believe \alert{people who see ghosts}", said Mannie, before spitting into the wind and riding his bike down the street at top speed. He then went home and ate peanut-butter and jelly sandwiches all day. Mannie really liked peanut-butter and jelly sandwiches. He ate them so much that his poor mother had to purchase a new jar of peanut butter every afternoon.
We have collected a report of every \alert{resident} in our community that has \alert{seen} a \alert{ghost}. Each \alert{resident} was asked "how many \alert{ghosts} have you \alert{seen}?", "describe the last \alert{ghost} you \alert{saw}", and "tell us about your mother." Afterwards, we compared the ghost reports between the different \alert{individuals}, and assessed whether or not they had actually \alert{seen} these \alert{apparitions}.
\textbf{Stop words} "seen a ghost" $\rightarrow$ "seen ghost"
\textbf{Stemming} "seen a ghost" $\rightarrow$ "see ghost"
\textbf{Lemmatization} "saw ghosts" $\rightarrow$ "see ghost"
\textbf{Tokenization} "see ghost" $\rightarrow$ ["see", "ghost"]
We might need some concept of synonyms.
Are these actually synonyms?
Now let's try our tools on the following text:
People see incredible things. One time I saw some people talking about things they had seen, and those people were so much fun. They saw clouds and they saw airplanes. Can you believe the amount of seeing done by these people? People are the best.
Let $df_v$ be the number of documents that contain the term $v$.
The inverse document frequency is
$$ \textrm{idf}_v = \log\left( \frac{D}{df_v} \right), $$
where $D$ is the number of documents.
Properties:
For words which are more common, we lower their weights.
(example)
Words which appear in \textit{many} of the documents are not going to help us pick \textit{one} document.
What is Natural Language Processing?
Two large challenges of Natural Language Processing:
How can we differentiate between these documents?
\url{https://github.com/nandanrao/text-mining/blob/master/Dependency-Tree%20Example.ipynb}
title: Text Mining for Social Sciences author: Nandan Rao date: April, 2019 ...
This is hot stuff.
Some examples:
\includegraphics[width=\textwidth]{assets/missingmigrants}
\includegraphics[width=\textwidth]{assets/newsfilter}
What is the effect of the explosion of freelancing websites on the labor market? Demand side:
"I need an experienced Business Strategist who can write content explaining all the important moving parts and pieces of building a business plan and/or business model. You'll be explaining to first time entrepreneurs and small business owners and diving into the importance"
"We are Ricardo Steak House Restaurant located in Harlem, New York. We are looking for an expert opinion and training on how to manage our accounting department"
"We are an 8Mil per year trucking company based out of NJ. Due to negative loss-runs, we lost ideal market coverage for insurance and forced to use Progressive Commercial. We need someone with both an accounting background and deep knowledge of commercial insurance..."
How can we measure the growing share of artificial intelligence in worldwide innovation?
Corpora: Github, Patents.
One-class classification
Keep classification fixed, improve embedding.
How can we use online labor markets as a realtime view on shifts in jobs and skills demanded?
How do the returns to tasks change over time?
What are tasks? Created manually? Extract from text?
Content type
Image
Digest
Size
2.4 GB
Last updated
over 8 years ago
docker pull nandanrao/text-mining