Week 6: Getting Data, Part 1: Web Scraping

Content for lectures 6A and 6B

  View materials:   MUSA-550-Fall-2023/week-6

  HTML slides:   Lecture 6A   Lecture 6B

  Executable slides:   Lecture 6A   Lecture 6B

Recommended readings

• BeautifulSoup and CSS Selectors
• Web scraping: a promising tool for geographic data acquisition
• Web scraping in the news
  • OpenAI sued by authors (AP News)c
  • Scraping the Web Is a Powerful Tool. Clearview AI Abused It (Wired)
  • Web scraping is legal, US appeals court reaffirms (Techcrunch)

New Packages

• Requests
• Beautiful Soup
• Selenium

Useful Links and Reference Materials

• Insights into Housing Markets from Scraping Craigslist
  • and accompanying academic article
• HTML tutorial
  • sections: introduction, basics, elements, attributes
• Tutorial on Web Scraping with Requests and BeautifulSoup
• CSS Selectors