Twitter API and Data Structure
The Twitter API allows developers to access Twitter data (such as Tweets and User data) that otherwise cannot be collected through the user interface. We use the Twitter API in the lab to collect historic data (via the REST API) and live tweets (via the Stream API). There are several ways to access the Twitter API.
How to collect Twitter data?
rtweet is one of the best packages to use for collecting Twitter data. The basic presentation file is here that explains useful functions of rtweet package: https://mkearney.github.io/nicar_tworkshop/#27
This video includes steps involved with the collection of twitter data (Dre Dyson, one of the student in this lab, gave a talk). https://usf.box.com/s/5ynyrgn84dlb8ayh7jx3f2snidx0kbxq
Scripts
Use rtweet R package to collect use timeline data (this script was used to collect user timeline data): https://usf.box.com/s/j4kx9whobkb6hqvnbtjfwxklp0o4kitm
The collected Twitter data is messy. This script shows how to filter-in selected fields only to a csv format from the original json file: https://usf.box.com/s/gleow46ey2rm1g0arpw2s8zrdmi07xi1
Creating a Developer Account
- Steps
Accessing the API
- R
- Python
- Postman
REST vs Stream
- REST
- Stream
JSON Objects and Attributes
The Twitter API returns data in JSON format. The key-value JSON pairs are grouped together into objects. The Twitter API has five object models (Tweet, User, Geo, Entities, and Extended Entities). Each object has a series of attributes that can be collected via the Twitter API.
https://developer.twitter.com/en/docs/twitter-api/v1/data-dictionary/overview
https://developer.twitter.com/en/docs/twitter-api/v1/data-dictionary/object-model/tweet
Some useful explanation on Twitter data (Created by Robert and Lilith)
https://github.com/Moncrief50/Updated_Hashtag_Frequency/blob/main/Twitter_JSON_Structure.txt
https://github.com/Moncrief50/Updated_Hashtag_Frequency/blob/main/Twitter_Column_Description.txt
Related
External Links
- Useful links on the internet related to this topic.
Comments (0)
You don't have permission to comment on this page.