--- title: Twitter importer plugin keywords: fastai sidebar: home_sidebar nb_path: "nbs/index.ipynb" ---
This plugin is designed on top of the Twitter API. The twitter API uses OAuth 1.0a and OAuth 2.0 Bearer Token. In this plugin, we are using OAuth 1.0a which is documented here. The plugin is accessing the twitter API using tweepy an easy-to-use Python library for accessing the Twitter API
In this plugin, the data fetched for a given user handle includes:
Data structure of a user, a contact(follower or friend) and a tweet
user = {
created_at: string,
id: integer,
id_str: string,
name: string,
screen_name: string,
location: string,
profile_location: string,
description: string,
url: string,
entities: dictionary,
protected: boolean,
followers_count: integer,
friends_count: integer,
listed_count: dictionary,
created_at: string,
favourites_count: integer,
utc_offset: string,
time_zone: string,
geo_enabled: boolean,
verified: boolean,
statuses_count: integer,
lang: string,
status: dictionary,
contributors_enabled":false,
is_translator: boolean,
is_translation_enabled: boolean,
profile_background_color: string,
profile_background_image_url: string,
profile_background_image_url_https: string,
profile_background_tile: boolean,
profile_image_url: string,
profile_image_url_https: string",
profile_link_color: string,
profile_sidebar_border_color: string,
profile_sidebar_fill_color: string,
profile_text_color: string,
profile_use_background_image: string,
has_extended_profile: boolean,
default_profile: boolean,
default_profile_image": boolean,
following": boolean,
follow_request_sent: boolean,
notifications: boolean,
translator_type: string,
withheld_in_countries: list
}
followings = {
"id": integer,
"id_str": string,
"name": string,
"screen_name": string,
"location": string,
"description": string,
"url": string,
"entities": string,
"protected": boolean,
"followers_count": integer,
"friends_count": integer,
"listed_count": integer,
"created_at": string,
"favourites_count": integer,
"utc_offset": string,
"time_zone": string,
"geo_enabled": boolean,
"verified": boolean,
"statuses_count": integer,
"lang": string,
"status": string,
"contributors_enabled": boolean,
"is_translator": boolean,
"is_translation_enabled": boolean,
"profile_background_color": string,
"profile_background_image_url": string,
"profile_background_image_url_https": string,
"profile_background_tile": boolean,
"profile_image_url": string,
"profile_image_url_https": string,
"profile_banner_url": string,
"profile_link_color": string,
"profile_sidebar_border_color": string,
"profile_sidebar_fill_color": string,
"profile_text_color": string,
"profile_use_background_image": boolean,
"has_extended_profile": boolean,
"default_profile": boolean,
"default_profile_image": boolean,
"following": boolean,
"live_following": boolean,
"follow_request_sent": boolean,
"notifications": boolean,
"muting": boolean,
"blocking": boolean,
"blocked_by": boolean,
"translator_type": string,
"withheld_in_countries":list
}
tweet = {
created_at: string,
id: integer,
id_str: string,
full_text: string,
truncated: boolean,
display_text_range: list,
entities: dictionary,
source: string,
in_reply_to_status_id: string,
in_reply_to_status_id_str: string,
in_reply_to_user_id: string,
in_reply_to_user_id_str: string,
in_reply_to_screen_name: string,
user: dictionary,
geo: string,
coordinates: string,
place: string,
contributors: string,
retweeted_status: dictionary,
is_quote_status: string,
retweet_count:,
favorite_count:,
favorited,
retweeted
}
Depending on your subscription category for the twitter developer account, you will have limitations to how many tweets you can fetch per month. Read more about it here. Ensure to log into your developer account and know how much you can fetch.
For each call made to the Twitter API, the maximum number of tweets that can be fetched is 200 items. However in this plugin, you can specify if you wou;d like to fetch less than 200 items in the run function by simply speficy how many items you want to fetch. This will apply on followers, followings and timeline items.
pip install -e . && nbdev_install_git_hooks
To run the notebook. cd into the project directory and run
jupyter notebook
To install the Python package, and correctly setup nbdev for development run:
Run git init to initialize git hooks in your project folder then run
pip install -e . && nbdev_install_git_hooks
The importer can be invoked by the Pod by launching a Docker container. To build the image for this container, run:
docker build -t twitter_importer .
Once you have a developer account, get your api credentials listed below
Use this to guide you on the naming
App Key === API Key === Consumer API Key === Consumer Key === Customer Key === oauth_consumer_key
App Key Secret === API Secret Key === Consumer Secret === Consumer Key === Customer Key === oauth_consumer_secret
Callback URL === oauth_callback
Access token === Token === resulting oauth_token
Access token secret === Token Secret === resulting oauth_token_secret
Open TwitterImporter.ipynb and run the cells. Ensure that you have a stable internet connection for best results
The Python integrators are written in nbdev (video). With nbdev, it is encouraged to write code in Jupyter Notebooks. Nbdev syncs all the notebooks in /nbs with the python code in /integrators. Tests are written side by side with the code in the notebooks, and documentation is automatically generated from the code and markdown in the notebooks and exported into the /docs folder. Check out the nbdev quickstart for an introduction, watch the video linked above, or see the nbdev documentation for a all functionalities and tutorials.
Simply reach out to memri for any requests or recommendations