--- title: Twitter importer plugin keywords: fastai sidebar: home_sidebar nb_path: "nbs/index.ipynb" ---
{% raw %}
{% endraw %}

About

This plugin is designed on top of the Twitter API. The twitter API uses OAuth 1.0a and OAuth 2.0 Bearer Token. In this plugin, we are using OAuth 1.0a which is documented here. The plugin is accessing the twitter API using tweepy an easy-to-use Python library for accessing the Twitter API

Data imported

In this plugin, the data fetched for a given user handle includes:

  • User
  • Contacts: Following and Followers
  • User timeline including tweets of Following the user’s own tweets
  • Tweets include Text Media: Photos, Videos (url) Links

Data structure of a user, a contact(follower or friend) and a tweet

    user = {
    created_at: string,
    id: integer,
    id_str: string,
    name: string,
    screen_name: string,
    location: string,
    profile_location: string,
    description: string,
    url: string,
    entities: dictionary,
    protected: boolean,
    followers_count: integer,
    friends_count: integer,
    listed_count: dictionary,
    created_at: string,
    favourites_count: integer,
    utc_offset: string,
    time_zone: string,
    geo_enabled: boolean,
    verified: boolean,
    statuses_count: integer,
    lang: string,
    status: dictionary,
    contributors_enabled":false,
    is_translator: boolean,
    is_translation_enabled: boolean,
    profile_background_color: string,
    profile_background_image_url: string,
    profile_background_image_url_https: string,
    profile_background_tile: boolean,
    profile_image_url: string,
    profile_image_url_https: string",
    profile_link_color: string,
    profile_sidebar_border_color: string,
    profile_sidebar_fill_color: string,
    profile_text_color: string,
    profile_use_background_image: string,
    has_extended_profile: boolean,
    default_profile: boolean,
    default_profile_image": boolean,
    following": boolean,
    follow_request_sent: boolean,
    notifications: boolean,
    translator_type: string,
    withheld_in_countries: list
   }

    followings = {
       "id": integer,
       "id_str": string,
       "name": string,
       "screen_name": string,
       "location": string,
       "description": string,
       "url": string,
       "entities": string,
       "protected": boolean,
       "followers_count": integer,
       "friends_count": integer,
       "listed_count": integer,
       "created_at": string,
       "favourites_count": integer,
       "utc_offset": string,
       "time_zone": string,
       "geo_enabled": boolean,
       "verified": boolean,
       "statuses_count": integer,
       "lang": string,
       "status": string,
       "contributors_enabled": boolean,
       "is_translator": boolean,
       "is_translation_enabled": boolean,
       "profile_background_color": string,
       "profile_background_image_url": string,
       "profile_background_image_url_https": string,
       "profile_background_tile": boolean,
       "profile_image_url": string,
       "profile_image_url_https": string,
       "profile_banner_url": string,
       "profile_link_color": string,
       "profile_sidebar_border_color": string,
       "profile_sidebar_fill_color": string,
       "profile_text_color": string,
       "profile_use_background_image": boolean,
       "has_extended_profile": boolean,
       "default_profile": boolean,
       "default_profile_image": boolean,
       "following": boolean,
       "live_following": boolean,
       "follow_request_sent": boolean,
       "notifications": boolean,
       "muting": boolean,
       "blocking": boolean,
       "blocked_by": boolean,
       "translator_type": string,
       "withheld_in_countries":list
       }

    tweet = {
        created_at: string,
        id: integer,
        id_str: string,
        full_text: string,
        truncated: boolean,
        display_text_range: list,
        entities: dictionary,
        source: string,
        in_reply_to_status_id: string,
        in_reply_to_status_id_str: string,
        in_reply_to_user_id: string,
        in_reply_to_user_id_str: string,
        in_reply_to_screen_name: string,
        user: dictionary,
        geo: string,
        coordinates: string,
        place: string,
        contributors: string,
        retweeted_status: dictionary,
        is_quote_status: string,
        retweet_count:,
        favorite_count:,
        favorited,
        retweeted
       }

Unsupported functionality

  • The plugin does not fetch user contact information such as phone numbers and emails.
  • The plugin does not fetch user direct messages.
  • The plugin does not support tweeting, commenting, retweeting on the speficied twitter handle account.

Limitations

Depending on your subscription category for the twitter developer account, you will have limitations to how many tweets you can fetch per month. Read more about it here. Ensure to log into your developer account and know how much you can fetch.

For each call made to the Twitter API, the maximum number of tweets that can be fetched is 200 items. However in this plugin, you can specify if you wou;d like to fetch less than 200 items in the run function by simply speficy how many items you want to fetch. This will apply on followers, followings and timeline items.

How to run project

  • First, you need to create an account with twitter developer here
  • Clone this repository
  • Download Anaconda. This project is using anaconda 3.5.
  • Install the version of Anaconda which you downloaded, following the instructions on the download page.
  • Run
pip install -e . && nbdev_install_git_hooks

To run the notebook. cd into the project directory and run

jupyter notebook

Install with pip

To install the Python package, and correctly setup nbdev for development run:

Run git init to initialize git hooks in your project folder then run

pip install -e . && nbdev_install_git_hooks

Install with Docker

The importer can be invoked by the Pod by launching a Docker container. To build the image for this container, run:

docker build -t twitter_importer .

To run the importer plugin

Once you have a developer account, get your api credentials listed below

  • ACCESS_TOKEN
  • ACCESS_SECRET
  • CONSUMER_KEY
  • CONSUMER_SECRET

Note that these maybe named differently depending on how recent your account is

Use this to guide you on the naming

Client credentials:
App Key === API Key === Consumer API Key === Consumer Key === Customer Key === oauth_consumer_key
App Key Secret === API Secret Key === Consumer Secret === Consumer Key === Customer Key === oauth_consumer_secret
Callback URL === oauth_callback

Token credentials:
Access token === Token === resulting oauth_token
Access token secret === Token Secret === resulting oauth_token_secret

Open TwitterImporter.ipynb and run the cells. Ensure that you have a stable internet connection for best results

Tools

Nbdev & Jupyter Notebooks

The Python integrators are written in nbdev (video). With nbdev, it is encouraged to write code in Jupyter Notebooks. Nbdev syncs all the notebooks in /nbs with the python code in /integrators. Tests are written side by side with the code in the notebooks, and documentation is automatically generated from the code and markdown in the notebooks and exported into the /docs folder. Check out the nbdev quickstart for an introduction, watch the video linked above, or see the nbdev documentation for a all functionalities and tutorials.

Tweepy

The plugin is accessing the twitter API using tweepy an easy-to-use Python library for accessing the Twitter API

To contribute

Simply reach out to memri for any requests or recommendations