twitter merge requestshttps://gitlab.memri.io/memri/plugins/twitter/-/merge_requests2022-12-16T09:20:31Zhttps://gitlab.memri.io/memri/plugins/twitter/-/merge_requests/32Draft: Twitter v22022-12-16T09:20:31ZAziz Berkay YesilyurtDraft: Twitter v2There are 1000 lines added 500 lines removed, but reviewing this MR hopefully still be straighforward.
I split the plugin into 4 main components. In the order of data flow: TweepyProducer, Plugin, PodQueue and ItemConsumer.
In summary,...There are 1000 lines added 500 lines removed, but reviewing this MR hopefully still be straighforward.
I split the plugin into 4 main components. In the order of data flow: TweepyProducer, Plugin, PodQueue and ItemConsumer.
In summary, this is what they do:
- TweepyProducer spits out tweets, accounts and related media
- Plugin creates PodUpdate items out of the above mentioned objects and places them on the queue (PodQueue)
- A PodUpdate consists of `create_items`, `create_edges` and `update_items`, just like before.
- ItemConsumer takes 50 or less PodUpdates at a time and puts them into pod.
- PodQueue is a priority queue and it prioritizes Tweets over everything else, then Accounts, then the rest in FIFO manner.
Remaining tasks:
- [ ] Port CI tests to v2
- [ ] Sending progress updates
- [ ] Update README with how to do authentication for development
- [ ] Integrate with FE (implement Oauth2 flow, see `cli/main.py`)Aziz Berkay YesilyurtAziz Berkay Yesilyurthttps://gitlab.memri.io/memri/plugins/twitter/-/merge_requests/26Draft: update docker image2022-11-25T15:20:36ZAziz Berkay YesilyurtDraft: update docker imageImprovements to docker container size.
From 1.1GB to 0.35GB.Improvements to docker container size.
From 1.1GB to 0.35GB.Aziz Berkay YesilyurtAziz Berkay Yesilyurthttps://gitlab.memri.io/memri/plugins/twitter/-/merge_requests/24Draft: Init new flow2022-11-25T11:25:45ZAlp Deniz OgutDraft: Init new flow- Redesigns the import sequence:
- Starts from Tweets. For each tweet:
- Author user is imported
- Author is added as following or follower to "me" account
- Mentioned users are imported
TODOs:
- Adapt progress. If ...- Redesigns the import sequence:
- Starts from Tweets. For each tweet:
- Author user is imported
- Author is added as following or follower to "me" account
- Mentioned users are imported
TODOs:
- Adapt progress. If possible, find out number of tweets at first.
- Handle "suspended user" import. (api.get_user on uspended users raises exception)
Question:
- Is it worth modifying the base Account item to include followers and following?Alp Deniz OgutAlp Deniz Ogut