There are 1000 lines added 500 lines removed, but reviewing this MR hopefully still be straighforward.
I split the plugin into 4 main components. In the order of data flow: TweepyProducer, Plugin, PodQueue and ItemConsumer.
In summary, this is what they do:
- TweepyProducer spits out tweets, accounts and related media
- Plugin creates PodUpdate items out of the above mentioned objects and places them on the queue (PodQueue)
- A PodUpdate consists of
create_items
,create_edges
andupdate_items
, just like before. - ItemConsumer takes 50 or less PodUpdates at a time and puts them into pod.
- PodQueue is a priority queue and it prioritizes Tweets over everything else, then Accounts, then the rest in FIFO manner.
Remaining tasks:
-
Port CI tests to v2 -
Sending progress updates -
Update README with how to do authentication for development -
Integrate with FE (implement Oauth2 flow, see cli/main.py
)