Some optimizations to speed up importing messages. Speed up is 2x, from 400 sec to 200 sec to import 10k messages, on local pod.
General idea is to:
- Use
set
instead oflist
when we check for existance of items.- This requires that
Item
is hashable, which is not. So I patched schema items with__hash__
function and usedexternal_id
s as hashes, which should be unique anyway.
- This requires that
- Use
lru_cache
when querying item by id, so we don't have to wait for pod to respond.