Skip to content
GitLab
Commit 984dc0cf authored by Aziz Berkay Yesilyurt's avatar Aziz Berkay Yesilyurt
Browse files

rename to gmail_imap

parent 5a3ff4df
Pipeline #10003 passed with stage
in 1 minute and 14 seconds
Hide whitespace changes
Inline Side-by-side
Showing
with 56 additions and 56 deletions
+56 -56
Dockerfile
+ 1
- 1
View file @ 984dc0cf
......@@ -13,7 +13,7 @@ ENV PYTHONIOENCODING=UTF-8
COPY ./setup.py ./setup.py
COPY ./README.md ./README.md
COPY ./gmail_importer ./gmail_importer
COPY ./gmail_imap ./gmail_imap
COPY ./nbs ./nbs
COPY ./cvu ./cvu
COPY ./settings.ini ./settings.ini
......
Makefile
+ 3
- 3
View file @ 984dc0cf
......@@ -2,11 +2,11 @@
SHELL := /bin/bash
SRC = $(wildcard nbs/*.ipynb)
all: gmail_importer docs
all: gmail_imap docs
gmail_importer: $(SRC)
gmail_imap: $(SRC)
nbdev_build_lib
touch gmail_importer
touch gmail_imap
sync:
nbdev_update_lib
......
README.md
+ 3
- 3
View file @ 984dc0cf
......@@ -33,7 +33,7 @@ located at `$HOME/.pymemri/plugins/instagram/credentials.json`.
Now, simulate entering the password using
```bash
simulate_enter_credentials --plugin="gmail_importer"
simulate_enter_credentials --plugin="gmail_imap"
```
After you run the plugin, following items will be saved to the pod:
......@@ -52,7 +52,7 @@ After you run the plugin, following items will be saved to the pod:
First, build the image
```
docker build -t gmail_importer .
docker build -t gmail_imap .
```
Create a new credentials for the pod using
......@@ -70,5 +70,5 @@ simulate_run_plugin_from_frontend --metadata "metadata.json"
Now, simulate entering the password using
```bash
simulate_enter_credentials --plugin="gmail_importer"
simulate_enter_credentials --plugin="gmail_imap"
```
docs/_config.yml
+ 1
- 1
View file @ 984dc0cf
baseurl: /gmail_importer/
baseurl: /gmail_imap/
collections:
tooltips:
output: false
......
docs/authenticator.html
+ 1
- 1
View file @ 984dc0cf
......@@ -40,7 +40,7 @@ nb_path: "nbs/authenticator.ipynb"
<div class="output_markdown rendered_html output_subarea ">
<h4 id="get_cvu" class="doc_header"><code>get_cvu</code><a href="https://gitlab.memri.io/memri/gmail_importer/tree/prod/gmail_importer/authenticator.py#L23" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>get_cvu</code>(<strong><code>name</code></strong>, <strong><code>base_path</code></strong>=<em><code>Path('/home/eelco/projects/plugins/gmail/cvu')</code></em>)</p>
<h4 id="get_cvu" class="doc_header"><code>get_cvu</code><a href="https://gitlab.memri.io/memri/gmail_imap/tree/prod/gmail_imap/authenticator.py#L23" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>get_cvu</code>(<strong><code>name</code></strong>, <strong><code>base_path</code></strong>=<em><code>Path('/home/eelco/projects/plugins/gmail/cvu')</code></em>)</p>
</blockquote>
</div>
......
docs/gmailimporter.html
+ 9
- 9
View file @ 984dc0cf
......@@ -75,7 +75,7 @@ nb_path: "nbs/gmailimporter.ipynb"
<div class="output_markdown rendered_html output_subarea ">
<h2 id="IMAPClient" class="doc_header"><code>class</code> <code>IMAPClient</code><a href="https://gitlab.memri.io/memri/gmail_importer/tree/prod/gmail_importer/imap_client.py#L21" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>IMAPClient</code>(<strong><code>username</code></strong>, <strong><code>app_pw</code></strong>, <strong><code>host</code></strong>=<em><code>'imap.gmail.com'</code></em>, <strong><code>port</code></strong>=<em><code>993</code></em>)</p>
<h2 id="IMAPClient" class="doc_header"><code>class</code> <code>IMAPClient</code><a href="https://gitlab.memri.io/memri/gmail_imap/tree/prod/gmail_imap/imap_client.py#L21" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>IMAPClient</code>(<strong><code>username</code></strong>, <strong><code>app_pw</code></strong>, <strong><code>host</code></strong>=<em><code>'imap.gmail.com'</code></em>, <strong><code>port</code></strong>=<em><code>993</code></em>)</p>
</blockquote>
</div>
......@@ -99,7 +99,7 @@ nb_path: "nbs/gmailimporter.ipynb"
<div class="output_markdown rendered_html output_subarea ">
<h4 id="IMAPClient.list_mailboxes" class="doc_header"><code>IMAPClient.list_mailboxes</code><a href="https://gitlab.memri.io/memri/gmail_importer/tree/prod/gmail_importer/imap_client.py#L37" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>IMAPClient.list_mailboxes</code>()</p>
<h4 id="IMAPClient.list_mailboxes" class="doc_header"><code>IMAPClient.list_mailboxes</code><a href="https://gitlab.memri.io/memri/gmail_imap/tree/prod/gmail_imap/imap_client.py#L37" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>IMAPClient.list_mailboxes</code>()</p>
</blockquote>
<p>Lists all available mailboxes</p>
......@@ -124,7 +124,7 @@ nb_path: "nbs/gmailimporter.ipynb"
<div class="output_markdown rendered_html output_subarea ">
<h4 id="IMAPClient.get_all_mail_ids" class="doc_header"><code>IMAPClient.get_all_mail_ids</code><a href="https://gitlab.memri.io/memri/gmail_importer/tree/prod/gmail_importer/imap_client.py#L68" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>IMAPClient.get_all_mail_ids</code>()</p>
<h4 id="IMAPClient.get_all_mail_ids" class="doc_header"><code>IMAPClient.get_all_mail_ids</code><a href="https://gitlab.memri.io/memri/gmail_imap/tree/prod/gmail_imap/imap_client.py#L68" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>IMAPClient.get_all_mail_ids</code>()</p>
</blockquote>
<p>retrieves all mail ids from the selected mailbox</p>
......@@ -149,7 +149,7 @@ nb_path: "nbs/gmailimporter.ipynb"
<div class="output_markdown rendered_html output_subarea ">
<h4 id="IMAPClient.get_mail" class="doc_header"><code>IMAPClient.get_mail</code><a href="https://gitlab.memri.io/memri/gmail_importer/tree/prod/gmail_importer/imap_client.py#L76" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>IMAPClient.get_mail</code>(<strong><code>id</code></strong>)</p>
<h4 id="IMAPClient.get_mail" class="doc_header"><code>IMAPClient.get_mail</code><a href="https://gitlab.memri.io/memri/gmail_imap/tree/prod/gmail_imap/imap_client.py#L76" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>IMAPClient.get_mail</code>(<strong><code>id</code></strong>)</p>
</blockquote>
<p>Fetches a mail given a id, returns (raw_mail, thread_id)</p>
......@@ -180,7 +180,7 @@ nb_path: "nbs/gmailimporter.ipynb"
<div class="output_markdown rendered_html output_subarea ">
<h2 id="GmailImporter" class="doc_header"><code>class</code> <code>GmailImporter</code><a href="https://gitlab.memri.io/memri/gmail_importer/tree/prod/gmail_importer/plugin.py#L80" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>GmailImporter</code>(<strong><code>client</code></strong>=<em><code>None</code></em>, <strong><code>pluginRun</code></strong>=<em><code>None</code></em>, <strong>*<code>args</code></strong>, <strong>**<code>kwargs</code></strong>) :: <code>PluginBase</code></p>
<h2 id="GmailImporter" class="doc_header"><code>class</code> <code>GmailImporter</code><a href="https://gitlab.memri.io/memri/gmail_imap/tree/prod/gmail_imap/plugin.py#L80" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>GmailImporter</code>(<strong><code>client</code></strong>=<em><code>None</code></em>, <strong><code>pluginRun</code></strong>=<em><code>None</code></em>, <strong>*<code>args</code></strong>, <strong>**<code>kwargs</code></strong>) :: <code>PluginBase</code></p>
</blockquote>
<p>Base class for plugins</p>
......@@ -218,7 +218,7 @@ nb_path: "nbs/gmailimporter.ipynb"
<div class="output_markdown rendered_html output_subarea ">
<h4 id="GmailImporter.get_content" class="doc_header"><code>GmailImporter.get_content</code><a href="https://gitlab.memri.io/memri/gmail_importer/tree/prod/gmail_importer/plugin.py#L125" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>GmailImporter.get_content</code>(<strong><code>message</code></strong>)</p>
<h4 id="GmailImporter.get_content" class="doc_header"><code>GmailImporter.get_content</code><a href="https://gitlab.memri.io/memri/gmail_imap/tree/prod/gmail_imap/plugin.py#L125" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>GmailImporter.get_content</code>(<strong><code>message</code></strong>)</p>
</blockquote>
<p>Extracts content from a python email message</p>
......@@ -243,7 +243,7 @@ nb_path: "nbs/gmailimporter.ipynb"
<div class="output_markdown rendered_html output_subarea ">
<h4 id="GmailImporter.create_item_from_mail" class="doc_header"><code>GmailImporter.create_item_from_mail</code><a href="https://gitlab.memri.io/memri/gmail_importer/tree/prod/gmail_importer/plugin.py#L149" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>GmailImporter.create_item_from_mail</code>(<strong><code>mail</code></strong>, <strong><code>thread_id</code></strong>=<em><code>None</code></em>)</p>
<h4 id="GmailImporter.create_item_from_mail" class="doc_header"><code>GmailImporter.create_item_from_mail</code><a href="https://gitlab.memri.io/memri/gmail_imap/tree/prod/gmail_imap/plugin.py#L149" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>GmailImporter.create_item_from_mail</code>(<strong><code>mail</code></strong>, <strong><code>thread_id</code></strong>=<em><code>None</code></em>)</p>
</blockquote>
<p>Creates a schema-item from an existing mail</p>
......@@ -268,7 +268,7 @@ nb_path: "nbs/gmailimporter.ipynb"
<div class="output_markdown rendered_html output_subarea ">
<h4 id="GmailImporter.run" class="doc_header"><code>GmailImporter.run</code><a href="https://gitlab.memri.io/memri/gmail_importer/tree/prod/gmail_importer/plugin.py#L200" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>GmailImporter.run</code>(<strong><code>client</code></strong>=<em><code>None</code></em>)</p>
<h4 id="GmailImporter.run" class="doc_header"><code>GmailImporter.run</code><a href="https://gitlab.memri.io/memri/gmail_imap/tree/prod/gmail_imap/plugin.py#L200" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>GmailImporter.run</code>(<strong><code>client</code></strong>=<em><code>None</code></em>)</p>
</blockquote>
<p>This is the main function of the Gmail importer. It runs the importer given information
provided in the plugin run.</p>
......@@ -353,7 +353,7 @@ reading owner_key from /home/eelco/.pymemri/pod_keys/keys.json
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">account</span> <span class="o">=</span> <span class="n">Account</span><span class="p">(</span><span class="n">service</span><span class="o">=</span><span class="s2">&quot;gmail_imap&quot;</span><span class="p">)</span>
<span class="n">run</span> <span class="o">=</span> <span class="n">PluginRun</span><span class="p">(</span><span class="s2">&quot;&quot;</span><span class="p">,</span> <span class="s2">&quot;gmail_importer.plugin&quot;</span><span class="p">,</span> <span class="s2">&quot;GmailImporter&quot;</span><span class="p">)</span>
<span class="n">run</span> <span class="o">=</span> <span class="n">PluginRun</span><span class="p">(</span><span class="s2">&quot;&quot;</span><span class="p">,</span> <span class="s2">&quot;gmail_imap.plugin&quot;</span><span class="p">,</span> <span class="s2">&quot;GmailImporter&quot;</span><span class="p">)</span>
<span class="n">run</span><span class="o">.</span><span class="n">add_edge</span><span class="p">(</span><span class="s2">&quot;account&quot;</span><span class="p">,</span> <span class="n">account</span><span class="p">)</span>
<span class="n">run</span><span class="o">.</span><span class="n">status</span> <span class="o">=</span> <span class="n">RUN_STARTED</span>
......
docs/imapclient.html
+ 1
- 1
View file @ 984dc0cf
......@@ -47,7 +47,7 @@ nb_path: "nbs/imapclient.ipynb"
<div class="output_markdown rendered_html output_subarea ">
<h2 id="IMAPClient" class="doc_header"><code>class</code> <code>IMAPClient</code><a href="https://gitlab.memri.io/memri/gmail_importer/tree/prod/gmail_importer/imap_client.py#L21" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>IMAPClient</code>(<strong><code>username</code></strong>, <strong><code>app_pw</code></strong>, <strong><code>host</code></strong>=<em><code>'imap.gmail.com'</code></em>, <strong><code>port</code></strong>=<em><code>993</code></em>)</p>
<h2 id="IMAPClient" class="doc_header"><code>class</code> <code>IMAPClient</code><a href="https://gitlab.memri.io/memri/gmail_imap/tree/prod/gmail_imap/imap_client.py#L21" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>IMAPClient</code>(<strong><code>username</code></strong>, <strong><code>app_pw</code></strong>, <strong><code>host</code></strong>=<em><code>'imap.gmail.com'</code></em>, <strong><code>port</code></strong>=<em><code>993</code></em>)</p>
</blockquote>
</div>
......
docs/importers.EmailImporter.html
+ 10
- 10
View file @ 984dc0cf
......@@ -52,7 +52,7 @@ nb_path: "nbs/importers.EmailImporter.ipynb"
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="ImapClient">ImapClient<a class="anchor-link" href="#ImapClient"> </a></h2><p>The <a href="/gmail_importer/importers.EmailImporter.html#EmailImporter"><code>EmailImporter</code></a> communicates with email providers over imap. We created a convenience class around pythons imaplib , called the <code>ImapClient</code> that lets you list your mailboxes, retriev your mails and get their content.</p>
<h2 id="ImapClient">ImapClient<a class="anchor-link" href="#ImapClient"> </a></h2><p>The <a href="/gmail_imap/importers.EmailImporter.html#EmailImporter"><code>EmailImporter</code></a> communicates with email providers over imap. We created a convenience class around pythons imaplib , called the <code>ImapClient</code> that lets you list your mailboxes, retriev your mails and get their content.</p>
</div>
</div>
......@@ -68,7 +68,7 @@ nb_path: "nbs/importers.EmailImporter.ipynb"
<div class="output_markdown rendered_html output_subarea ">
<h2 id="IMAPClient" class="doc_header"><code>class</code> <code>IMAPClient</code><a href="https://gitlab.memri.io/memri/gmail_importer/tree/prod/gmail_importer/gmail.py#L23" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>IMAPClient</code>(<strong><code>username</code></strong>, <strong><code>app_pw</code></strong>, <strong><code>host</code></strong>=<em><code>'imap.gmail.com'</code></em>, <strong><code>port</code></strong>=<em><code>993</code></em>, <strong><code>inbox</code></strong>=<em><code>'"[Gmail]/All Mail"'</code></em>)</p>
<h2 id="IMAPClient" class="doc_header"><code>class</code> <code>IMAPClient</code><a href="https://gitlab.memri.io/memri/gmail_imap/tree/prod/gmail_imap/gmail.py#L23" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>IMAPClient</code>(<strong><code>username</code></strong>, <strong><code>app_pw</code></strong>, <strong><code>host</code></strong>=<em><code>'imap.gmail.com'</code></em>, <strong><code>port</code></strong>=<em><code>993</code></em>, <strong><code>inbox</code></strong>=<em><code>'"[Gmail]/All Mail"'</code></em>)</p>
</blockquote>
</div>
......@@ -92,7 +92,7 @@ nb_path: "nbs/importers.EmailImporter.ipynb"
<div class="output_markdown rendered_html output_subarea ">
<h4 id="IMAPClient.list_mailboxes" class="doc_header"><code>IMAPClient.list_mailboxes</code><a href="https://gitlab.memri.io/memri/gmail_importer/tree/prod/gmail_importer/gmail.py#L31" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>IMAPClient.list_mailboxes</code>()</p>
<h4 id="IMAPClient.list_mailboxes" class="doc_header"><code>IMAPClient.list_mailboxes</code><a href="https://gitlab.memri.io/memri/gmail_imap/tree/prod/gmail_imap/gmail.py#L31" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>IMAPClient.list_mailboxes</code>()</p>
</blockquote>
<p>Lists all available mailboxes</p>
......@@ -117,7 +117,7 @@ nb_path: "nbs/importers.EmailImporter.ipynb"
<div class="output_markdown rendered_html output_subarea ">
<h4 id="IMAPClient.get_all_mail_ids" class="doc_header"><code>IMAPClient.get_all_mail_ids</code><a href="https://gitlab.memri.io/memri/gmail_importer/tree/prod/gmail_importer/gmail.py#L35" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>IMAPClient.get_all_mail_ids</code>()</p>
<h4 id="IMAPClient.get_all_mail_ids" class="doc_header"><code>IMAPClient.get_all_mail_ids</code><a href="https://gitlab.memri.io/memri/gmail_imap/tree/prod/gmail_imap/gmail.py#L35" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>IMAPClient.get_all_mail_ids</code>()</p>
</blockquote>
<p>retrieves all mail ids from the selected mailbox</p>
......@@ -142,7 +142,7 @@ nb_path: "nbs/importers.EmailImporter.ipynb"
<div class="output_markdown rendered_html output_subarea ">
<h4 id="IMAPClient.get_mail" class="doc_header"><code>IMAPClient.get_mail</code><a href="https://gitlab.memri.io/memri/gmail_importer/tree/prod/gmail_importer/gmail.py#L43" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>IMAPClient.get_mail</code>(<strong><code>id</code></strong>)</p>
<h4 id="IMAPClient.get_mail" class="doc_header"><code>IMAPClient.get_mail</code><a href="https://gitlab.memri.io/memri/gmail_imap/tree/prod/gmail_imap/gmail.py#L43" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>IMAPClient.get_mail</code>(<strong><code>id</code></strong>)</p>
</blockquote>
<p>Fetches a mail given a id, returns (raw_mail, thread_id)</p>
......@@ -173,7 +173,7 @@ nb_path: "nbs/importers.EmailImporter.ipynb"
<div class="output_markdown rendered_html output_subarea ">
<h2 id="ImporterBase" class="doc_header"><code>class</code> <code>ImporterBase</code><a href="https://gitlab.memri.io/memri/gmail_importer/tree/prod/gmail_importer/gmail.py#L116" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>ImporterBase</code>()</p>
<h2 id="ImporterBase" class="doc_header"><code>class</code> <code>ImporterBase</code><a href="https://gitlab.memri.io/memri/gmail_imap/tree/prod/gmail_imap/gmail.py#L116" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>ImporterBase</code>()</p>
</blockquote>
</div>
......@@ -197,7 +197,7 @@ nb_path: "nbs/importers.EmailImporter.ipynb"
<div class="output_markdown rendered_html output_subarea ">
<h2 id="EmailImporter" class="doc_header"><code>class</code> <code>EmailImporter</code><a href="https://gitlab.memri.io/memri/gmail_importer/tree/prod/gmail_importer/gmail.py#L118" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>EmailImporter</code>(<strong>*<code>args</code></strong>, <strong>**<code>kwargs</code></strong>) :: <a href="/gmail_importer/importers.EmailImporter.html#ImporterBase"><code>ImporterBase</code></a></p>
<h2 id="EmailImporter" class="doc_header"><code>class</code> <code>EmailImporter</code><a href="https://gitlab.memri.io/memri/gmail_imap/tree/prod/gmail_imap/gmail.py#L118" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>EmailImporter</code>(<strong>*<code>args</code></strong>, <strong>**<code>kwargs</code></strong>) :: <a href="/gmail_imap/importers.EmailImporter.html#ImporterBase"><code>ImporterBase</code></a></p>
</blockquote>
<p>Imports emails over imap.</p>
......@@ -244,7 +244,7 @@ nb_path: "nbs/importers.EmailImporter.ipynb"
<div class="output_markdown rendered_html output_subarea ">
<h4 id="EmailImporter.get_content" class="doc_header"><code>EmailImporter.get_content</code><a href="https://gitlab.memri.io/memri/gmail_importer/tree/prod/gmail_importer/gmail.py#L153" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>EmailImporter.get_content</code>(<strong><code>message</code></strong>)</p>
<h4 id="EmailImporter.get_content" class="doc_header"><code>EmailImporter.get_content</code><a href="https://gitlab.memri.io/memri/gmail_imap/tree/prod/gmail_imap/gmail.py#L153" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>EmailImporter.get_content</code>(<strong><code>message</code></strong>)</p>
</blockquote>
<p>Extracts content from a python email message</p>
......@@ -269,7 +269,7 @@ nb_path: "nbs/importers.EmailImporter.ipynb"
<div class="output_markdown rendered_html output_subarea ">
<h4 id="EmailImporter.create_item_from_mail" class="doc_header"><code>EmailImporter.create_item_from_mail</code><a href="https://gitlab.memri.io/memri/gmail_importer/tree/prod/gmail_importer/gmail.py#L176" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>EmailImporter.create_item_from_mail</code>(<strong><code>mail</code></strong>, <strong><code>thread_id</code></strong>=<em><code>None</code></em>)</p>
<h4 id="EmailImporter.create_item_from_mail" class="doc_header"><code>EmailImporter.create_item_from_mail</code><a href="https://gitlab.memri.io/memri/gmail_imap/tree/prod/gmail_imap/gmail.py#L176" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>EmailImporter.create_item_from_mail</code>(<strong><code>mail</code></strong>, <strong><code>thread_id</code></strong>=<em><code>None</code></em>)</p>
</blockquote>
<p>Creates a schema-item from an existing mail</p>
......@@ -294,7 +294,7 @@ nb_path: "nbs/importers.EmailImporter.ipynb"
<div class="output_markdown rendered_html output_subarea ">
<h4 id="EmailImporter.run" class="doc_header"><code>EmailImporter.run</code><a href="https://gitlab.memri.io/memri/gmail_importer/tree/prod/gmail_importer/gmail.py#L219" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>EmailImporter.run</code>(<strong><code>importer_run</code></strong>, <strong><code>pod_client</code></strong>=<em><code>None</code></em>, <strong><code>verbose</code></strong>=<em><code>True</code></em>)</p>
<h4 id="EmailImporter.run" class="doc_header"><code>EmailImporter.run</code><a href="https://gitlab.memri.io/memri/gmail_imap/tree/prod/gmail_imap/gmail.py#L219" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>EmailImporter.run</code>(<strong><code>importer_run</code></strong>, <strong><code>pod_client</code></strong>=<em><code>None</code></em>, <strong><code>verbose</code></strong>=<em><code>True</code></em>)</p>
</blockquote>
<p>This is the main function of the Email importer. It runs the importer given information
provided in the importer run. if you pass a pod client it will add the new items to the graph.</p>
......
docs/importers.GmailImporter.html
+ 9
- 9
View file @ 984dc0cf
......@@ -52,7 +52,7 @@ nb_path: "nbs/importers.GmailImporter.ipynb"
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="ImapClient">ImapClient<a class="anchor-link" href="#ImapClient"> </a></h2><p>The <a href="/gmail_importer/importers.GmailImporter.html#EmailImporter"><code>EmailImporter</code></a> communicates with email providers over imap. We created a convenience class around pythons imaplib , called the <code>ImapClient</code> that lets you list your mailboxes, retriev your mails and get their content.</p>
<h2 id="ImapClient">ImapClient<a class="anchor-link" href="#ImapClient"> </a></h2><p>The <a href="/gmail_imap/importers.GmailImporter.html#EmailImporter"><code>EmailImporter</code></a> communicates with email providers over imap. We created a convenience class around pythons imaplib , called the <code>ImapClient</code> that lets you list your mailboxes, retriev your mails and get their content.</p>
</div>
</div>
......@@ -68,7 +68,7 @@ nb_path: "nbs/importers.GmailImporter.ipynb"
<div class="output_markdown rendered_html output_subarea ">
<h2 id="IMAPClient" class="doc_header"><code>class</code> <code>IMAPClient</code><a href="https://gitlab.memri.io/memri/gmail_importer/tree/prod/gmail_importer/gmail.py#L22" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>IMAPClient</code>(<strong><code>username</code></strong>, <strong><code>app_pw</code></strong>, <strong><code>host</code></strong>=<em><code>'imap.gmail.com'</code></em>, <strong><code>port</code></strong>=<em><code>993</code></em>, <strong><code>inbox</code></strong>=<em><code>'"[Gmail]/All Mail"'</code></em>)</p>
<h2 id="IMAPClient" class="doc_header"><code>class</code> <code>IMAPClient</code><a href="https://gitlab.memri.io/memri/gmail_imap/tree/prod/gmail_imap/gmail.py#L22" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>IMAPClient</code>(<strong><code>username</code></strong>, <strong><code>app_pw</code></strong>, <strong><code>host</code></strong>=<em><code>'imap.gmail.com'</code></em>, <strong><code>port</code></strong>=<em><code>993</code></em>, <strong><code>inbox</code></strong>=<em><code>'"[Gmail]/All Mail"'</code></em>)</p>
</blockquote>
</div>
......@@ -92,7 +92,7 @@ nb_path: "nbs/importers.GmailImporter.ipynb"
<div class="output_markdown rendered_html output_subarea ">
<h4 id="IMAPClient.list_mailboxes" class="doc_header"><code>IMAPClient.list_mailboxes</code><a href="https://gitlab.memri.io/memri/gmail_importer/tree/prod/gmail_importer/gmail.py#L30" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>IMAPClient.list_mailboxes</code>()</p>
<h4 id="IMAPClient.list_mailboxes" class="doc_header"><code>IMAPClient.list_mailboxes</code><a href="https://gitlab.memri.io/memri/gmail_imap/tree/prod/gmail_imap/gmail.py#L30" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>IMAPClient.list_mailboxes</code>()</p>
</blockquote>
<p>Lists all available mailboxes</p>
......@@ -117,7 +117,7 @@ nb_path: "nbs/importers.GmailImporter.ipynb"
<div class="output_markdown rendered_html output_subarea ">
<h4 id="IMAPClient.get_all_mail_ids" class="doc_header"><code>IMAPClient.get_all_mail_ids</code><a href="https://gitlab.memri.io/memri/gmail_importer/tree/prod/gmail_importer/gmail.py#L34" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>IMAPClient.get_all_mail_ids</code>()</p>
<h4 id="IMAPClient.get_all_mail_ids" class="doc_header"><code>IMAPClient.get_all_mail_ids</code><a href="https://gitlab.memri.io/memri/gmail_imap/tree/prod/gmail_imap/gmail.py#L34" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>IMAPClient.get_all_mail_ids</code>()</p>
</blockquote>
<p>retrieves all mail ids from the selected mailbox</p>
......@@ -142,7 +142,7 @@ nb_path: "nbs/importers.GmailImporter.ipynb"
<div class="output_markdown rendered_html output_subarea ">
<h4 id="IMAPClient.get_mail" class="doc_header"><code>IMAPClient.get_mail</code><a href="https://gitlab.memri.io/memri/gmail_importer/tree/prod/gmail_importer/gmail.py#L42" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>IMAPClient.get_mail</code>(<strong><code>id</code></strong>)</p>
<h4 id="IMAPClient.get_mail" class="doc_header"><code>IMAPClient.get_mail</code><a href="https://gitlab.memri.io/memri/gmail_imap/tree/prod/gmail_imap/gmail.py#L42" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>IMAPClient.get_mail</code>(<strong><code>id</code></strong>)</p>
</blockquote>
<p>Fetches a mail given a id, returns (raw_mail, thread_id)</p>
......@@ -209,7 +209,7 @@ nb_path: "nbs/importers.GmailImporter.ipynb"
<div class="output_markdown rendered_html output_subarea ">
<h2 id="EmailImporter" class="doc_header"><code>class</code> <code>EmailImporter</code><a href="https://gitlab.memri.io/memri/gmail_importer/tree/prod/gmail_importer/gmail.py#L105" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>EmailImporter</code>(<strong>*<code>args</code></strong>, <strong>**<code>kwargs</code></strong>) :: <code>ImporterBase</code></p>
<h2 id="EmailImporter" class="doc_header"><code>class</code> <code>EmailImporter</code><a href="https://gitlab.memri.io/memri/gmail_imap/tree/prod/gmail_imap/gmail.py#L105" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>EmailImporter</code>(<strong>*<code>args</code></strong>, <strong>**<code>kwargs</code></strong>) :: <code>ImporterBase</code></p>
</blockquote>
<p>Imports emails over imap.</p>
......@@ -262,7 +262,7 @@ nb_path: "nbs/importers.GmailImporter.ipynb"
<div class="output_markdown rendered_html output_subarea ">
<h4 id="EmailImporter.get_content" class="doc_header"><code>EmailImporter.get_content</code><a href="https://gitlab.memri.io/memri/gmail_importer/tree/prod/gmail_importer/gmail.py#L134" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>EmailImporter.get_content</code>(<strong><code>message</code></strong>)</p>
<h4 id="EmailImporter.get_content" class="doc_header"><code>EmailImporter.get_content</code><a href="https://gitlab.memri.io/memri/gmail_imap/tree/prod/gmail_imap/gmail.py#L134" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>EmailImporter.get_content</code>(<strong><code>message</code></strong>)</p>
</blockquote>
<p>Extracts content from a python email message</p>
......@@ -287,7 +287,7 @@ nb_path: "nbs/importers.GmailImporter.ipynb"
<div class="output_markdown rendered_html output_subarea ">
<h4 id="EmailImporter.create_item_from_mail" class="doc_header"><code>EmailImporter.create_item_from_mail</code><a href="https://gitlab.memri.io/memri/gmail_importer/tree/prod/gmail_importer/gmail.py#L157" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>EmailImporter.create_item_from_mail</code>(<strong><code>mail</code></strong>, <strong><code>thread_id</code></strong>=<em><code>None</code></em>)</p>
<h4 id="EmailImporter.create_item_from_mail" class="doc_header"><code>EmailImporter.create_item_from_mail</code><a href="https://gitlab.memri.io/memri/gmail_imap/tree/prod/gmail_imap/gmail.py#L157" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>EmailImporter.create_item_from_mail</code>(<strong><code>mail</code></strong>, <strong><code>thread_id</code></strong>=<em><code>None</code></em>)</p>
</blockquote>
<p>Creates a schema-item from an existing mail</p>
......@@ -312,7 +312,7 @@ nb_path: "nbs/importers.GmailImporter.ipynb"
<div class="output_markdown rendered_html output_subarea ">
<h4 id="EmailImporter.run" class="doc_header"><code>EmailImporter.run</code><a href="https://gitlab.memri.io/memri/gmail_importer/tree/prod/gmail_importer/gmail.py#L211" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>EmailImporter.run</code>(<strong><code>importer_run</code></strong>=<em><code>None</code></em>, <strong><code>pod_client</code></strong>=<em><code>None</code></em>, <strong><code>verbose</code></strong>=<em><code>True</code></em>)</p>
<h4 id="EmailImporter.run" class="doc_header"><code>EmailImporter.run</code><a href="https://gitlab.memri.io/memri/gmail_imap/tree/prod/gmail_imap/gmail.py#L211" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>EmailImporter.run</code>(<strong><code>importer_run</code></strong>=<em><code>None</code></em>, <strong><code>pod_client</code></strong>=<em><code>None</code></em>, <strong><code>verbose</code></strong>=<em><code>True</code></em>)</p>
</blockquote>
<p>This is the main function of the Email importer. It runs the importer given information
provided in the importer run. if you pass a pod client it will add the new items to the graph.</p>
......
docs/index.html
+ 1
- 1
View file @ 984dc0cf
......@@ -65,7 +65,7 @@ nb_path: "nbs/index.ipynb"
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="Docker">Docker<a class="anchor-link" href="#Docker"> </a></h1>
<pre><code>docker build -t gmail_importer .</code></pre>
<pre><code>docker build -t gmail_imap .</code></pre>
<p>To run the plugin:</p>
<pre><code>simulate_run_plugin_from_frontend --config_file="example_config.json"</code></pre>
......
gmail_importer/__init__.py → gmail_imap/__init__.py
+ 0
- 0
View file @ 984dc0cf
File moved
gmail_importer/_nbdev.py → gmail_imap/_nbdev.py
+ 2
- 2
View file @ 984dc0cf
......@@ -25,8 +25,8 @@ modules = ["plugin.py",
"smtp_client.py",
"utils.py"]
doc_url = "http://memri.docs.memri.io/gmail_importer/"
doc_url = "http://memri.docs.memri.io/gmail_imap/"
git_url = "https://gitlab.memri.io/memri/gmail_importer/tree/prod/"
git_url = "https://gitlab.memri.io/memri/gmail_imap/tree/prod/"
def custom_doc_links(name): return None
gmail_importer/imap_client.py → gmail_imap/imap_client.py
+ 0
- 0
View file @ 984dc0cf
File moved
gmail_importer/plugin.py → gmail_imap/plugin.py
+ 1
- 1
View file @ 984dc0cf
......@@ -30,7 +30,7 @@ from .utils import part_to_str, _get_all_parts, get_username_password
from .schema import EmailMessage
from .smtp_client import GmailSMTPClient, send_mail_listener
import gmail_importer
import gmail_imap
from pymemri.plugin.pluginbase import run_plugin
# Cell
......
gmail_importer/schema.py → gmail_imap/schema.py
+ 0
- 0
View file @ 984dc0cf
File moved
gmail_importer/settings.py → gmail_imap/settings.py
+ 2
- 2
View file @ 984dc0cf
......@@ -4,11 +4,11 @@ __all__ = ['BASE_PATH', 'SERVICE_NAME', 'GMAIL_SMTP_HOST', 'GMAIL_SMTP_PORT', 'G
'SEEN_FLAG']
# Cell
import gmail_importer
import gmail_imap
import pathlib
# Cell
BASE_PATH = pathlib.Path(gmail_importer.__file__).parent
BASE_PATH = pathlib.Path(gmail_imap.__file__).parent
SERVICE_NAME = "gmail"
GMAIL_SMTP_HOST = 'smtp.gmail.com'
GMAIL_SMTP_PORT = 465
......
gmail_importer/smtp_client.py → gmail_imap/smtp_client.py
+ 0
- 0
View file @ 984dc0cf
File moved
gmail_importer/utils.py → gmail_imap/utils.py
+ 1
- 1
View file @ 984dc0cf
......@@ -8,7 +8,7 @@ from pymemri.data.basic import HOME_DIR, read_json
# Cell
def get_username_password():
credentials_path = HOME_DIR / ".pymemri" / "plugins" / "gmail_importer" / "credentials.json"
credentials_path = HOME_DIR / ".pymemri" / "plugins" / "gmail_imap" / "credentials.json"
credentials = read_json(credentials_path)
username, password = credentials["username"], credentials["password"]
return username,password
......
metadata.json
+ 3
- 3
View file @ 984dc0cf
{
"containerImage": "gmail_importer",
"pluginModule": "gmail_importer.plugin",
"plugin": "gmail_importer.plugin.GmailImporter",
"containerImage": "gmail_imap",
"pluginModule": "gmail_imap.plugin",
"plugin": "gmail_imap.plugin.GmailImporter",
"pluginName": "GmailImporter",
"status": "not started",
"account": {
......
nbs/gmailimporter.ipynb
+ 8
- 8
View file @ 984dc0cf
%% Cell type:code id: tags:
``` python
%load_ext autoreload
%autoreload 2
# default_exp plugin
```
%% Cell type:code id: tags:
``` python
# export
import json
import email
import math
from email import policy
from email.utils import getaddresses
from nbdev.showdoc import show_doc
import time
import pathlib
import quopri
import threading
from pymemri.pod.client import PodClient
from pymemri.data.basic import *
from pymemri.data.schema import MessageChannel, Person
from pymemri.plugin.schema import PluginRun, Account
from pymemri.plugin.pluginbase import PluginBase
from pymemri.plugin.authenticators.password import PasswordAuthenticator
from pymemri.plugin.states import RUN_COMPLETED, RUN_USER_ACTION_NEEDED, RUN_ABORTED
from pymemri.plugin.states import RUN_USER_ACTION_COMPLETED, RUN_STARTED, RUN_FAILED
from gmail_importer.imap_client import IMAPClient
from gmail_importer.settings import *
from gmail_importer.utils import part_to_str, _get_all_parts, get_username_password
from gmail_importer.schema import EmailMessage
from gmail_importer.smtp_client import GmailSMTPClient, send_mail_listener
import gmail_importer
from gmail_imap.imap_client import IMAPClient
from gmail_imap.settings import *
from gmail_imap.utils import part_to_str, _get_all_parts, get_username_password
from gmail_imap.schema import EmailMessage
from gmail_imap.smtp_client import GmailSMTPClient, send_mail_listener
import gmail_imap
from pymemri.plugin.pluginbase import run_plugin
```
%% Cell type:markdown id: tags:
# Email importer
%% Cell type:markdown id: tags:
This importers fetches your emails and accounts over IMAP, it uses the python built-in imap client and some convenience functions for easier usage, batching and importing to the pod. This importer requires you to login with your email address and an app password. It is tested on gmail, but should work for other IMAP-servers.
%% Cell type:markdown id: tags:
> Note: **The recommended usage for Gmail is to enable two-factor authentication. In this case, make sure you allow [SMTP-connections](https://www.gmass.co/blog/gmail-smtp/) and set an application password (explained in the same link)**
%% Cell type:markdown id: tags:
## EmailImporter
%% Cell type:code id: tags:
``` python
# export
SERVICE_NAME = "gmail"
class GmailImporter(PluginBase):
MAX_RUN_TIME = 43200
MAX_IMPORTS = 100000
MAX_LOGIN_ATTEMPTS = 10
BATCH_SIZE = 100
MAILS_PER_BULK = 500
"""Imports gmail emails over imap."""
def __init__(self, username=None, password=None, run_as_daemon=True, **kwargs):
super().__init__(**kwargs)
self.start_time = time.time()
self.imap_client = None
self.smtp_client = None
self.authenticator = None
self._daemon = run_as_daemon
self.username = username
self.password = password
# Dicts for available accounts/channels.
self.accounts = dict()
self.message_channels = dict()
self.inboxes = dict()
# Set with externalIds of accounts that have been synced this run.
self.synced_accounts = set()
self.synced_channels = set()
self.synced_imap_ids = set()
def login(self, username=None, password=None):
self.imap_client = IMAPClient(username=username, app_pw=password)
print("IMAP client logged in.", flush=True)
self.smtp_client = GmailSMTPClient(username=username, password=password)
print("SMTP client logged in.", flush=True)
def setup_send_mail_listener(self):
thread = threading.Thread(
target=send_mail_listener,
args=(self.client, self.smtp_client.send_mail),
daemon=True)
thread.start()
@staticmethod
def get_timestamp_from_message(message):
date = message["date"]
parsed_time = email.utils.parsedate(date)
dt = email.utils.parsedate_to_datetime(date)
return dt
def get_accounts(self, message, field):
addresses = getaddresses(message.get_all(field, []))
accounts = []
for name, address in addresses:
name = name if name != "" else None
if address in self.accounts:
account = self.accounts[address]
else:
account = Account(externalId=address, identifier=address, displayName=name, service=SERVICE_NAME)
self.accounts[address] = account
accounts.append(account)
return accounts
@staticmethod
def get_content(message, verbose=False):
"""Extracts content from a python email message"""
def fix_quopri(payload):
if payload.isascii() and '=3D"' in payload:
return quopri.decodestring(payload).decode('utf-8')
else:
return payload
maintype = message.get_content_maintype()
if maintype == 'multipart':
parts = _get_all_parts(message)
res = None
html_parts = [part_to_str(part) for part in parts if part.get_content_type() == "text/html"]
if len(html_parts) > 0:
if len(html_parts) > 1:
error_msg = "\n AND \n".join(html_parts)
print(f"WARNING: FOUND MULTIPLE HTML PARTS IN ONE MESSAGE")
if verbose:
print(error_msg)
return fix_quopri(html_parts[0])
else:
return fix_quopri(parts[0].get_payload())
elif maintype == 'text':
return fix_quopri(message.get_payload())
@staticmethod
def get_attachments(message):
return list(message.iter_attachments())
def create_item_from_mail(self, mail, flags, thread_id=None):
"""Creates a schema-item from an existing mail"""
message = email.message_from_bytes(mail, policy=policy.SMTP)
message_id, subject = message["message-id"], message["subject"]
timestamp = self.get_timestamp_from_message(message)
content = self.get_content(message)
attachments = self.get_attachments(message)
is_read = SEEN_FLAG in flags
email_item = EmailMessage(
externalId=message_id,
subject=subject,
dateSent=timestamp,
content=content,
read=is_read
)
for a in self.get_accounts(message, 'from'):
email_item.add_edge('sender', a)
for a in self.get_accounts(message, 'to'):
email_item.add_edge('receiver', a)
for a in self.get_accounts(message, 'reply-to'):
email_item.add_edge('replyTo', a)
for a in self.get_accounts(message, 'cc'):
email_item.add_edge('cc', a)
for a in self.get_accounts(message, 'bcc'):
email_item.add_edge('bcc', a)
if thread_id is not None:
if thread_id in self.message_channels:
thread_item = self.message_channels[thread_id]
else:
thread_item = MessageChannel(externalId=thread_id)
self.message_channels[thread_id] = thread_item
email_item.add_edge('messageChannel', thread_item)
return email_item
@staticmethod
def batch(iterable, n=1):
l = len(iterable)
for ndx in range(0, l, n):
yield iterable[ndx:min(ndx + n, l)]
def get_mails_batched(self, mail_ids, batch_size=100, max_content_length=4000000):
"""Gets mails from a list of mail ids. You can pass an importer run and podclient
to update the progress of the process"""
n_batches = math.ceil(len(mail_ids) / batch_size)
for i, batch_ids in enumerate(self.batch(mail_ids, n=batch_size)):
mails = []
print(f"downloading and converting batch {i}/{n_batches}")
for mail, thread_id, flags in self.imap_client.get_mails(batch_ids):
try:
item = self.create_item_from_mail(mail, flags, thread_id=thread_id)
if len(item.content) <= max_content_length:
mails.append(item)
except Exception as e:
print(f"Skipping message: {e}")
progress = (i + 1) / n_batches * 1.0
yield mails, batch_ids
def sync_to_pod(self, mails):
# Filter synced items
accounts_to_sync = [v for k, v in self.accounts.items() if k not in self.synced_accounts]
channels_to_sync = [v for k, v in self.message_channels.items() if k not in self.synced_channels]
# Bulk
items = mails + accounts_to_sync + channels_to_sync
edges = [e for email_item in mails for e in email_item.get_all_edges()]
self.client.bulk_action(create_items=items, create_edges=edges)
self.synced_accounts.update([a.externalId for a in accounts_to_sync])
self.synced_channels.update([c.externalId for c in channels_to_sync])
def run(self):
"""This is the main function of the Gmail importer. It runs the importer given information
provided in the plugin run."""
if self.username is None:
# request user credentials
print("Starting authentication...", flush=True)
self.authenticator = PasswordAuthenticator(client=self.client, pluginRun=self.pluginRun)
self.authenticator.authenticate(self.login)
self.pluginRun = self.client.get(self.pluginRun.id)
self.account = self.pluginRun.account[0]
else:
# login with credentials defined during init
self.login(self.username, self.password)
self.account = Account(
identifier=self.username, secret=self.password, service=SERVICE_NAME
)
self.client.create(self.account)
if self._daemon:
self.setup_send_mail_listener()
# TODO: This is not linked to the other accounts (from mails) yet
me_account = Account(isMe=True, identifier=self.imap_client.username, externalId=self.imap_client.username,
service=SERVICE_NAME)
self.client.create(me_account)
mail_ids = self.imap_client.get_all_mail_ids()
print(f"Found {len(mail_ids)} emails")
if self.MAX_IMPORTS:
mail_ids = mail_ids[:int(self.MAX_IMPORTS)]
print(f"Importing latest {len(mail_ids)} emails...")
# First import is a single batch, to make the plugin respond fast.
first_import = True
mails_to_sync = []
for batch, batch_ids in self.get_mails_batched(mail_ids, batch_size=self.BATCH_SIZE):
mails_to_sync.extend(batch)
# NOTE for incremental syncing: synced_imap_ids are not message ids (or externalIds).
self.synced_imap_ids.update(batch_ids)
if first_import or len(mails_to_sync) >= self.MAILS_PER_BULK:
first_import = False
self.sync_to_pod(mails_to_sync)
mails_to_sync = []
# Write remaining mails to pod
if len(mails_to_sync):
self.sync_to_pod(mails_to_sync)
if self._daemon:
print("Finished initial import, starting daemon...", flush=True)
self.run_daemon()
self.imap_client.client.logout()
print("Done.", flush=True)
def stop_daemon(self):
"""
Stop daemon when run time > self.MAX_RUN_TIME
or pluginRun.status == RUN_ABORTED
"""
if time.time() - self.start_time > self.MAX_RUN_TIME:
return True
self.pluginRun = self.client.get(self.pluginRun.id)
if self.pluginRun.status == RUN_ABORTED:
return True
return False
def run_daemon(self):
while not self.stop_daemon():
print(f"[DAEMON] Starting IDLE")
self.imap_client.idle()
mail_ids = self.imap_client.get_all_mail_ids()
new_mails_ids = [uid for uid in mail_ids if uid not in self.synced_imap_ids]
if self.MAX_IMPORTS:
new_mails_ids = new_mails_ids[:int(self.MAX_IMPORTS)]
if len(new_mails_ids):
print(f"[DAEMON] {len(new_mails_ids)} new emails found, syncing...")
else:
print(f"[DAEMON] No new mails found")
continue
mails_to_sync = []
for batch, batch_ids in self.get_mails_batched(new_mails_ids, batch_size=self.BATCH_SIZE):
mails_to_sync.extend(batch)
# NOTE for incremental syncing: synced_imap_ids are not message ids (or externalIds).
self.synced_imap_ids.update(batch_ids)
if len(mails_to_sync) >= self.MAILS_PER_BULK:
self.sync_to_pod(mails_to_sync)
mails_to_sync = []
if len(mails_to_sync):
self.sync_to_pod(mails_to_sync)
def add_to_schema(self):
self.client.add_to_schema(EmailMessage, MessageChannel, Person, Account)
```
%% Cell type:markdown id: tags:
## Usage
%% Cell type:markdown id: tags:
### Run in notebook
%% Cell type:code id: tags:
``` python
client = PodClient.from_local_keys()
username, password = get_username_password()
run = PluginRun("", "gmail_importer.plugin", "GmailImporter")
run = PluginRun("", "gmail_imap.plugin", "GmailImporter")
client.create(run)
gmail_plugin = GmailImporter(username=username, password=password, pluginRun=run, client=client, run_as_daemon=False)
gmail_plugin.add_to_schema()
```
%% Output
reading database_key from /home/eelco/.pymemri/pod_keys/keys.json
reading owner_key from /home/eelco/.pymemri/pod_keys/keys.json
%% Cell type:code id: tags:
``` python
gmail_plugin.run()
```
%% Output
IMAP client logged in.
SMTP client logged in.
Found 12 emails
Importing latest 12 emails...
downloading and converting batch 0/1
BULK: Writing 61/61 items/edges
Completed Bulk action, written 61 items/edges
Done.
%% Cell type:markdown id: tags:
### Run from CLI
%% Cell type:code id: tags:
``` python
pod_client = PodClient.from_local_keys()
print("Existing emails:", len(pod_client.search({"type": "EmailMessage"})))
```
%% Output
reading database_key from /home/eelco/.pymemri/pod_keys/keys.json
reading owner_key from /home/eelco/.pymemri/pod_keys/keys.json
Existing emails: 36
%% Cell type:code id: tags:
``` python
# For local testing
def create_pluginrun(pod_client):
account = Account(service="gmail_imap")
run = PluginRun("", "gmail_importer.plugin", "GmailImporter")
run = PluginRun("", "gmail_imap.plugin", "GmailImporter")
run.add_edge("account", account)
run.status = RUN_STARTED
pod_client.create(run)
pod_client.create(account)
pod_client.create_edge(run.get_edges("account")[0])
print(f"Created PluginRun {run.id}")
```
%% Cell type:code id: tags:
``` python
# Test with CLI
# To simulate front-end, run: $ password_simulator --run_id <run_id>
run_plugin(metadata="../metadata.json")
```
%% Cell type:markdown id: tags:
### Run from Docker
%% Cell type:code id: tags:
``` python
# hide
from nbdev.export import *
notebook2script()
```
%% Output
Converted authenticator.ipynb.
Converted gmailimporter.ipynb.
Converted imap_client.ipynb.
Converted index.ipynb.
Converted schema.ipynb.
Converted settings.ipynb.
Converted smtp_client.ipynb.
Converted utils.ipynb.
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment