<h4id="run_plugin"class="doc_header"><code>run_plugin</code><ahref="https://gitlab.memri.io/memri/pymemri/tree/prod/pymemri/plugin/pluginbase.py#L145"class="source_link"style="float:right">[source]</a></h4><blockquote><p><code>run_plugin</code>(<strong><code>pod_full_address</code></strong>:<code>Param object at 0x7fad54cdaa10></code>=<em><code>None</code></em>, <strong><code>plugin_run_id</code></strong>:<code>Param object at 0x7fad54cdaa50></code>=<em><code>None</code></em>, <strong><code>database_key</code></strong>:<code>Param object at 0x7fad54cdacd0></code>=<em><code>None</code></em>, <strong><code>owner_key</code></strong>:<code>Param object at 0x7fad54cdaa90></code>=<em><code>None</code></em>)</p>
<h4id="run_plugin"class="doc_header"><code>run_plugin</code><ahref="https://gitlab.memri.io/memri/pymemri/tree/prod/pymemri/plugin/pluginbase.py#L139"class="source_link"style="float:right">[source]</a></h4><blockquote><p><code>run_plugin</code>(<strong><code>pod_full_address</code></strong>:<code>Param object at 0x7fdc47df5e90></code>=<em><code>None</code></em>, <strong><code>plugin_run_id</code></strong>:<code>Param object at 0x7fdc47a4d7d0></code>=<em><code>None</code></em>, <strong><code>database_key</code></strong>:<code>Param object at 0x7fdc496dcf10></code>=<em><code>None</code></em>, <strong><code>owner_key</code></strong>:<code>Param object at 0x7fdc496dcf90></code>=<em><code>None</code></em>, <strong><code>from_pod</code></strong>:<code>Param object at 0x7fdc44203c50></code>=<em><code>False</code></em>, <strong><code>container</code></strong>:<code>Param object at 0x7fdc4a0afad0></code>=<em><code>None</code></em>)</p>
<p>To start a plugin on your local machine, you can use the CLI. This will create a client for you, and run the code defined in <code><myplugin>.run()</code></p>
<p>In production, we start plugins by making an API call to the pod, which in turn creates an environment for the plugin and starts it (currently on docker is supported). We can start this process using the CLI by provding <code>--from_pod==True</code> and providing a <code>--container</code> (the docker container used by the pod). <strong>Note that the provided docker container should be installed within the Pod environemnt (e.g. <code>docker build -t pymemri .</code> for this repo) in order to start it.</strong></p>
> Pymemri is a python client for the Memri Personal online datastore (pod). This client can be used to build plugins in python. Plugins connect and add the information to your Pod. Plugins that <b>import your data from external services</b> are called **Importers** (Gmail, WhatsApp, etc.). Plugins that <b>connect new data to the existing data</b> are called **indexers** (face recognition, spam detection, object detection, etc.). Lastly there are plugins that <b>execute actions</b> (sending messages, uploading files).
%% Cell type:markdown id: tags:
[](https://gitlab.memri.io/memri/pymemri/-/pipelines/latest)
This repository is built with [nbdev](https://github.com/fastai/nbdev), which means that the repo structure has a few differences compared to a standard python repo.
%% Cell type:markdown id: tags:
## Installing
%% Cell type:markdown id: tags:
### As a package
```bash
pip install pymemri
```
%% Cell type:markdown id: tags:
### Development
To install the Python package, and correctly setup nbdev for development run:
```bash
pip install-e.&& nbdev_install_git_hooks
```
The last command configures git to automatically clean metadata from your notebooks before a commit.
%% Cell type:markdown id: tags:
## Quickstart
To use the pymemri `PodClient`, we first need to have a pod running. The quickest way to do this is to install from the [pod repo](https://gitlab.memri.io/memri/pod), and run `./examples/run_development.sh` from within that repo.
The Python integrators are written in [nbdev](https://nbdev.fast.ai/)([video](https://www.youtube.com/watch?v=9Q6sLbz37gk&t=1301s)). With nbdev, it is encouraged to write code in
[Jupyter Notebooks](https://jupyter.readthedocs.io/en/latest/install/notebook-classic.html). Nbdev syncs all the notebooks in `/nbs` with the python code in `/pymemri`. Tests are written side by side with the code in the notebooks, and documentation is automatically generated from the code and markdown in the notebooks and exported into the `/docs` folder. Check out the [nbdev quickstart](wiki/nbdev_quickstart.md) for an introduction, **watch the video linked above**, or see the [nbdev documentation](https://nbdev.fast.ai/) for a all functionalities and tutorials.
"In production, we start plugins by making an API call to the pod, which in turn creates an environment for the plugin and starts it (currently on docker is supported). We can start this process using the CLI by provding `--from_pod==True` and providing a `--container` (the docker container used by the pod). **Note that the provided docker container should be installed within the Pod environemnt (e.g. `docker build -t pymemri .` for this repo) in order to start it.** "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Used arguments passed to `run_plugin()` (ignoring environment)\n",
The memri [pod](https://gitlab.memri.io/memri/pod) uses a plugin system to add features to the backend memri backend. Plugins, can import your data (importers), change your data (indexers), or call other serivces. Users can define their own plugins to add new behaviour to their memri app. Let's use the following plugin as an example of how we can start plugins.
%% Cell type:code id: tags:
``` python
# export
# hide
classMyItem(Item):
properties=Item.properties+["name","age"]
edges=Item.edges
def__init__(self,name=None,age=None,**kwargs):
super().__init__(**kwargs)
self.name=name
self.age=age
classMyPlugin(PluginBase):
""""""
properties=PluginBase.properties
edges=PluginBase.edges
def__init__(self,**kwargs):
super().__init__(**kwargs)
self.pluginPackage="pymemri.plugin.pluginbase"
defrun(self,run,client):
print("running")
client.create(MyItem("some person",20))
defadd_to_schema(self,client):
client.add_to_schema(MyItem("my name",10))
```
%% Cell type:markdown id: tags:
Memri plugins need to define at least 2 methods: `.run()` and `.add_to_schema()`. `.run()` defines the logic of the plugin. `.add_to_schema()` defines the schema for the plugin in the pod. Note that currently, `add_to_schema` requires all item to **have all properties defined that are used in the plugin**. In the future, we might replace add_to_schema, to be done automatically, based on a declarative schema defined in the plugin.
Plugins can be started using the pymemri `run_plugin` CLI. To use the CLI, you can either pass your run arguments as parameters, or set them as environment variables. If both are set, the CLI will prefer the passed arguments.
%% Cell type:code id: tags:
``` python
# hide
# export
defrun_plugin_from_run_id(run_id,client):
run=client.get(run_id)
plugins=run.plugin
iflen(plugins)==0:
raiseValueError(f"plugin run {run_id} has no plugin attached to it. Make sure there is a 'plugin' \
edge from your run to the actual plugin object.")
iflen(plugins)>1:
raiseValueError("Too many plugins attached to run")
In production, we start plugins by making an API call to the pod, which in turn creates an environment for the plugin and starts it (currently on docker is supported). We can start this process using the CLI by provding `--from_pod==True` and providing a `--container` (the docker container used by the pod). **Note that the provided docker container should be installed within the Pod environemnt (e.g. `docker build -t pymemri .` for this repo) in order to start it.**
# res = requests.post(f"{self.base_url}/run_importer", json=body)
# if res.status_code != 200:
# print(f"Failed to start importer on {url}:\n{res.status_code}: {res.text}")
# else:
# print("Starting importer")
# except requests.exceptions.RequestException as e:
# print("Error with calling importer {e}")
```
%% Cell type:markdown id: tags:
Pymemri communicates with the pod via the `PodClient`. The PodClient requires you to provide a [database key](https://gitlab.memri.io/memri/pod/-/blob/dev/docs/HTTP_API.md#user-content-api-authentication-credentials) and an [owner key](https://gitlab.memri.io/memri/pod/-/blob/dev/docs/HTTP_API.md#user-content-api-authentication-credentials). During development, you don't have to worry about these keys, you can just omit the keys when initializing the `PodClient`, which creates a new user by defining random keys. *Note that this will create a new database for your every time you create a PodClient, if you want to access the same database with multiple PodClients, you have to set the same keys* When you are using the app, setting the keys in the pod, and passing them when calling an integrator is handled for you by the app itself.
%% Cell type:code id: tags:
``` python
client=PodClient()
success=client.test_connection()
assertsuccess
```
%% Output
Succesfully connected to pod
%% Cell type:markdown id: tags:
## Creating Items and Edges
%% Cell type:markdown id: tags:
Now that we have access to the pod, we can create items here and upload them to the pod. All items are defined in the schema of the pod. When Initializing an Item, always make sure to use the from_data classmethod to initialize.
[{'item': Person (#806a88787321a9a81054e63d17ad2fbb), 'name': 'sender'}]
[{'item': Person (#1282de00d7dbd7d3e24d36ffe180b0d3), 'name': 'sender'}]
%% Cell type:markdown id: tags:
If we use the normal `client.get` (without `expanded=False`), we also get items directly connected to the Item.
%% Cell type:code id: tags:
``` python
email_from_db=client.get(email_item.id)
```
%% Cell type:code id: tags:
``` python
assertisinstance(email_from_db.sender[0],Person)
```
%% Cell type:markdown id: tags:
# Fetching and updating Items
%% Cell type:markdown id: tags:
## Normal Items
%% Cell type:markdown id: tags:
We can use the client to fetch data from the database. This is in particular useful for indexers, which often use data in the database as input for their models. The simplest form of querying the database is by querying items in the pod by their id (unique identifier).
When we don't know the ids of the items we want to fetch, we can also search by property. We can use this for instance when we want to query all items from a particular type to perform some indexing on. We can get all `Person` Items from the db by:
In the near future, Pod will support searching by user defined properties as well. This will allow for the following. **warning, this is currently not supported**
To work with files, the `PodClient` has a file api. The file api works by posting a blob to the `upload_file` endpoint, and creating an Item with a property with the same sha256 as the sha used in the endpoint.
%% Cell type:code id: tags:
``` python
from pymemri.data.photo import *
```
%% Cell type:code id: tags:
``` python
x = np.random.randint(0, 255+1, size=(640, 640), dtype=np.uint8)
photo = IPhoto.from_np(x)
file = photo.file[0]
succes = client.create(file)
succes2 = client._upload_image(x)
assert succes
assert succes2
```
%% Cell type:code id: tags:
``` python
data = client.get_file(file.sha256)
arr = np.frombuffer(data, dtype=np.uint8)
assert (arr.reshape(640,640) == x).all()
```
%% Cell type:markdown id: tags:
### Photo API
%% Cell type:markdown id: tags:
For photos we do this automatically using `PodClient.create` on a Photo and `PodClient.get_photo`:
%% Cell type:code id: tags:
``` python
x = np.random.randint(0, 255+1, size=(640, 640), dtype=np.uint8)
