" Creates a File item in POD and uploads it's contents\n",
" Requires raw file data.\n",
" Optional keyword arguments may set filename or it's edges (in form of Edge() objects)\n",
"\n",
" \"\"\"\n",
" sha = sha256(data).hexdigest()\n",
" file = File(sha256=sha)\n",
" for key, value in kwargs.items():\n",
" if hasattr(file, key):\n",
" setattr(file, key, value)\n",
" self.create(file)\n",
" self.upload_file_b(data)\n",
" return file\n",
"\n",
" def create_photo_file(self, photo):\n",
" # TODO: currently this only works for numpy images\n",
" file = photo.file[0]\n",
" self.create(file)\n",
" return self._upload_image(photo.data)\n",
"\n",
" def _upload_image(self, arr):\n",
" # TODO: currently this only works for numpy images\n",
" return self.upload_file(arr.tobytes())\n",
"\n",
" def upload_file(self, file):\n",
" # TODO: currently this only works for numpy images\n",
" def upload_file(self, data):\n",
" try:\n",
" sha = sha256(data).hexdigest()\n",
" result = requests.post(f\"{self.base_url}/upload_file/{self.database_key}/{sha}\", data=data)\n",
" if result.status_code != 200:\n",
" print(result, result.content)\n",
" return False\n",
" else:\n",
" return True\n",
" except requests.exceptions.RequestException as e:\n",
" print(e)\n",
" return False\n",
"\n",
" def upload_file_b(self, data):\n",
" \"\"\"\n",
" Uploads file contents to the POD using auth json.\n",
" Plugins must use this method only.\n",
" \"\"\"\n",
" try:\n",
" sha = sha256(file).hexdigest()\n",
" result = requests.post(f\"{self.base_url}/upload_file/{self.database_key}/{sha}\", data=file)\n",
" sha = sha256(data).hexdigest()\n",
" auth = quote(json.dumps(self.auth_json))\n",
" result = requests.post(f\"{self.base_url}/upload_file_b/{auth}/{sha}\", data=data)\n",
" if result.status_code != 200:\n",
" print(result, result.content)\n",
" return False\n",
...
...
@@ -914,6 +951,21 @@
"To work with files, the `PodClient` has a file api. The file api works by posting a blob to the `upload_file` endpoint, and creating an Item with a property with the same sha256 as the sha used in the endpoint."
Pymemri communicates with the pod via the `PodClient`. The PodClient requires you to provide a [database key](https://gitlab.memri.io/memri/pod/-/blob/dev/docs/HTTP_API.md#user-content-api-authentication-credentials) and an [owner key](https://gitlab.memri.io/memri/pod/-/blob/dev/docs/HTTP_API.md#user-content-api-authentication-credentials). During development, you don't have to worry about these keys, you can just omit the keys when initializing the `PodClient`, which creates a new user by defining random keys. *Note that this will create a new database for your every time you create a PodClient, if you want to access the same database with multiple PodClients, you have to set the same keys* When you are using the app, setting the keys in the pod, and passing them when calling an integrator is handled for you by the app itself.
%% Cell type:code id: tags:
``` python
client=PodClient()
success=client.test_connection()
assertsuccess
```
%% Output
Succesfully connected to pod
%% Cell type:markdown id: tags:
## Creating Items and Edges
%% Cell type:markdown id: tags:
Now that we have access to the pod, we can create items here and upload them to the pod. All items are defined in the schema of the pod. When Initializing an Item, always make sure to use the from_data classmethod to initialize.
[{'item': Person (#d2baa57bb8e414314b8cca154236695c), 'name': 'sender'}]
%% Cell type:markdown id: tags:
If we use the normal `client.get` (without `expanded=False`), we also get items directly connected to the Item.
%% Cell type:code id: tags:
``` python
email_from_db=client.get(email_item.id)
```
%% Cell type:code id: tags:
``` python
assertisinstance(email_from_db.sender[0],Person)
```
%% Cell type:markdown id: tags:
# Fetching and updating Items
%% Cell type:markdown id: tags:
## Normal Items
%% Cell type:markdown id: tags:
We can use the client to fetch data from the database. This is in particular useful for indexers, which often use data in the database as input for their models. The simplest form of querying the database is by querying items in the pod by their id (unique identifier).
When we don't know the ids of the items we want to fetch, we can also search by property. We can use this for instance when we want to query all items from a particular type to perform some indexing on. We can get all `Person` Items from the db by:
In the near future, Pod will support searching by user defined properties as well. This will allow for the following. **warning, this is currently not supported**
To work with files, the `PodClient` has a file api. The file api works by posting a blob to the `upload_file` endpoint, and creating an Item with a property with the same sha256 as the sha used in the endpoint.
x = np.random.randint(0, 255+1, size=(640, 640), dtype=np.uint8)
photo = IPhoto.from_np(x)
file = photo.file[0]
succes = client.create(file)
succes2 = client._upload_image(x)
assert succes
assert succes2
```
%% Cell type:code id: tags:
``` python
data = client.get_file(file.sha256)
arr = np.frombuffer(data, dtype=np.uint8)
assert (arr.reshape(640,640) == x).all()
```
%% Cell type:markdown id: tags:
### Photo API
%% Cell type:markdown id: tags:
For photos we do this automatically using `PodClient.create` on a Photo and `PodClient.get_photo`:
%% Cell type:code id: tags:
``` python
x = np.random.randint(0, 255+1, size=(640, 640), dtype=np.uint8)
photo = IPhoto.from_np(x)
```
%% Cell type:code id: tags:
``` python
succes = client.add_to_schema(IPhoto.from_np(x))
```
%% Cell type:code id: tags:
``` python
assert client.create(photo)
```
%% Cell type:code id: tags:
``` python
res = client.get_photo(photo.id, size=640)
```
%% Cell type:code id: tags:
``` python
res
```
%% Output
IPhoto (#4744838b374de38cc4183cac677d5cf6)
%% Cell type:code id: tags:
``` python
assert (res.data == x).all()
```
%% Cell type:markdown id: tags:
## Bulk API
%% Cell type:markdown id: tags:
Currently, an api call takes a lot of time (~0.1 second). If you want to write many data items to the pod, you can use the bulk api for efficiency reasons. Currently, only creating `Items` and `Edges` is supported, support for updating and deletion will follow.
%% Cell type:code id: tags:
``` python
dogs = [Dog(name=f"dog number {i}") for i in range(100)]
