|
|
**An importer plugin is considered done when**
|
|
|
|
|
|
## General
|
|
|
- Your code is reasonably composed. Use small reusable blocks of code where possible.
|
|
|
- Your plugin is installable via a setup.py.
|
|
|
- There is a Docker file that installs all requirements to run the plugin
|
|
|
- Approved schema is added to https://gitlab.memri.io/memri/schema for those fields that are common
|
|
|
- The [MPPL license](https://memri.io/license.html) is added to the repository
|
|
|
- The right icon/logo of the service is added to the repository (when applicable)
|
|
|
|
|
|
## Data
|
|
|
- All relevant data is imported to the Pod (defined per plugin)
|
|
|
- Data is imported incrementally: Data is not duplicated / stored twice when the plugin imports data multiple times.
|
|
|
|
|
|
## Tests
|
|
|
- A test account for the corresponding service, which holds example data of all the imported data types, with enough variation to catch most edge cases
|
|
|
- Mocking for the tests dependent on the external service
|
|
|
- CI runs tests to see whether all data types are downloaded, stored and connected in the Pod
|
|
|
- A test that runs your plugin end-to-end on a small sample of data.
|
|
|
- There are unit tests where possible: for instance for parsing, inference, etc..
|
|
|
- If you are creating a plugin as a regular python project, use pytest for testing.
|
|
|
- If you are using nbdev, you can use nbdev_test_nbs.
|
|
|
|
|
|
## Authentication
|
|
|
- Authentication is implemented
|
|
|
- A clear description of how credentials are made available to the plugin, in order to enable front end and platform developers to implement that
|
|
|
- A clear description of what trust we are asking from users, e.g. are they storing their credentials on the server their Pod is hosted, is communication encrypted, etc.
|
|
|
|
|
|
## Documentation
|
|
|
- The plugin has comprehensive documentation on the:
|
|
|
- Usage
|
|
|
- Implementation
|
|
|
- Data fields imported, including datatypes of those fields
|
|
|
- What data is not imported
|
|
|
- Frequency the plugin can run (this should be <5 min for social services, and not e.g. only daily because of use of GDPR requests)
|
|
|
- Time it takes to run the plugin
|
|
|
- System requirements (RAM, CPU, disk and others if relevant)
|
|
|
- The authentication flow: What login mechanism is used, how is 2FA handled, etc.
|
|
|
|
|
|
## When using nbdev
|
|
|
- Notebooks should live in the /nbs folder.
|
|
|
- In nbdev notebooks are meant for building the lib and writing tests, not for scripts. If you want to add a scripts, create a /scripts folder and add it there.
|
|
|
- If you have ci setup to generate docs, request a memri maintainer to activate gitlab pages. |
|
|
\ No newline at end of file |