We recently completed the “New Engine Claiming and Management” milestone on our development roadmap. Our goal was to make it easier for Engine owners to build, configure and test an engine, and then join the PolySwarm Marketplace, so we’ve completely redesigned the architecture.
Build and Configure
To make it easier to build and configure engines, we reduced the complexity of an engine’s minimum required functionality. An engine now consists of three standard components: a web service, a worker, and a configuration in the PolySwarm UI.
Web Service
Engine owners must run a publicly accessible web service to listen for webhook calls from the PolySwarm Marketplace. The new architecture uses webhooks to send bounties to engines. The body of the webhook call contains the information an engine needs to download an artifact and return a result. The web service communication uses standard HTTP protocols, so engine owners can easily use their preferred HTTP server technologies. Our Getting Started documentation provides guidance to use our example Python implementation in the microengine-webhooks-py project.
Worker
The worker is the component responsible for processing the content of a webhook call to download the artifact, process it, and return a result to the Marketplace’s result URI. The implementation of the worker is completely up to the engine developer. Our Getting Started documentation provides guidance to use our example Python implementation in the microengine-webhooks-py project.
Configuration
The PolySwarm UI now allows an engine owner to create webhooks and engines. When creating an engine, the engine owner defines the configuration for the engine. This configuration defines the operational parameters of the engine, such as types of artifacts supported, maximum file size, and mime types. The marketplace will use this configuration to decide which engines will receive a webhook call for each bounty. The PolySwarm Marketplace will send webhooks to all verified engines with a configuration indicating that they are capable of processing that bounty. See our latest documentation describing how to Create Your Engine in PolySwarm UI.
We removed the management of PolySwarm NCT tokens and wallets from the engine implementation and moved it internally to the PolySwarm Marketplace to be managed by the engine owner using the PolySwarm UI. The PolySwarm UI pages to manage tokens and wallets are under development and will be published before engines need to use mainnet NCT.
Testing
To make it easier to test engines, we removed the need to run a local copy of the entire marketplace and added features to test an engine directly against the live marketplace.
Since engine implementations now use webhooks and callbacks, integration testing is as easy as running your engine and web service locally, and calling the webhook endpoint in the engine’s web service. See our latest Integration Testing documentation for more information.
A common request that engine developers have been asking for is a way to test an engine against the live PolySwarm Marketplace. Since we do not want testing data polluting the live Marketplace data, we now create a private “Development Community” for each engine to use for live testing. See our latest Testing in the Marketplace documentation for more information.
What about existing engines that do not use webhooks?
All existing engines must be converted to the new webhook architecture. For all engines hosted by PolySwarm, we will do this conversion on behalf of the engine owner during the next 4-6 weeks. For all other engines, the engine owners will need to do this conversion before August 31st, 2021. Only engines supporting the webhook architecture will work on PolySwarm’s Mainnet when it is released. See our latest guidance for Migrating a PolySwarm-Client-based Engine to a Webhook-based Engine for more information.
Cheers,
Nick
