I wrote in my previous post about Mautic 3 that the Marketplace is the greatest weakness of Mautic today. Go to that paragraph and read what I mean by Marketplace so we are all on the same page.
At mautic.org there already is a section called Marketplace but that's not what I consider a marketplace. It's just a list of plugins.
Why is Marketplace such a big deal?
Enables modular architecture
Everyone probably knows already that monolithical projects are bad and why is that. I'll save you from my rants about monoliths. Is Mautic a monolith? I would say so. It is slightly extendable monolith. If there would be an easy way how to install additional modules then we could break Mautic down into the core and the plugins and users would install the plugins they need. A problem within one plugin would not hold back the release of a new Mautic core version. A problem in one plugin would not require a new Mautic core release that affects everyone, but it would affect only the users who use the plugin. The marketplace should allow real modular architecture.
The marketplace shouldn't also allow users to upgrade a plugin or Mautic core until all modules are compatible so users won't end up with broken Mautic.
One more thought on this topic, imagine iOS or Android without the App Stores. It would be a useless piece of software. App Store is what makes the whole platform valuable.
Allow users to run up to date software
It's important for security and performance reasons to run the up to date software. We've seen it many times. Mautic is also heavy for integrations with other APIs. It happens way too often that the API version that a Mautic integration supports suddenly ends. It's essential to get an update that supports the new API version to all users in very short period of time. Right now, there is no channel that could do that. The marketplace should be that channel.
Allow developers provide professional services
Software companies need to get their software and updates to their users reliably. Right now Mautic doesn't allow that. And that's the reason there is not many developers in the Mautic ecosystem. Once there will be a way how to easily install plugins and themes and keep them up to date then professional develoment teams can keep their high level of service. The marketplace should make that possible.
What I mean by marketplace
A basic marketplace has 2 parts:
- A global registry of plugins/themes/...
- A Mautic core plugin that will allow users to browse, install and update the plugins published to the global registry.
There are other problems to consider.
What if a plugin will need some other dependencies (PHP libraries)? Should they be part of the plugin? If so, there would have to be some build process that each plugin developer would have to take care of.
Battle tested package manager as a base
Hmm, PHP packages, dependency management, package registry, installation, updates. That reminds me of what Composer does. It would be silly to build all that for Mautic only when there is already all that and it's standard for PHP packages.
My proposal is to use Composer for Mautic plugin installation. And we are already a half way there thanks to Don Gilbert's Mautic Composer Plugin. It will allow you to publish Mautic plugins to Composer registry called Packagist and install them with Composer.
Let's see what we need to build a basic version of the Marketplace:
- Package registry (Packagist) - DONE
- Dependency manager (Composer) - DONE
- Install script (Mautic Composer Plugin) - DONE
- Mautic Marketplace Plugin - MISSING
Or is it? I have a prove of concept written in one evening. The code in this work in progress pull request if you want to peek into the code: https://github.com/mautic/mautic/pull/8257
I implemented only several commands so far, no UI (proove of concept). Here they are:
List of plugins
Browse available plugins
$ bin/console mautic:marketplace:list
+------------------------------------------+--------------------------------------------------------------------------------+-----------+--------+
| name | description | downloads | favers |
+------------------------------------------+--------------------------------------------------------------------------------+-----------+--------+
| thedmsgroup/mautic-contact-client-bundle | Create custom integrations without writing code. | 3853 | 36 |
| raow/mautic-rss-to-email-bundle | | 514 | 42 |
| mautic/mautic-saelos-bundle | | 3966 | 11 |
| koco/mautic-recaptcha-bundle | This plugin brings reCAPTCHA integration to mautic. | 718 | 15 |
| thedmsgroup/mautic-extended-field-bundle | Extends custom fields for scalability and HIPAA/PCI compliance. | 2874 | 11 |
| thedmsgroup/mautic-enhancer-bundle | Various contact enhancer integrations for Mautic. | 2707 | 20 |
| thedmsgroup/mautic-dashboard-warm-bundle | Improves the performance of the dashboard by sharing/extending/warming caches. | 1744 | 9 |
| mtcextendee/mautic-plivo-bundle | | 162 | 2 |
| kuzmany/mautic-recommender-bundle | | 111 | 14 |
| thedmsgroup/mautic-segment-extras-bundle | Extends Mautic Lead Bundle Lead List (Segment) functionality. | 1322 | 5 |
| thedmsgroup/mautic-media-bundle | Pulls cost data from media advertising services. | 917 | 6 |
| thedmsgroup/mautic-health-bundle | Checks the health of the Mautic instance. | 2058 | 7 |
| thedmsgroup/mautic-contact-source-bundle | Creates API endpoints for receiving contacts from external sources. | 2730 | 19 |
| thedmsgroup/mautic-contact-ledger-bundle | Historical accounting for contacts | 2625 | 7 |
| thedmsgroup/mautic-campaign-watch-bundle | Visual improvements for campaigns. | 1659 | 7 |
+------------------------------------------+--------------------------------------------------------------------------------+-----------+--------+
Execution time: 507 msThis command uses an API request to the Packagist registry. It's a public API so no authentication is needed. There are already some Mautic plugins available to instal thanks to Don's Composer plugin mentioned above. And look (scroll the table to the right), we already have rating included. We can see how many times each plugin was downloaded and how many stars it has. If you imagine this table in Mautic's administration, there will be also a link to the Packagist detail page that looks for example like this for the thedmsgroup/mautic-contact-client-bundle plugin. It has some stats, description, screenshots, license info, requirements. All a package registry should have.
Install a plugin
Let's install the first plugin from the list above.
$ bin/console mautic:marketplace:install thedmsgroup/mautic-contact-client-bundle
Using version ^2.15 for thedmsgroup/mautic-contact-client-bundle
./composer.json has been updated
Loading composer repositories with package information
Updating dependencies (including require-dev)
Dependency resolution completed in 51.319 seconds
Analyzed 23326 packages to resolve dependencies
Analyzed 471046 rules to resolve dependencies
Dependency resolution completed in 0.014 seconds
Package operations: 9 installs, 0 updates, 0 removals
Installs: phpseclib/phpseclib:2.0.23, league/flysystem:1.0.61, league/flysystem-sftp:1.0.22, dropbox/dropbox-sdk:v1.1.7, league/flysystem-dropbox:1.0.4, league/flysystem-aws-s3-v3:1.0.23, namshi/cuzzle:2.0.3, mustache/mustache:v2.13.0, thedmsgroup/mautic-contact-client-bundle:2.15.1
- Installing phpseclib/phpseclib (2.0.23): Loading from cache
Extracting archive - Installing league/flysystem (1.0.61): Downloading (100%)
Extracting archive - Installing league/flysystem-sftp (1.0.22): Downloading (100%)
Extracting archive - Installing dropbox/dropbox-sdk (v1.1.7): Downloading (100%)
Extracting archive - Installing league/flysystem-dropbox (1.0.4): Downloading (100%)
Extracting archive - Installing league/flysystem-aws-s3-v3 (1.0.23): Downloading (100%)
Extracting archive - Installing namshi/cuzzle (2.0.3): Downloading (100%)
Extracting archive - Installing mustache/mustache (v2.13.0): Downloading (100%)
Extracting archive - Installing thedmsgroup/mautic-contact-client-bundle (2.15.1): Downloading (100%)
Extracting archive
Writing lock file
Generating autoload files
0
Execution time: 124407 msAfter this the plugin will appear in the plugins directory. We still need to clear the Mautic cache and run bin/console mautic:plugins:install to add it to the list of installed plugins and let the plugin to create its own database schema for example. It could all be part of this command but it already takes so long. Read the drawbacks bellow.
Remove a plugin
I won't copy-paste the output of this command here. It's similar to the install command. Composer will clean good.
Drawbacks
Execution time
I measure execution time bellow each command. That's the biggest issue so far. Composer takes too much time as it checks all dependencies. Installing a plugin takes around 2 minutes. That's too long for a HTTP request. The default timeout is 30 seconds. I tried to speed it up by disabling some Composer features but this is the fastest time I got. And then it needs to also clear Mautic's cache which may take another minute and install the plugin to the database which usually takes a few seconds. If we could break it down to more steps that would take less time we can use consecutive ajax requests to trigger them. But that's not an option here.
The only solution I can think of is to run the installation on background. So it would need another cron job. It sucks. Users will click the install button and then they will have to wait several minutes to see the plugin installed.
I'm open to ideas here.
Update 1 day later:
There are 2 main suggestions to speed up Composer:
1. Disable Xdebug
I have it disabled by default and enable it with special command prefix. So I thought I'm fine there. But just to be sure I tried to disable it like this: $ mv /usr/local/etc/php/7.2/conf.d/ext-xdebug.ini /usr/local/etc/php/7.2/conf.d/ext-xdebug.ini.disabled and it worked. Installing a plugin was done in 28 seconds. But that was because all the packages were cached locally. When I cleared Composer's global cache with $ composer clear-cache it took 90 seconds.
2. Install hirak/prestissimo Composer plugin
This plugin installs dependencies asynchronously. So it will speed up the installation a bit if the plugin has many dependencies. So If I clear Composer global cache and install the plugin again we get it in 50 seconds.
That's the best I can do so far. Again, it's too much for a HTTP request, but maybe we could guide people to increase their time (at least 90 seconds) and memory (at least 1GB) limits accordingly to install packages.
I also profiled the installation. I wanted to see if Composer is doing something more I could disable. But I don't see anything there. There is simply that much to do to resolve the dependencies.
Mautic update
If we install several plugins this way and then update Mautic it can break as Mautic is not installed via Composer but by rewriting files like barbars. But we can inspire by TYPO3 team who solved this. Then we could use Composer to update Mautic itself.
Update: Read the follow up article.
Conclusion
The work remaining is only a couple of evenings to build the GUI. But until we deal with the 2 identified drawbacks I don't see a reason to continue. So the point why I'm telling you all about this is to come up with solutions and make Mautic easily extendable. Easy for both the Mautic users and the plugin developers.
Let's discuss here: https://forum.mautic.org/t/mautic-marketplace/12108