Mautic 3 (M3) was announced a few months back but there is no list of specific changes with technical details that should happen in this breaking change (BC) version. My concern is that we plan to make too big changes and we'll never see it finished. M3 should include only the most necessary changes. The breaking changes that we cannot do in a minor release.

Disclamer

I work for Mautic, Inc. but thoughts in this post are my own as a Mautic community member. My goal is to keep the discussion going and provide some specifics that should be base for the M3 version. And what can be done now with M2 already or any time, really.

Things I'd like to see in Mautic 3, but...

... but those could be developed in any time. They do not really break backward compatibility.

Marketplace

Missing marketplace is by my opinion the greatest weakness of Mautic ecosystem today. The way the marketplace should work is that in the Mautic administration there will be a list of plugins with description, user rating, comments and the Install button. That's it. I hate how complicated it is right now:

1. download a package from xxx,
2. unzip the package ...
3. clear the cache and finally
4. click the Install button.

This is terrible. If our objective is to build system for marketers then this isn't it.

And for the plugin developers? There should be central repository where they would git push their new version and the marketplace would send the notice to all users who have the plugin installed that a new version is available. And again, update with 1 click. Optimally, the marketplace should also make payments for plugins easy. For 3 reasons:

1. If it will be profitable for devs to build Mautic plugins then we'll see more of them with quality implementation.
2. Mautic ORG could get some small % from processing the payments and host the plugin repository to fund its community activities.
3. Customers could trust Mautic ORG that the plugin will be delivered after the payment. At least it's more trustworthy than paying some random guy on the internet and trust that he'll deliver.

But is this a breaking change? No.

Config in database

Whenever a change to the config file (app/config/local.php) is made then Mautic needs the cache cleared to know about the change. That's resource and time consuming and very inconvenient. If the config file would contain only database credentials then the rest of the configuration can be in a database table. No cache clearing needed doing changes.

But is this a breaking change? No. A migration can handle it.

Twig, Annotations, Yaml

Every developer that starts working on Mautic asks about Twig, entity and route annotations and YAML for config files. The decision to avoid pre-processing from different formats to PHP and use PHP directly was made from the beginning. I know, the PHP version of those different formats is generated and cached once during the cache warmup and then the app runs fast. But when you consider how much trouble Mautic has with the minimal cache even now...

But is this a breaking change? Likely. Do we need different formats for what's working now? I don't think so.

Composer

Wouldn't it be great if users could install Mautic with just 1 command (composer install mautic) instad of downloading, unzipping and uploading? And plugins, themes too. In theory, composer is a PHP app as any other so we should be able to build some UI on top of it so users could click buttons instead of executing commands.

There are issues to be solved with this approach. Mautic's composer.json and composer.lock are tracked by git so if some plugin would install a new dependency then Mautic update would remove it. Hopefully we will be able to find a way how plugins will have their own composer config files and vendor folders. Ideally we would be able to really disable each installed plugin so if there is a problem in a plugin it won't make the whole Mautic unavailable.

This is clearly something I need to research more. Food for next post maybe.

But is this a breaking change? Hopefully not.

Microservices

DBH sees M3 as a bunch of microservices. I can understand the reasoning. But let's look at the current state first.

Mautic 2 is divided into several bundles. Page Bundle, Email Bundle, Campaign Bundle, etc. That sounds a little like microservices architecture already. I know, it's modular architecture, but hear me out. The idea beyond bundles is that those are packaged features. Remove one and Mautic will still work. Just the 1 feature would be gone. I'm not saying that you can do that now. But that's the theory.

Bundles of course must communicate between each other. Email Bundle must tell the Campaign Bundle that it can provide functionality for the Send Email action and Email Opened decision. The way it works now is with events. An event being a PHP object with necessary data dispatched during the PHP runtime in this case. Campaign Bundle dispatches an event that it's about to open the Campaign Builder. All other bundles can subscribe to receive this event and add new actions, decisions or conditions to the campaign builder.

Microservices are stand-alone apps that communicate between each other. If we break bundles into stand-alone services we will need to find a different communication channels. There are 2 main types of messages. Synchronous (like a HTTP request) and asynchronous (a queue like RabbitMQ). Each is good for different tasks.

For example if a contact clicks on a link in an email then the email service must tell this to the contact service. Contact service will provide contact ID and email service will set the contact ID to the cookie. This must be a synchronous message as the contact ID must be set in the same HTTP response.

An example of good use for asynchronous message could be if there is a webhook configured on email click. The email service would queue the message that the click has happened and wouldn't wait for any response. The webhook service would pick it up and processed the webhook whenever it could.

Pros and cons

Bundles VS Microservices Bundles have fast communication channel right now. It cannot get any faster. VS Additional HTTP requests between services will slow it down. A bug in one bundle can break whole Mautic.VS A bug in a microservice will disable that one microservice. Communication happens in the PHP itself. VS Needs a queue system like RabbitMQ to communicate. Impossible to scale up one bundle. VS Each service can be scaled up as needed.

One last though on this. We can start writing microservices within the M2 era and simply let users choose whether they want to use the internal bundle or configure the service replacement instead.

Edit 2019-04-02: I just found this post about microservices vs modules which goes deeper into the topic.

So is this a breaking change? It isn't.

Preparation on headless

DBH wants M3 to be headless with React or Vue UI. I love that idea! Although in my humble opinion this cannot be done in 1 year time with all the other changes combined. What I suggest is to start the preparation for it. The main step would be GraphQL API. The reason being that it's very bendable. The React/Vue UI could be built on top of the current REST API but I'm afraid we'd run on its limits very soon. GraphQL allows the UI app to query for the data it needs. Even associated data in multiple layers.

The Mautic-UI should be another repository anyway, right? It should not be bound to the Mautic-API-server at all. So it will have different release cycle. I say let's start working on it during the M3 lifespan. Users could switch to the React-UI any time.

So is this a breaking change? No.

What changes M3 really needs

Now here are some changes that are breaking backward compatibility and therefore must be made in major version.

Increase minimal PHP version

PHP 5.6 and 7.0 are not supported anymore. See php.net/supported-versions. PHP 7.1 have 1 more year of support but it will take something like 1 year to get M3 out so it doesn't make sense to support it. I suggest PHP 7.2+.

Increase minimal MySql version

Could we go with MySQL8? It would allow us to use some cool features that would speed up the platform and the development. Like invisible indexes, querying JSON documents and so on. Also, now we support MySql and silently also MariaDB, but since version 8 the both databases aren't really compatible. Even 5.7 introduced differences. Can we support only one of them to save some sleepless nights?

Upgrade dependencies

Mautic runs on Symfony 2.8 and other outdated packages close to end of life or already dead versions. We cannot upgrade them without BC changes in the code and in the Mautic requirements so M3 is a great chance to do this.

Autowiring

New Symfony also comes with some goodies like autowiring. It will save a couple thousand rows defining all the dependencies but mainly it will improve the developer experience and speed up development.

Remove webhosting requirement

Mautic 1 and 2 were built with many compromises because it was suppose to run on a shared webhosting. I've read dozens of discussions where the community advised against running Mautic on a shared hosting. The reason being that Mautic needs some server fine tuning to work well. It's mostly different on each hosting. Last problem in the discussion I was involved in was outdated LIBXML version (2.7.7) that the webhosting was not able to upgrade and because this one outdated PHP module the user had to move his mautic to another server.

Without SSH access it's very hard to run commands. Without running the commands the user has very limited control over Mautic. Some tasks takes longer than a HTTP request can handle. Like the UI updater. It's the reason why Mautic got bad reputation with upgrades.

I'd like Mautic to be professional tool for VPS only where SSH access is a requirement. Yes, Mautic should be built for marketers, but no, marketers should not be the group who will install and maintain it. I wouldn't expect that average marketer also repairs her/his own car nor fix plumbing in his a house. Similarly the IT guys shouldn't do marketing ;) Some tasks should be done by professionals.

Finish renaming

During Mautic 1 and 2 lifetime some renaming happened. Lead > Contact, List > Segment. It was renamed in the UI, but the code is a mess now. Some old parts use the old name and some the new one. Renaming would be a BC change. Let's finish this madness in M3.

And stop renaming, please. It was made because the new terms are "industry standard". I've heard in some community discussion a question why don't Mautic use leads instead of contacts as that's the industry standard. Most CRMs have leads and contacts. The term doesn't really matter. Renaming just cause headaches for developers, translators, documentation maintainers and for the users.

Multibyte UTF8 database

Many Mautic users experienced troubles with multibyte characters like Japanese language or emoticons which is a UTF8 charset limitation. Even though M2 tries to deal with it with transliteration and emoticon dictionary, it will never be working on all columns and for all emoticons. Mautic use UTF8 as default char set. All those problems would be solved if it would use the UTF8MB4 char set.

There already was an attempt to change that in M2 but it was considered as a BC break.

Bigint Unsigned autoincrement for stat tables

For a reason I don't really understand, Doctrine creates the autoincrement IDs as INT signed. No one noticed at first that this could be a problem. When the statistical database tables like email_stats grow into billions of rows, it becomes an issue. I tried to fix it at least for the new Mautic instances, but it was reverted later as it was causing problems with a doctrine command. Heath hit that limit and wrote an excelent article about what happened, how to configure monitoring to give you a heads up and how to fix it. Even though it won't be an easy migration. We have to fix it for M3 and build an upgrade path for M2 instances.

Custom field rewrite

Adding a new custom field means creating a new column in the leads table. That's problematic because MySql has a limit of how many indexes can be on a table. Another problem comes when adding a new column to a table with lots of rows. It can take longer than HTTP request timeout. That makes creating new columns impossible via UI.

So I suggest to re-build the database schema to such form that doesn't require a schema changes when adding a new field. And maybe let forms and other bundles which needs it use the new schema for fields instead of the new table per form?

I may or may not work (who knows?) on some stuff internally so I'll write a post on this topic once I'll have some performance test data.

Consider removing some features from the core

Mautic core is quite heavy. We should go through list of bundles and their main features and decide whether those shouldn't be moved to a plugin that can be additionally installed if the user needs it. Does anyone use internal landing pages? Everybody needs the SalesFoce integration? I'm not sure about that.

WYSIWYG editor

The current WYSIWYG editor should be replaced with an alternative as its licensing has changed. I'm torn if this is a BC change since we've replaced the editor once already. But I think it should be done in M3 as it's quite big change and the new editor can support only subset of features of the old one.

All events in one table

While Event Sourcing is becoming a buzzword in the IT community I think that we should implement at least a light version of it to Mautic. Right now the events (page hits, email stats, campaign logs, audit logs, ...) are located in many tables. This makes it very hard and slow to pull all events at one place. Like the contact detail page. Proper pagination is an impossible task with this structure. Even though there is pagination you may notice it doesn't show correct events. Mostly the order of the events is wrong. It pulls 20 events from each table. There is no way how to fix it without having a global table.

I know that each event needs different columns. That's fine. There may be specialized tables for some events. Those tables can be joined together if needed. The centralized table should keep simple information like what event happened, what time, what entity is related. Additions only. No updates.

I suggest the columns for this global event table would be:

• id
• contact_id
• type
• date

We'll of course have to make some performance tests with large number of rows before going this way. But as this table would be read-write only (no updates nor deletes) we could use partitioning.

It'd be interesting to research if we could make automatic audit log this way on each entity save.

Controllers and Commands as a service

Controllers and commands are very hard to unit-test because they are loading dependencies all over the place from the container. I'd like them to define them as all other services. Plus controllers have very deep inheritance layer. But it would be very hard to remove with current UI. I'd leave that when Mautic will be headless one day.

Use doctrine service instead of Entity Manager

I steal this from Alan who pointed this out in Slack:

Random thought for M3 as I'm dealing with this in M2 at the moment, we should not pass Doctrine's entity manager directly into services. Rather use the doctrine service, a factory or helper that allows us to restart the entity manager if we encounter a SQL error. Currently, restarting the entity manager requires the doctrine service and it recreates the entity manager object leaving the one passed to services closed and so we can't recover from it.