Hubzilla 5.0 Todo WIP

Mario Vavti via Zotlabs|Hubzilla Development

Tue, 14 Jan 2020 10:54:52 +0000 last edited: Thu, 26 Mar 2020 15:43:30 +0000

This is a list of tasks we need to accomplish before switching to zot6 and releasing 5.0
Please feel free to comment if you need to add something, know something or find something missing and i will complement the list...

Implement zot6 DM in the frontend
Move the mail app to addons
Revisit the state of rss contacts with zot6 (iirc there were issues?)
Create some useful widgets for HQ
Make HQ the default landing page
Implement shared ICONFIG variables (see here)

!!Zotlabs|Hubzilla Development
@M. Dent @Max Kostikov @Mike Macgirvin

show all 24 Comments

Hilmar R (u-462)

Tue, 14 Jan 2020 12:32:21 +0000

The folder library/ contains a very lot stuff and some of this appears unused on the site. In many cases, some subfolders have content provided by the vendor, like example files to execute. If such parts are not intented to be used (even in the near future), it better should become removed (or at least updated). Especially I want to point to cropper.js, where the contained version has a very lot of known serious security issues, that should not be distributed together with the Hz source when there is no need for.

Mario Vavti

Tue, 14 Jan 2020 13:12:09 +0000

@hilmar we do use cropper.js it was just updated to the most recent version in dev branch.

Hilmar R (u-462)

Tue, 14 Jan 2020 13:20:26 +0000

Another observation is the big diff between composer.json and composer.lock. Currently, I don't oversee how and how consistent composer is applied to Hz. May be it's worth to clean up sth.

Wed, 15 Jan 2020 08:50:03 +0000

Implement zot6 DM in the frontend

A little confused about what is hiding beneath this...

Mario Vavti

Wed, 15 Jan 2020 08:57:58 +0000

@Max Kostikov since mail is not available in zot6 we need a new interface to display direct messages. Zap implements this via a filter in mod network. I think we can do that the same way. I think in the backend we already support this kind of messages (item_private = 2).

Direct messages in zot6 are messages that are not addressed to a group but only to selected channels only. This can be accomplished via @! tags or the ACL tool.

Mario Vavti

Wed, 15 Jan 2020 09:02:32 +0000

@hilmar i am all for cleaning up the /library folder. Do you have a list of unused libraries in this folder?

Mario Vavti

Wed, 15 Jan 2020 09:05:51 +0000

@hilmar i think we don't actually use the .lock file since the /vendor directory is shipped with Hubzilla directly.

Wed, 15 Jan 2020 09:09:48 +0000

@Mario Vavti Thanks, I got it.
And what about current personal mail system? Will we save ability to communicate with Diaspora?
Also direct messages in standard stream looks not so convenient as separate messaging interface.

Mario Vavti

Wed, 15 Jan 2020 09:23:52 +0000

And what about current personal mail system? Will we save ability to communicate with Diaspora?

We could move /mail to addons. It could then be installed along with diaspora if needed. Not sure if it's possible to rewrite diaspora mail to save DMs in the item table instead.

Also direct messages in standard stream looks not so convenient as separate messaging interface.

Yeah, definitely. We could use a filter until a separate interface is ready. The filter is pretty straight forward iirc.

poVoq

Wed, 15 Jan 2020 16:57:49 +0000

Reading about Zap it seems like the ActivityPub addon might need some adjustments?
Furthermore any thoughts on Mastodon compatible polls and account migration?

Mario Vavti

Sat, 18 Jan 2020 09:13:33 +0000

@poVoq

Reading about Zap it seems like the ActivityPub addon might need some adjustments?

What are you referring to?

poVoq

Sat, 18 Jan 2020 18:26:40 +0000

@Mario Vavti According to some texts about Zap, it is internally quite similar to Activitypub and thus works closer together? But I have honestly no idea how the ActivityPub addon works in Hubzilla. I always assumed it has to translate everything in a more complex manner.

Mario Vavti

Sat, 18 Jan 2020 18:33:59 +0000

@poVoq zot6 is based on activitystreams2 and activitypub is mostly activitystreams2. The pubcrawl addon implements Activitypub which is included in zap core.

poVoq

Sat, 18 Jan 2020 20:09:04 +0000

@Mario Vavti yeah that is what I am referring to, so if it is in Zap core, maybe it also needs to be moved into Hubzilla core for optimal performance?

Mario Vavti

Sat, 18 Jan 2020 20:20:27 +0000

@poVoq there is no, at least to me, measurable performance difference between core and addon components.

Tue, 21 Jan 2020 16:43:34 +0000 last edited: Tue, 21 Jan 2020 16:39:05 +0000

The method in which addons are called in Hubzilla (and Zap for that matter) mean that there is no difference between "core" and "addon". Even if there were any significant difference, because of the architecture, it would most likely be the other way around - CORE components would run more slowly (as there is actually more overhead) and "addons" would likely run more quickly. But because of the program flow, assuming a properly configured PHP environment, I would be very surprised if there were any marked difference at all in performance in "core" versus "addon".

The only real distinction between "core" and "non-core" is organizational --- where do the files sit in the directory tree. Are they in the addon directory? Or are they in the Zotlabs/Module directory? In truth, that's really the only difference.

Tue, 21 Jan 2020 16:52:01 +0000 last edited: Tue, 21 Jan 2020 16:48:13 +0000

ADDITION: Implement shared ICONFIG variables (or verify with certainty these aren't needed by any current core function or addon) -- AFAIK - "shared" ICONFIG variables currently must be translated into AS objects/attributes/entities (don't know the proper term, sorry) and then "untranslated" at the other side. I haven't yet audited whether there are any modules or addons that rely on shared ICONFIG variables (except for those that I personally created that do). But if there are any, these will break.

The "workflow" addon has a method in place for workflow related iconfig variables that could be expanded and altered as needed. It's a bit "hackish" IMHO since it doesn't actually define every variable - but the method is allowed as far as I can tell in the AS standards.

Tue, 21 Jan 2020 17:02:14 +0000

@M. Dent
BTW I worked over one custom plugin for Roundcube webmail this month and found how there work with plugins are organized.
RC have contexts which based on which part of code is now running (e.g. login, config, mail and etc.) and plugins activate (or not) hooks depend on current context. I think this was implement for performance purposes.
Probably this approach may be interesting for us as well.

Hilmar R (u-462)

Wed, 22 Jan 2020 11:54:18 +0000 last edited: Wed, 22 Jan 2020 11:55:51 +0000

@M. Dent
Thanks for sharing your knowledge about Addon implemention.

At least for me, Im far away about the logic of apps and how that interfares with core. While working on the invite app, I see all essential codes as a part within core. Even the invire code is placed in core. On the other hand, without "installing" the addons, ie invite, there is no chance to call invite. In the hamburger app menu, the app is shown and leads to url/invite. Seeing only that, an app could be treated as an menu entry only. But probably there will be more behind. I also do not really detect what the stuff of the *.apd files should be; well, there are a version number, the url pointer, dependencies. Looking into the addon tree, there is no invite (it is in core). Two tables interfare as well: app and addon. The addon table tells about if installed or not, but has empty version numbers. Is there a thread running through that all? ... thru the terms Addon, App, Feature ,, Widget ...?

Thu, 23 Jan 2020 18:57:47 +0000 last edited: Thu, 23 Jan 2020 18:53:37 +0000

@hilmar

To some extent, there is a distinction between what is considered an "app" on the front end, and an "addon" on the backend. "Apps" are not necessarily tied to processes installed on the server hosting the channel. "Addons" are the backend code which runs when the "app" is accessed. With this system, it is possible to add an "app" to your hamburger menu that actually runs code not on the hub where your channel is housed, but is run on another server entirely (sometimes permission based using OWA).

Related to the bits of the invite code in core: Earlier on, many things were core which have since been moved to "modules" ('core addons'). In that process, remnants of some things were left in core itself and weren't fully moved into the module - or even moved into an addon. So, honestly, I think what you are seeing is simply partially completed work at moving things to "addons"/"modules". Sometimes because it was too complex and complicated to fully complete it at the time it was done, in other cases, because maybe it does need to be in the core code itself and the decision to try to separate it into an "addon"/"module" was hasty. (I'm not making any statements about any particular instance, just laying out various possibilities).

Is there a thread running through that all? ... thru the terms Addon, App, Feature ,, Widget ...?

Sort of but not really. If I remember correctly (I haven't looked in a long time), the only time a thread really processes through those items is when it is necessary to display something related to them. For the most part, things are fired using the "hook" system - and once a hook is fired, if it's part of an addon/widget/app/feature, the function would check to see if the necessary installation requirements are fulfilled. So (generally) the functions are not called based on whether or not there is a table entry, but hooked functions are expected to check for themselves whether they should or should not run based on what is in those tables.

It's a little inverted from how most people think of a program as a thread where the calling function decides whether or not the called function should run, but rather, the called functions should check the state of the system to know whether or not the requirements of whether or not it should run have been met.

Sat, 25 Jan 2020 15:15:49 +0000 last edited: Sat, 25 Jan 2020 15:11:08 +0000

@Max Kostikov - Interesting about the RoundCube hooks.  (And thanks for the reminder, I have an RC setup running for 5-6 years that needs upgrading!)

I don't see too many inefficiencies in the current hook system.  In a sense, such "contextualization" of hooks already exists based on the call_hooks() system.  There are a couple of areas that can be tightened up, but I think they can be remedied easily enough by simple awareness and maybe some minor tweaks to the current system.

The biggest I've encountered is that the majority of hook data is read from the database when the app loads.  As addons create and use more hooks, that's going to be a bottle neck.  We also seem to have the occassional problem of incomplete removal or partial installation of hooks for addons (which is why we end up needing to do the whole uninstall/reinstall thing on upgrades).  Also, misspellings of hook names and such during add/remove can create strange artifacts in the database and corrupt a site (yes, I learned from personal experience!)

I've been thinking about this for a while, and I think there is a fairly simple answer:
(1) Fix hook prioritization for Hook::Insert() (Currently, hooks inserted with Hook::insert() are simply appended to the list and are not sorted by priority, prioritization is important!)
(2) Encourage addon developers to use the init_1 hook to load their hooks using Hook::insert() instead of storing them in the database with Hook::register().  To improve efficiency, we may want to consider adding a new hook specifically for loading addon hooks sometime after the channel and the observer are determined so addons can insert hooks conditionally based on whether or not their addon even need to be added for that channel/observer.

I think with those modifications, we would be about as efficient as possible.

Fri, 21 Feb 2020 06:31:57 +0000

how do we use the init_1hook?

Fri, 21 Feb 2020 13:15:29 +0000

Nah, that's perfect, thank you for the explanation.

I'm on my second addon now, (it's giving me a bit of trouble to be honest) so I'm eager to know anything to make my addons more robust, and more loved by the community.

Fri, 21 Feb 2020 13:15:45 +0000 last edited: Fri, 21 Feb 2020 13:09:51 +0000

It's a hook like any other, it just runs very early in boot.php before any real work is done.

The proposal for addon developers that I outline above is basically to only have a single Hook::register('init_1','addon/{dirname}/{filename}','{func_name}',1,{priority}); in the loader and then place all the other hooks you want to use in the registered function using Hook::insert().

In 99% of cases, the order of hooks won't matter, so most addons can implement this immediately.

But there are some cases where the order of things becomes important.  If that's the case, you must use Hook::register() at this point because Hook::insert() does not actually offer the ability to place hooks in order.

Hook::register() registers the hook in the database, so when the system initializes, the hooks are read from the database in priority order and put into an array.  Hook::insert() simply appends the inserted hooks to that array.

Another thing is that init_1 runs before the environment is set up, so at this point, addons cannot insert hooks based on whether or not the addon is installed on that particular channel or the observer would need that hook.  A better solution is to figure out the right place to insert a new hook after authentication is complete and after we parse the URL and have any other information we may want/need so that hooks can be inserted intelligently based on whether or not they will be needed.  That will improve efficiency on startup.  It will aslo reduce the need for every hooked function in an addon to check whether or not it should run, and (very minorly) save some memory as unneeded hooks won't be in the array structure.

On startup, currently ALL hooks are loaded from the database into a memory structure.  This takes time and memory to create.  Also, currently it is necessary for every function referenced in a [code]Hook::register]/code] to run checks as to whether or not the addon is installed on the channel being viewed and any other conditions for whether or not that hook should run are met.  The system simply runs through the whole array of hooks from top-to-bottom and runs the functions that are registered/inserted into the array. The hooked function must then determine whether or not it needs to run, and return before processing anything if not.  I haven't run any tests, but the plain fact is that calling unnecessary functions only to have them return is going to take SOME time.  And on a large hub with lots of addons that time logically adds up.

The other problem is that storing all the hooks in the database as is currently done in most addons increases the possibility of a certain race condition - especially on busy servers - where a given addon may be in a state of partially installed/partially uninstalled when a request comes in.  So PART of the addon gets run and part does not.  It is a fairly rare condition and doesn't show up often.  But I've run into it a few times.  Major improvements were made a while ago to avoid it, but it is still a possibility when installing/uninstalling addons or in certain conditions if a request comes in while certain maintenance tasks are going on.

It shouldn't take MUCH to fix, but since the hook system is so integral to the framework, it's delicate.   Of course, since it is so integral, usually problems show up quickly in debugging.

(More than you wanted, no doubt, but that's a fuller picture of the architecture and issues as they stand currently.  To be fair, the hook system is why I love Hubzilla/Zap.  It's very extensible.  Hub owners/admins aren't locked int and addons can significantly change (hopefully for the better!) how things operate.  It makes this platform FAR superior to many others.  But, from what is seen above, that underlying system needs just a little bit of TLC at this stage.)