Any ideas on how to sync to ressource on different hubs?

Mon, 01 Oct 2018 15:10:16 +0000

I need you thoughts and advice on possible the solutions at the end of this post?

The addon Flashcards uses a box of flashcards to learn languages. Two users "me" and "alien" are registered on different hubs. Both want to edit the content of the box and share (sync) changes forth and back.

The plan: Make a copy of a box and keep both original and copy in sync.
User "me" creates a box "boxItalian" on https//my-hub.org.
The app in the browser accesses it like:

Download GET: https//my-hub.org/flashcards/me/get/boxItalian
Upload POST: https//my-hub.org/flashcards/me/update/boxItalian

A user "alien" downloads and stores a copy "boxItalianCopy" on his hub https://alien-hub.org. Access is the same for user "alien":

Download GET: https://alien-hub.org/flashcards/alien/get/boxItalianCopy
Upload POST: https://alien-hub.org/flashcards/alien/update/boxItalianCopy

Ok, both hubs must have the addon "flashcards" installed.

Why it is better two have a copy of the box for every learner?
If user "alien" starts to learn on https//my-hub.org he has two problems:

User "alien" looses his box and his learn progress if https//my-hub.org disappears or the channel "me" on https//my-hub.org disappears.
User "alien" does not want to store his learn progress on https//my-hub.org because he wants to keep it private.

Is one of these solutions possible, or both?

The app in the browser ( https://alien-hub.org) of user "alien" uploads (POST) "boxItalianCopy" to https//my-hub.org/flashcards/me/update/boxItalian Question: Will this post use magic auth automatically or do have to use additional parameters or other things?
The hub https://alien-hub.org uploads (POST) "boxItalianCopy" to https//my-hub.org/flashcards/me/update/boxItalian What PHP function should be used for it.

What would be the preferred solution?

Do you any other ways?
In addion: How to keep the boxes in sync if user "me" clones his account (and box) and shuts down is channel (and box) at https//my-hub.org ?

#synchronisation #resources #addon #flashcards

synchronisation flashcards resources addon

show all 28 Comments

Einer von Vielen

Mon, 01 Oct 2018 15:14:15 +0000

Sorry for some incorrect spellings.

Mon, 01 Oct 2018 15:44:33 +0000

Where/how are you storing the data currently? In a separate database table? Or in pconfig? How is it structured? (I know you've answered this before, but I can't remember).

Personally, I would lean toward your 2nd solution. I think you'll run into some cross-site-scripting issues with the first one (though I could be wrong.)

Leaving aside the data propagation problem for a moment: As you have described it, there is an implicit trust that "me" wants to accept the changes made by "alien". Is that a reasonable trust? (I personally wouldn't think so, but it isn't my app) Also, is it desirous that the change sent from "alien" to "me" would then be propagated further from "me" to whoever else has a copy of "me's" box? Then you have loop prevention to deal with also.

Part of me wishes Zot had a CUSTOM message type that allowed addons to take advantage of ZOT's propagation directly. That would seem to me to be the best solution. I thought about this a couple of weeks ago. For my own purposes, I can cobble a solution by hooking into the propagation of the data of pconfig because I'm only right now interested in dealing with data sharing with channel clones. Your question is about message passing by an app between non-cloned channels and introduces some wrinkles.

I haven't asked @Mike Macgirvin about arbitrary message passing by addons yet because I wasn't ready to do anything with it. But he may already have this sort of thing in mind and have a system built into ZOT to deal with it and I just haven't run across it yet.

Einer von Vielen

Mon, 01 Oct 2018 17:04:26 +0000

Storage

Where/how are you storing the data currently? In a separate database table? Or in pconfig? How is it structured?

The box content is sent and stored as JSON.
The "meta data" of a box is stored in table item. Because the JSON (box content) can be to big it is stored in table iconfig. That's how @Mike Macgirvin proposed it. It works fine. The creation, update and deletion works perfect for me. I think it should be synched between clones automatically what I remembered from debugging.

Propagation
Do you know what PHP function I should use for the second solution (hub of "me" is sending a message to hub of "alien")? A while ago I saw some PHP function that triggerd a curl. I forget where exactly. There are so many well thought functionality implemented and ready to use. I fear that I might overlook something.

Loop prevention
I would only let the copy of the box do start the sync with the original. On both hubs the addon would merge the content of the box.

Trust

As you have described it, there is an implicit trust that "me" wants to accept the changes made by "alien". Is that a reasonable trust?

Two ideas:

I would let the user "me" set the permission with the ACL modal for the original box. The user "alien" should not have the ACL modal. He can not "re-share" the box. But, may be he could "unlink" his copy from the original. The copy would "forget" his original and sync with the original would no longer be possible. In this case "alien" make the copy to an original and will be able to set the permission with the ACL modal dialogue.
and/or the user "me" could his original box make "read only".

Custom message type

Part of me wishes Zot had a CUSTOM message type that allowed addons to take advantage of ZOT's propagation directly. That would seem to me to be the best solution.

Interesting, never thought of this. The addon would still have to hook in before the item is written into the DB because the content must be merged.

Mon, 01 Oct 2018 18:29:08 +0000

Hmmm... I'll have to take a look at how you're currently doing it. Perhaps "item" is the way to go for some of the things I'm thinking also. Not sure how soon I'll get to it, but it'll get to the list.

Thinking quickly - you should be able to do something similar to creating a "comment" and somehow putting the update in the iconfig fields. Idempotency may be something to consider in that scheme. But it shouldn't be too hard.

Einer von Vielen

Mon, 01 Oct 2018 20:16:10 +0000

I think the only part missing is that the PHP part of the addon sends the copied box as JSON to the hub that holds the original. The addon then merges the box into item/iconfig, sends the merge result back and then merge this bos with the copy.

I plan to use the PHP method fllashcards_post() as endpoint to sent the JSON to, forth and back. The browser uses this endpoint already.

I just have to figure how to send the JSON from the PHP.

Einer von Vielen

Mon, 01 Oct 2018 20:17:20 +0000

Any idea?

Mon, 01 Oct 2018 20:28:46 +0000

Creating your own endpoint like that will require dealing with authentication. The browser uses the cookie/session to do that for you, but if it is hub-to-hub, you'll need to have a mechanism to handle that. That is why I was trying to work through the existing hub-to-hub communications structures rather than "roll-your-own". Honestly, I trust Mike's ability to get it right MUCH MORE than I trust my own. (Having built authentication schemes for API's and such, it's just a head-ache to make sure you think of everything).

Einer von Vielen

Mon, 01 Oct 2018 20:55:04 +0000

I think I let the browser send the synch. So undecided... It looks much simpler. The only thing is the cross side scripting. Do you think that this is a real problem?

Hmm...

Einer von Vielen

Tue, 02 Oct 2018 10:00:41 +0000

Creating your own endpoint like that will require dealing with authentication. The browser uses the cookie/session to do that for you, but if it is hub-to-hub, you'll need to have a mechanism to handle that.

True. That's why I ask

I think we should keep the addon simple for a first step and in a first stable version

Treat the boxes like a webpage or wiki. That means a user "alien" (that is not the owner "me") can only learn on the hub that holds the box (https//my-hub.org) and stores the learning progress there only. The box does not leave the hub.
Every user can make a backup of the box manually just to make sure nothing is lost for him. This is implement yet. The box is shown as JSON string in a text field under "Menu" > "Local Backup". It is possible to import a JSON via this text field later on. This is implemented to.
Another advantage we get is, that one can be sure the addon is intalled on the hub. (If you would send a copy to the hub of user "alien" ( https://alien-hub.org) you will never know if it is installed. Ok, one could "ask" the hub what plugins are installed.)

After this we could think about to include the addon into the official addons.

A second step could be to let the user "alien" copy a box to his home hub ( https://alien-hub.org) and synch both boxes copy and original.

@Mike Macgirvin, please could you say something on the following?
Before we should check out if the addon can discover a box with a unique item ID. My thought is....

ZAP can discover a channel in a grid by using the unique ID of a channel no matter on what hub(s) it lives at the moment.
Once you have the channel it should be generally possible to get an item using the unique ID of the item.

I am not sure if this is implemented yet. It is most likely Mike has thought about it.

Tue, 02 Oct 2018 23:43:46 +0000

ZAP can discover a channel in a grid by using the unique ID of a channel no matter on what hub(s) it lives at the moment.

As long as the channel isn't top-secret you should be able to ask a directory server.

Once you have the channel it should be generally possible to get an item using the unique ID of the item.

Yeah.

Einer von Vielen

Wed, 03 Oct 2018 14:04:05 +0000

Just to sum up what the addon would have to implement to synch copied boxes. I will not implement it right now.

Copy a Box
User "me" (unique ID) stores a box "Italian-German" (unique ID) at https//my-hub.org.
User "me" can clone his channel to different hubs. The box will cloned to (rows in table item/iconfig). The unique IDs for the channel "me" and the box are always the same on every hub they are stored.
User "alien" copies the box "Italian-German" to his home hub https://alien-hub.org. The copy will get a new unique ID. The copy holds the unique ID of it's original.

Permissions
User "me" can set permissions. User "me" can change permission.
User "alien" can not change the permission (will no have the "lock" / ACL modal dialog).User alien can "un-link" his copy from the original. The copy will turn to an "original". After this user "alien" will be able to set the permissions.

Copy a Copy?
What if a third user "alien B" will click on the URL for the copy of user "alien"? If user "alien B" is allowed by the ACL (user "alien" can not change) he will see the box and can make a copy.The copied copy will point to the original from user "me". The right thing to do would be to let user "alien B" make a copy from the original only. Why? Because user "me" as owner might have changed the permissions meanwhile. This would have to use the mechanics described for the synch (discovery of a channel and then the box).

Synch Boxes
If the addon wants to synch the copied box with the original...

User "alien" or user "me" starts a synch.
The addon checks if the box is a copy or an original. If it is a copy start the synch. If it is an original stop the process. In our example the hub of user "alien" only will go on.
Find the hub (primary or clone) where user "me" "lives" at the moment. The unique ID of "me" is used for this (by ZOT/ZAP).
Check if the addon is installed on the hub where "me" "lives". If not? Hmm, iterate the clones of "me". Or finish the synch in a first implementation?
Send the copied box to the hub of "me".
The hub of "me" tries to find the original box. (The original will always have the same ID on every hub, primary, clones)
Check the permissions. User "me" could have changed them.
Merge the content of the box.
Send the merged box back to hub of user "alien".

This should always work as long as the channel "me" exists in the grid and carries the (original) box with his channel. This should work even after "me" leaves every hub and appears on a complete new different hub. This synch should even work after years provided ZOT and the addon stay compatible.

This feels magic. Basically the channel discovery makes it possible.

Einer von Vielen

Wed, 03 Oct 2018 14:13:32 +0000

@Mike Macgirvin Couldn't be a merge become a standard feature of ZOT?

Let's say an addon edits and shares a LibreOffice document, a picture, a JSON, whatever...
ZOT does the synch from hub to hub (using channel discovery and unique ID for the ressource). As part of the process an addon can be called (can hook themself in) to do something like a merge. The ressource could be anything, a picture, video, document. ZOT does not care about the merge itself and leaves it to the addon how to merge.

Einer von Vielen

Wed, 03 Oct 2018 14:30:52 +0000

Ok, one could say a resource has always ONE owner only like a wiki of webpage has. And the work (merge) is always done using the original.

Hmm, the addon Flashcards is a special case where you do want to keep your learning progress private and only want to share the language part.

But other addons might share a ressouces the same way

Share the public part of the resource
Keep the private part of the resource private (should not leave the home hub)

I can not really come up with some real live use cases. May be:

A teacher shares documents and want to make private notes in the document. Or the students make some comments, hints, that are not intended to share.
If a patient shares information with a doctor, but some parts only. He shares the heart rate and speed while running, but not the geo position.
Or if the resource is so important for a user that he wants to keep a copy at his own hub

Just thoughts.

Wed, 03 Oct 2018 14:47:25 +0000

In that case, I think a way to accomplish it is to use the xconfig table for the private data (cat='{addon_name}', k='keyname-{itemid}", with v= whatever value you want to store). If you wanted it cloned to all the users's channel clones, you could use the pconfig table instead of the xconfig table.

Einer von Vielen

Wed, 03 Oct 2018 15:43:44 +0000

@M. Dent I can't find both table is my DB, neither pconfig nor xconfig.

Einer von Vielen

Wed, 03 Oct 2018 15:48:16 +0000

Oh, hang on. I had to look at the next page at phpmyadmin. Yes, of course they are there. You think I should store a box there in xconig? The column "v" is Mediumtext. That is big enough.

Einer von Vielen

Wed, 03 Oct 2018 15:58:31 +0000

Sorry "pconfig". Hmm, thinking...

Why not? Is there a good example that uses "pconfig"? Creation, search, update,...

Wed, 03 Oct 2018 16:06:27 +0000

Most of the addons use pconfig for their configuration at some level. You can see it used extensively in the cart addon. You can also find info about the usage in Zotlabs/Lib/Pconfig.php or in include/pconfig.php (I'm not logged into a shell anywhere to check which file the functions are in).

Einer von Vielen

Wed, 03 Oct 2018 16:37:28 +0000

@M. Dent Many thanks! I will let the addon store the boxes in pconfig.

Meanwhile I wrote down some thoughts on how the addon should work: Provide two versions to share boxes. I will start to implement the first version Share boxes with other users on the same Hub Many words few code

On the bright side: It became more clear to me what I want to do.

All takes sooo long. To comfort me... I learn a lot. I use the addon every day to learn and it works. I think if the the first version Share boxes with other users on the same Hub I could publish the addon.

Einer von Vielen

Wed, 03 Oct 2018 16:39:40 +0000

Correction... I learn a lot about Hubzilla, ZOT,... And I course I learn Italian too.

Leaning is the most important aspect to me.

Einer von Vielen

Thu, 04 Oct 2018 17:54:23 +0000

@M. Dent Table "pconfig" is for storing config values (only/originally). Storing a box as a file could be an interesting alternative. ZAP will synch files between clones. An easy backup with over webDAV is possible. Any other "offline" client could use the JSON. One could even set the permissions outside the addon.

Thu, 04 Oct 2018 19:32:48 +0000

I had proposed a little while ago an APIfor plugin authors to do what I called arbitrary storage in the way you describe. I will be needing it in the not to distant future myself for some things, but it hasn't yet made it to the top of my list. Not sure when it will, but I agree, that is a definite thing to consider.

Fri, 05 Oct 2018 00:36:58 +0000

pconfig is intended for arbitrary storage of anything that is "owned" by a channel. It stores up to mediumtext on mysql which is 16M.

Einer von Vielen

Fri, 05 Oct 2018 10:08:39 +0000

Thanks for the clarification @Mike Macgirvin.

Talking to myself...

DB table pconfig or files to store a box?

If the addon uses the cloud file storage it is very likely that some users delete files or change permissions by accident or change the content to try something. This is a kind of build-in error by design.

To avoid errors is a top priority. So I go for pconfig.

Fri, 05 Oct 2018 10:15:41 +0000

File storage has the additional benefit of having built-in access control. So yes it could get modified to be unreadable, but pconfig has no access control - and by default it is only visible to the channel that owns it.

Another possibility is item storage. Just claim a new constant for the item_type so that nobody will conflict with it later. Wikis, doc pages, articles and cards are all items. There's a bit more of a learning curve and preparation, but this really is general purpose storage, and you can sync it, and it has access control.

Einer von Vielen

Fri, 05 Oct 2018 10:43:33 +0000

Three ways to store a box

table item - Is used by the addon now. Because the field "obj" is to small for box (JSON) the box is stored in two tables
- box meta data > stored in item
- box content > stored in iconfig
table pconfig - big enough, synch between clones but without build-in permissions
cloud file - has everything (permissions, synch between clones, easy to use) but might be prone to errors

@Mike Macgirvin Question on synchronization between clons: Will an entry in iconfig be synched along with it's "parent" entry in table item automatically?

Atm I update the box like this in "item" and "iconfig" like this...

    // Hint: This adds the box as iconfig to the item array.
    // It does NOT store the box in the DB.
    if(! set_iconfig($item, 'flashcards', 'box', $object, true)) {
        return array('status'  => false);;
    }
    // Hint: This updates the iconfig implicitly, more precise it first deletes
    // and then inserts the box into table iconfig
    $ret = item_store_update($item);

Will both be synched now and in the FUTURE?

Hmm, an element divided in two parts. Things can go wrong more easily.

That speaks more for storing a file in cloud storage. The freedom to decide is hard

Einer von Vielen

Fri, 05 Oct 2018 16:15:18 +0000 last edited: Fri, 05 Oct 2018 16:18:36 +0000

Sorry, a file has of course a representation in the DB (table attach). If the box is stored as a file the box is also "divided" in two parts, 1st the "meta data" (table attach) and 2nd the content (file in OS).

/update/ I think I will try cloud files. (Sorry for spamming.)

Einer von Vielen

Mon, 08 Oct 2018 14:02:29 +0000

@M. Dent See this recent commit of @Mike Macgirvin . It should be a good start for me.

file_import app/addon (f285ec85) · Commits · Mike Macgirvin / osada-addons

Framagit