Zot channel represented as ActivityPub

Königsberg, Russia, Sun, 22 Sep 2019 13:04:30 +0000 last edited: Sun, 22 Sep 2019 17:59:17 +0000

!!zotlabs.org | Hubzilla Development

I'm trying to deal with one issue with clonned channels and found one confusing thing that I can not understand.
Some non-local Zot channels are persist in the xchan table as from ActivityPub network.
Here is the sample for jeronimo@zotum.net channel from tiksi.net hub side.

root@localhost [hubzilla]> SELECT * FROM `xchan` WHERE xchan_hash = 'https://zotum.net/channel/jeronimo'\G
*************************** 1. row ***************************
          xchan_hash: https://zotum.net/channel/jeronimo
          xchan_guid: https://zotum.net/channel/jeronimo
      xchan_guid_sig:
        xchan_pubkey: -----BEGIN PUBLIC KEY-----
MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAsynbe7VTAlFqImZ8t7yP
3w6aZyps0LPOVFW5SA3HfGzrmQr00++rMjDKdZsPoylZkSXrz3/D1uvpTV2l/MiE
s0wirHIOG1Nyfr3jsBhjt8F+6YHIG0XBKCTGO10SyweZ3QUeaMFi2dpbVnuC0lw7
rvALY8772m6zJSa6dGkcmbgGMn+nOWgHaaYTiesizq2whpxHXhfYsFgOe1e2DUbF
DZkwcjviRKnyLThX2SfSKgB4ncgTn0TXDtLFnoe7fyTgulDsjZU9nq8qpaUAIR9g
1PncBCdISjdbU4kyee1qmzp5uZ2pKvE0GKljvMZemwAyoEkxQkWi6yybudrRZ5Xk
4ib/3hTiGeus4P/cfGc5sHRHxwakX3PAW5Os4BclGgWYMgi2GdTYPMGcQudGkVQr
U7yJGVGJJODxsN/tOWmQRRqqClEmomDGRGaTrZig+3eWtFQoNwaHWgWZSfUHYpv+
6G9UJNbTd0kDb/WpFqauVVr6gCU5l4Gu2QI9CGy5ViTQKe2CdIWDCgaL9eWSXFgW
cdyj7AIihRJftaKjhWDRMuHed0IQS60nChb6thilaEDhcUrtdV84TQyI1PCQBFko
3q9fVexAcHkPacHUcGaKusQZrWka1gO6OVQM4GiSrZy5YcchDSMqmPSmbstUFToe
M5cliH17G62kt1S/5X6taWcCAwEAAQ==
-----END PUBLIC KEY-----

xchan_photo_mimetype: image/jpeg
       xchan_photo_l: https://tiksi.net/photo/d4026db5-a278-4aa5-9173-7fb84106933d-4
       xchan_photo_m: https://tiksi.net/photo/d4026db5-a278-4aa5-9173-7fb84106933d-5
       xchan_photo_s: https://tiksi.net/photo/d4026db5-a278-4aa5-9173-7fb84106933d-6
          xchan_addr:
           xchan_url: https://zotum.net/channel/jeronimo
       xchan_connurl:
        xchan_follow:
      xchan_connpage:
          xchan_name: jeronimo
       xchan_network: activitypub
  xchan_instance_url:
         xchan_flags: 0
    xchan_photo_date: 2019-09-22 07:47:31
     xchan_name_date: 2019-09-22 07:47:30
        xchan_hidden: 0
        xchan_orphan: 0
      xchan_censored: 0
  xchan_selfcensored: 0
        xchan_system: 0
      xchan_pubforum: 0
       xchan_deleted: 0
1 row in set (0.00 sec)

In comment from this channel on post originated from ActivityPub network in author_xchan is https://zotum.net/channel/jeronimo instead correct value.
On the hub where channel is located (zotum.net) there are no such issues.

Don't sure if this is correct so asking here.
@Mike Macgirvin @Mario Vavti

show all 28 Comments

hEARt PhoniX

Sun, 22 Sep 2019 17:40:14 +0000

Well, I learned that you just simply can and should not use cloning when using AP or any other non-Zot protocol. These are singleton protocols, as @Mike Macgirvin calls them. If you want to change hubs you need to reconnect to all your AP etc. connections from that new hub.

Sun, 22 Sep 2019 17:47:28 +0000

@h.ear.t
Thanks, I know it.
But this issue looks related to Hubzilla procedures not in interaction between Zot and AP / Diaspora.

hEARt PhoniX

Sun, 22 Sep 2019 18:06:35 +0000

OK then we will need to wait for someone way more knowledgable than I. :)

Mario Vavti

Sun, 22 Sep 2019 19:06:57 +0000

Well, it's complicated... Basically what happens is that if we import an author via activitypub or diaspora we assume that the author uses activitypub or diaspora and we create an xchan according to that info. Also each channel has a zot and a zot6 xchan atm. We try to pick the right xchan if we need it. There were issues with picking the right xchan in the past but most of them got fixed afaik. At least i did not notice any lately...

Sun, 22 Sep 2019 19:20:48 +0000 last edited: Sun, 22 Sep 2019 19:23:00 +0000

@Mario Vavti
Thanks for explanation. And what about xchan_guid, is this correct? As I see for Diaspora we use Zot hash instead URL.
Frankly say I have started to dig this because found quite strange behaviour when commenting ActivityPub posts from cloned channels. If comment goes from channel primary location (kostikov@tiksi.net in my case) all looks fine and comment syncs with clone. But if I comment from secondary location (kostikov@zotum.net) no syncing with clone happens. I don't sure if this was here before because I never actively tested such activity before.

Sun, 22 Sep 2019 19:45:46 +0000

I have some thoughts, but would need a clearer description of the flow of messages.

As I understand the issue related in the OP:

1) Message was created by a 3rd party on ActivityPub
2) Jeronimo replied to the AP message
3) The xchan record in the OP appeared on a hub that holds a clone of channel "jeronimo"

Is that correct?

Sun, 22 Sep 2019 19:51:39 +0000 last edited: Sun, 22 Sep 2019 19:48:58 +0000

(OH... and..)

4) Both the hubs that host the 'jeronimo' channel are connected to ActivityPub.

Sun, 22 Sep 2019 19:54:26 +0000 last edited: Sun, 22 Sep 2019 19:58:27 +0000

@M. Dent 'Jeronimo' was just for the example. This is not cloned channel and there is no issues with it.
But I see the same for my channel kostikov@tiksi.net and its clone kostikov@zotum.net where I have issues mentioned above so I thought that this can be related.
I don't sure when exactly AP xchan record appeared.

Sun, 22 Sep 2019 19:59:57 +0000

@M. Dent

4) Both the hubs that host the 'jeronimo' channel are connected to ActivityPub.

Yes, both my cloned channels are connected to AP channel (kostikov@bsd.network).

Sun, 22 Sep 2019 20:57:52 +0000 last edited: Sun, 22 Sep 2019 20:55:12 +0000

Ok, here's my theory - I'm not currently using AP, so I'm not sure the validity.

The xchan was fed in through AP - possibly under the following circumstance.

* An original message was posted to AP.
* The channel in question replied to the AP post from one hub (call it HUB-A).
* The message made it to the AP node and was sent to the cloned hub (HUB-B) before the sync message was received from (HUB-A).
* (HUB-B) encoded an xchan record as sourced from "activitypub" for the AUTHOR because that is where it got the information.

If the SYNC message from (HUB-A) had arrived at (HUB-B) before the activitypub message, the AP message would likely have been ignored. But, if the SYNC message was delayed and the AP message got to the second hub first, this seems (to me) the most likely explanation.

Again, I don't do much with AP (at least not at this point, not sure I'll ever get there)... but based on what I've studied of the HZ internals, that seems the most plausible explanation.

Sun, 22 Sep 2019 21:00:35 +0000

@M. Dent
Sounds pretty believable for me.

Thu, 26 Sep 2019 12:43:14 +0000 last edited: Thu, 26 Sep 2019 12:44:26 +0000

Ok, it seems I got what happens there.
1. As I understand at this moment we send comments on AP posts to host where from original post went only. Mastodon (I don't sure about other AP networks) do not send back those comments to comment author host so no other connections of original post author see those comments on source hub.
2. If comment on AP post send from cloned channel secondary location it will be supressed at primary location because delivery via channel sync expected but for some reason it not happens. In opposite direction all works as expected.

As fix I propose to deliver comments to local subscribers of original AP post localy via Zot and to disable check described in 2. Dear collegues, please check MR. I hope I understood correctly how things works there.

https://framagit.org/hubzilla/addons/merge_requests/290

Thu, 26 Sep 2019 13:44:36 +0000 last edited: Thu, 26 Sep 2019 13:41:49 +0000

I looked it over - I don't use AP at this point, so I have no way to test and so would prefer someone else to test and merge it.

The one thing I notice is that the changes in notifier are not at all restricted to the activity type. I honestly haven't looked at the rest of the logic close enough to know - but do you want to somehow restrict which activities get forwarded on (eg, "Create/Update" only, or "Create/Update/Like/Unlike" or whatever?) I guess the question is, of all the different $AS->type packets that could be processed, are there any that you would NOT want to forward on or that would cause loops or whatever?

Thu, 26 Sep 2019 14:33:52 +0000

@M. Dent
Proposed changes affect delivery to original AP post author subscribers on hub where comment was made (and their cloned channels via sync of course) so I believe that additional checks are not necessary. But maybe I'm wrong.

Mario Vavti

Sat, 28 Sep 2019 10:38:42 +0000

@Max Kostikov i merged this for testing: https://framagit.org/hubzilla/addons/commit/cec1aa84187f62a734ebf892866453c1c31508b2

There will possibly be issues with this approach, but let's see how it works for a while. What i can see so far, this won't work for private items as it is now. Also i am not very sure that this is the right approach to fix your issue...

Sat, 28 Sep 2019 11:06:13 +0000

@Mario Vavti

Also i am not very sure that this is the right approach to fix your issue

Me too so I'm asking

As I see it fixes those issues but I'm not sure about private items.
Anyway if there is a better way to solve this I will be happy to see it in MR.

Sat, 28 Sep 2019 11:52:13 +0000 last edited: Sat, 28 Sep 2019 11:52:38 +0000

BTW I found also an issue with cloned channels sync while answering on direct mail message from Mastodon. Both channels receives incoming message but on answering there is no sync between them and each clone see only own answers.

Sat, 28 Sep 2019 16:50:39 +0000

This fixes issue with private items sync with clones
https://framagit.org/hubzilla/core/merge_requests/1739
@Mario Vavti @M. Dent

Mario Vavti

Sun, 29 Sep 2019 05:51:00 +0000

Anyway if there is a better way to solve this I will be happy to see it in MR.

IMO we should remove clone syncing of activitypub/diaspora content.

Sun, 29 Sep 2019 09:42:29 +0000 last edited: Sun, 29 Sep 2019 09:45:57 +0000

@Mario Vavti

IMO we should remove clone syncing of activitypub/diaspora content.

Yes, I understand your point. This can confuse user if, for example, one channel is connected to remote AP / D* channel and other is not and if he will answer to post went from those networks from location which is not connected to same channel. As result no comments will not be delivered outside of Hubzilla but cloned channel will see these comments.
May be I need to modify code in same place where the changes in MR 1739 are suggested to check a source of the thread top message before sync and to check if this contact is connected on cloned channel side before message save.

Mario Vavti

Sun, 29 Sep 2019 09:46:23 +0000

Also it will break as soon as the primary location will be down...

Sun, 29 Sep 2019 09:54:13 +0000 last edited: Sun, 29 Sep 2019 13:18:11 +0000

@Mario Vavti

Also it will break as soon as the primary location will be down...

Yes, but if connection is only exist for primary location.
At this moment we can connect each cloned channel location to AP / D* separately and as result all clones can receive incoming posts and have ability to answer on them.

Sun, 29 Sep 2019 13:14:26 +0000 last edited: Sun, 29 Sep 2019 13:27:39 +0000

May be something like this.
https://framagit.org/hubzilla/core/merge_requests/1739/diffs#diff-content-18d6ebb91108ace17b93cb34093aab074518fb73
@Mario Vavti

Mario Vavti

Mon, 30 Sep 2019 10:00:39 +0000 last edited: Mon, 30 Sep 2019 10:03:28 +0000

I don't quite understand this change: https://framagit.org/hubzilla/core/merge_requests/1739/diffs#9e348a07aba9da8a53da92597456efd075ae1bb6_1242_1235

Why parents or private comment items only?

Mon, 30 Sep 2019 10:26:45 +0000 last edited: Mon, 30 Sep 2019 10:34:09 +0000

@Mario Vavti

Why parents or private items only?

The delivery logic, as I understood while testing scenarios with different protocols looks like that.
If we (cloned channel) are an top post author we need to sync it between clones because there is no other source to receive it.
If thread top post is from AP / D* we no need the sync between channels because we will get comment reflection via those networks in clones if they are connected to the top post author. Direct messages (private items) deliveried to initial post author only and if initial message went from AP network it not forwarded to other channel locations (because AP knows nothing about that) so we need to force sync with clones.
In case if top post went from Zot I don't sure how things works but I see no issues with content replication between clones.

Mario Vavti

Mon, 30 Sep 2019 10:55:22 +0000

Ok...

Mario Vavti

Mon, 30 Sep 2019 11:01:31 +0000

https://framagit.org/hubzilla/addons/blob/dev/pubcrawl/pubcrawl.php#L418

This code was initially used for repeats only, hence it returns on private items. Is this related to what you try to solve?

Mon, 30 Sep 2019 11:27:08 +0000

@Mario Vavti

This code was initially used for repeats only, hence it returns on private items. Is this related to what you try to solve?

Yes, I saw this but I'm no sure if just removing will be a better solution because of, firstly, as I undrestand, this prevent private items to being shared with other channels and, secondly, I believe will be better to leave Zot related interchange in Hubzilla core code and the right way to sync private items between clones is to send sync packet.
But perhaps I'm wrong here so don't hesitate to tell.