content_import

Fri, 21 Jun 2019 16:28:12 +0000

I had a need to use the content_import tool and found a couple bugs that I'm working on a PR for.  But I also found a functionality issue that is outside the plugin itself.

The API endpoint .../item/export/page calls include/channel.php::channel_export_items_page() which uses resource_type='' in the where clause of the item query.  This means that WIKI items and (at least some) event items, certain portions of web pages (PDL files) and anything with resource_type set by an addon (eg., the chess addon) are not included in the results.

I'm sort of guessing that this was so that PHOTOS were excluded, since they would be "files" - but it'd be nice to get some confirmation or some other explanation.  My gut instinct is to set the query to    resource_type != 'photo' so that it catches everything.    I just want to make sure nobody knows of a good reason not to do this.

(sorry for the dupe for those that saw it.... I'm SO out of practice!)

show all 17 Comments

Mario Vavti

Fri, 21 Jun 2019 18:41:06 +0000

You sure wikis, webpages etc. aren't dealed with somewhere else?

I admit i have not looked at that addon yet :neutral_face:

Fri, 21 Jun 2019 18:48:44 +0000

No, it's very very simple/basic. Do a poll to /api/z/1.0/item/export/page, cycle through the list. If the "files" option is selected, do a poll to /api/z/1.0/files, cycle through the list. Then, call it a day.

I don't see anything else that currently uses .../export/page.

I thought about adding a query variable for what to include, but there's no way for hubA to know everything that might be on hubB. Maybe a query variable like, "fullclone=1" if you want everything (except photos since they should be handled by files)?

or make "fullclone" the default for now (I already submitted a PR to do so), and then back-off in the future and create options if there is an actual need?

Fri, 21 Jun 2019 18:53:33 +0000

I was only "tipped off" because after setting up the new clone, I didn't have my WIKI items... Even after running content_import, they weren't there. And after over a day of the new clone being up, they didn't show up either.

Fri, 21 Jun 2019 18:59:13 +0000

Also, I just noticed, the ZAP API endpoint is "api/z/1.0/item/export_page" but the Hubzilla API endpoint is "api/z/1.0/item/export/page" (both defined in include/api_zot.php). When I submitted the PR's, I just assumed that Hubzilla's api_zot.php was correct - and I didn't look at Zap's version. It doesn't really matter which we choose at this point, but we need to choose one, I think at this stage, so we don't have to support both into the future.

The help files do not have a specification for the endpoint either way.

@Mike Macgirvin

Fri, 21 Jun 2019 18:59:13 +0000

Mario Vavti

Fri, 21 Jun 2019 19:03:36 +0000

You can not clone from hubzilla to zap or vice versa.

Fri, 21 Jun 2019 19:18:19 +0000

True, but we are talking about V1 of the ZOT API  ("/api/z/1.0").

If we need to enforce the inability to clone from one ZOT platform to another, there should be something in the logic that checks whether the endpoint is Hubzilla or ZAP to prohibit cloning if necessary.

But if we aren't changing the API versioning, shouldn't the internals of the API be consistent?

I don't think you really want multiple specifications floating around.  We already have enough trouble getting people to consider Zot.  Separate API specifications for the same protocol on different platforms seems like it would be making things harder on any number of fronts.

El Dorado

Sat, 22 Jun 2019 02:03:45 +0000

wikis and pdls are exported in different export "sections" from items and are included by default in the channel_basic_export. Conversational items usually cannot be due to memory issues primarily because there are a lot more of them.

If there is a conflict between the way Hubzilla does something and the way Zap does something, as a general rule the Zap variant is the way we want to do it going forward and the other was left as-is for historical compatibility. At some point most projects have to break old compatibility because maintenance and support of 5-10 year old variants is impractical. Hubzilla still mostly supports cloning from redmatrix. With Zap we mostly started fresh when it comes to historical compatibility.

Code is already in place to prevent import to an incompatible platform.

Sat, 22 Jun 2019 11:24:26 +0000

Ok, I'll update the pull requests for the API paths to the ZAP model.

I see the code in channel_basic_export for the different sections as you say, but the content_import addon to import data over the wire instead of via files doesn't use that from what I can see. content_import hits /api/z/1.0/item/export_page which calls include/channel.php::channel_export_items_page() which only will return item.resource_type = ''. Wiki info has a resource type of 'nwiki%' and PDL resources from what I can tell are type 'pdl'.

A result, the content_import addon does not get all the items and won't get, for example, any items from the "chess" addon or other addons that use the resource_type field.

If we need to segment them by types, that's fine. But looking at the code, I saw no obvious harm in just expanding the query to include the missing types in channel_export_items_page() since from what I could see, the content_import addon was the only thing that (currently) uses it.

But if this is the wrong approach, give me a push in the right direction and I'll see what I can do.

On demand full synchronization of new channel clones over the wire would seem to me a good thing - which is what I basically thought the content_import addon was for.

El Dorado

Sun, 23 Jun 2019 02:24:57 +0000

There are two steps to the cloning process precisely because of php memory limitations. The first is to get the channel basic data. This is done at channel creation and includes everything except conversations and files. Except in rare circumstances this all fits into memory and you can choose to leave out some sections if it doesn't and fetch them separately. For conversations and files we need to fetch one item (or a low number of items) per process from the other end so that we don't whitescreen. I think I did try to do a small chunk of conversation items but files has to be one per process. Most wikis and webpages and other item types are in the Kbytes, Conversations get into tens of megabytes pretty quickly if your channel has a long history and a lot of comments.

The other consideration is that the channel basic data can be loaded from a thumbdrive if your old server is dead. You can also load backed up conversations if the old server is dead. Unfortunately we can currently only pull files from a running server. The content_import addon just combined the conversation and file import to a one step process rather than manually import one month or year at a time and have a separate process for importing files.

So it was one step in consolidation. It's probably possible now to do everything at once - but only if the old server is still running. I hadn't gotten that far figuring two steps was at least a step forward and there's really not much we can do about files if the old server is dead. That's going to take streaming json and potentially gigabytes of backup files and is a development task for another day.

So feel free to improve stuff. I don't have an issue with adding additional types to channel_export_items_page(), only that if you're cloning a channel we've already got those items from the basic export and it might be redundant.

Sun, 23 Jun 2019 16:31:42 +0000 last edited: Sun, 23 Jun 2019 22:13:30 +0000

In the present instance, I was using the "Import from an existing hub" at the channel creation step. In that case, it hadn't pulled in the PDL/Wiki & Web page stuff. I haven't tested the file based export/import procedure - but it appears that the "from online hub" does not get these things automatically - at least it didn't in this instance. It may have been configuration related on my side as I did have a problem with the "tempfile" directory initially. But that kind of makes obvious the need (or at least the desire) to have a full "synchronize channel clone(s)" tool. Content_import provides a good start - I have some ideas - but at least with the changes I'm thinking of, we'll have at least a baseline tool.

It'd be nice to have some sort of ongoing synchronization tool among clones in the event of temporary network splits and the desynchronization that might come about because of it. But I think that's a much longer term thing than is immediately needed.

Mon, 24 Jun 2019 00:54:10 +0000

@Mario Vavti   @Max Kostikov

The following MR's gets things to at least a baseline:

https://framagit.org/hubzilla/core/merge_requests/1680
https://framagit.org/hubzilla/core/merge_requests/1683
https://framagit.org/hubzilla/addons/merge_requests/268

It may be worth pushing these to Master - without 1683 and 268, the content_import does not function at all.  I think the potential for duplication of some resources that may come across from a file import is worth it at this point, so I would advocate for 1680 also.

Future work can deal with a fuller synchronization strategy that deals more properly with duplicates (and verify that it deals with revisions).  I think we should also consider making sure we include importation of conversations which have comments authored by the user even if the original post doesn't belong to them.  From a cursory review of the current status, these will not be cloned/synchronized into a new channel clone.

Mario Vavti

Mon, 24 Jun 2019 08:50:14 +0000

@M. Dent i wonder in 268 why you needed the channel reddress instead of url. In every other situation for Zot6/HTTPSig::create_sig() we use the url. reddress was used in Web/HTTPSig::create_sig() which, i think, is legacy now.

Was the channel you imported from not yet converted to zot6?

El Dorado

Mon, 24 Jun 2019 11:36:29 +0000

This is a due to a mess I created and I may need to sort it out. @M. Dent did the right thing in order to fix the immediate problem.

Mon, 24 Jun 2019 11:50:08 +0000

WHEW! --- Thanks for mentioning it. I was trying to backtrack my steps to explain what led me to do it, but I couldn't figure out the actual "why" it was necessary.

(Let me guess... it's one of those things that would take longer to explain and get someone else up to speed? From a quick look, it should be avoidable if we just update api_auth to also search for keyId as hubloc_id_url instead of just hubloc_addr. Am I missing something? hubzilla::include/api_auth.php:97 )

Mon, 24 Jun 2019 19:38:40 +0000

If updating api_auth.php:97 is the way to go, I can submit that patch then we can undo the changes in 268 and everything should be happy and (more or less) consistent. But if we will be creating a bigger mess by trying to fix a small mess, I'd rather do it the right way the first time.

El Dorado

Tue, 25 Jun 2019 02:50:57 +0000

The issue is that for 'zot' hublocs we don't have a hubloc_id_url and that's what we are looking for. We could probably look for the 'zot6' hubloc and then use channel_portable_id (the zot6 channel_hash) to link that to a specific channel instead of (zot) channel_hash. That may be the best way to go.

I was trying to accomplish this just using the 'zot' hubloc but it's quite difficult without a hubloc_id_url - which is precisely why that column was added for zot6. In zot we could do most everything by address. In zot6 most of the protocol communications use urls. The zot6 version of this auth function can use either.