Zotlabs|Hubzilla Development

Sat, 05 Jan 2019 17:06:31 +0000

After spending quite a number of hours yesterday tracking down a race condition in the experimental queue worker, I've decided I'm going to need to make a new table for worker queue processing. I'm unhappy about this, but I see no way around the situation. As a proof of concept, the use of the config table is fine. As a permanent solution it is not.

If you are using the experimental queue processor in production, one workaround is to put

sleep (5);

at the top of the Process function in Zotlabs/Daemon/Master.php.

As a couple of my instances NEED this functionality, it is a priority and a more robust solution is in the works sooner rather than later. At the same time, I will likely be moving it to a separately installable addon. Hooks are already loaded by this point in processing so there shouldn't be a huge performance hit. If a site is using OPCache (which they ought to be if they think they need this feature), loading the addon file should be nearly trivial - so moving to an addon should not make much of a difference.

!!Hubzilla Development

Mario Vavti

Sat, 05 Jan 2019 13:28:06 +0000 last edited: Sat, 05 Jan 2019 13:33:59 +0000

Are there any regex experts out there?

I am looking for a way to determine if there is text between <img> tags or respectively text between images which are wrapped in <a> tags.

Eventually if there are more than X images in a row, they sould get wrapped in a new <div> tag.

!!Hubzilla Development

show all 5 Comments

Mario Vavti

Sat, 05 Jan 2019 18:41:39 +0000 last edited: Sat, 05 Jan 2019 18:42:14 +0000

Here is an example:

This is some text in a post
<img src="some_image.jpg">
Some more text
<img src="some_image.jpg">
<img src="some_image.jpg">
<img src="some_image.jpg">

I would like to detect that there is no text between the last 3 images. Then wrap a div around so that we can create a nice view of the 3 images (like in photos).

Probably that is insane though since the poster might have wanted it that way...

Maybe restrict this to posts where there are only images (no text). Would be much easier to accomplish as well and probably fibnally makes more sense...

Alexandre Hannud Abdo

Sat, 05 Jan 2019 22:17:01 +0000 last edited: Sat, 05 Jan 2019 22:14:34 +0000

Hmm. I don't see why would we should transform the output when somebody posts a series of images within a post... do people expect that from other networks? Alternatively, if we want a gallery built from an ad hoc selection of images, we could simply provide a [zgallery] tag or something like that which would consume a list of addresses, then could be automatically applied when creating image posts from multiple image uploads.

Mario Vavti

Sun, 06 Jan 2019 09:27:33 +0000

Custom tags don't federate. We need to do that when rendering the content.

Anyway, it would be part of the gallery addon if at all...

gallery 0.4

Mario Vavti

Sat, 05 Jan 2019 12:47:18 +0000

I have opened a merge request to update the gallery app to version 0.4

This implements the gallery addon for the conversation modules (network, channel, pubstream, etc)
Images in a post or comment larger than 300px width and height can be opened in the gallery on click.

I have spent much time over how to deal with linked images. Eventually i decided to also open those in the gallery on click and provide the link from inside the gallery.

Here are the merge requests:
https://framagit.org/hubzilla/core/merge_requests/1459
https://framagit.org/hubzilla/addons/merge_requests/178

!!Hubzilla Development !!Hubzilla Support Forum

Alexandre Hannud Abdo

Sat, 05 Jan 2019 16:49:43 +0000 last edited: Sat, 05 Jan 2019 16:47:13 +0000

Sounds great !

muppeth

Sun, 06 Jan 2019 19:07:20 +0000

hmmm... Why there are two seperate apps Gallery and Photos? Is it an issue with my instance perhaps? Gallery seems to do very little (can diplay some of the pictures) while Photos can display everything, comment, and post new.

Mario Vavti

Sun, 06 Jan 2019 19:09:32 +0000

@muppeth gallery just shows the latest photo of an album. If you click the photo you can zap the album.

FB Image Album Importer

Thu, 03 Jan 2019 03:37:29 +0000

I'm done with a first pass at an import tool for FB archive files for Hubzilla (should also work on Osada - but is yet untested).

Currently imports only photo albums but it's a start.

Images are stored as viewable by "only me" and stored in directories based on the albums used on FB so that permissions can easily be changed as desired after importing them.

Importation is done in the background and parallelized and, if in use, will controllable through the experimental queue worker system.

Will probably initially release it as part of the dm42-hubzilla addons for testing before this weekend. Needs a little UI work, but not much.

The process is pretty simple:
(1) Download FB archive (either the whole thing or just images) in JSON format
(2) Upload the ZIP file to your file space on Hubzilla
(3) Go to the /fbimport endpoint - it will present you with a list of all the .zip files in your file space.
(4) Select your FB archive, click "import". Images will be imported into /FBImport/photoalbums/{album_name}/{image}

In the process - found a couple of minor issues with the pconfig timestamps and the experimental queue worker that need to be addressed. Neither serious - mostly just duplicated work. The worst is a potential refusal to delete a pconfig variable because the timestamp is not in the past (yet). I'll be addressing them in the next few days (on another thread it's noted that the experimental worker queue needs a minor overhaul anyway)

!!Hubzilla Development
!!Hubzilla Support Forum

poVoq

Wed, 09 Jan 2019 03:50:53 +0000

Facebook migration tools are actually a good idea. If only there was a better way to get people off that horrible WhatsApp messenger. For now I guess XMPP with Transwhat has to do :(

Wed, 09 Jan 2019 07:55:40 +0000

Great idea! Will you be able to import (public?) posts in the future, too?

Wed, 09 Jan 2019 20:42:56 +0000

Will you be able to import (public?) posts in the future, too?

MAYBE. The posts themselves are relatively easy. I don't have a reference archive in front of me, but I don't think it will be possible (or necessarily desirable given GDPR and other laws) to import the whole conversation (right to be forgotten and all that rot).

Mario Vavti

Wed, 02 Jan 2019 19:18:52 +0000

@M. Dent i had to revert your subquery proposal due to issues with limit in a subquery.
See here: https://framagit.org/hubzilla/core/merge_requests/1451#note_349691

!!Hubzilla Development

show all 7 Comments

Wed, 02 Jan 2019 19:38:45 +0000

Grrr.... odd I'm not running into that problem (or, I guess maybe I am and don't know it?!)

OK, back to the drawing board. The whole point to the query structure is to avoid a race condition with 2 processes "claiming" the same workitem.

I guess I'll need to rewrite with explicit table locks. I don't like that solution, but I see no other way.

Wed, 02 Jan 2019 20:03:17 +0000

If anyone here knows the intricacies of MYSQL/INNODB row locking - please pipe up.

The situation is, I want to update 1 (and only 1) of a query that could return any number of results. But I want to do so atomically - so that no other process will simultaneously try to "grab" the same record before it is updated.

Here's the code I need to run - but again, exclusively and atomically, so that the id returned in the 1st query will not be returned by another process if it happens to run the same select before the 2nd query executes.

select id from config where cat='workqueue' limit 1;
update config set cat='workqueue:{uniqid}' where id={id};

Ideally - the solution will work the same way in MySQL/MariaDB as well as PostgreSQL.

Wed, 02 Jan 2019 20:26:37 +0000

A scan of MYSQL docs indicates I may be able to use the FOR UPDATE in combination with SKIP LOCKED. I see that it first became available in PGSQL 9.2 - so all currently supported versions of PGSQL have this feature.

MYSQL 5.x does not support skip locked. From what I can tell, this feature is not available until MYSQL 8.x. MYSQL 5.7 goes EOL in October 2020 with extended support until 2023.

We could just say that the "experimental" queue worker is only available for certain DB versions (it'd be good to have a check in the code itself). Or - take it out of core and find a way to make it an addon and separately installed (which would allow us to put more warnings and restrictions related to system requirements - and as part of the install - check for those things)

As usual - thoughts welcome!

Sat, 05 Jan 2019 17:09:59 +0000

UPDATE: this thread for further information and progress.

Mario Vavti

Wed, 09 Jan 2019 06:53:48 +0000

@M. Dent i don't have permissions for that thread...

Giacomino

Wed, 09 Jan 2019 18:36:01 +0000

ot
also Mario comments-post have delay, hours-delay in this case ....

Wed, 09 Jan 2019 20:50:50 +0000

Delay was my fault (my hub's delivery processes got messed up). As for the proper reference - oops sorry - forgot about the privacy setting - the conversation is rendered moot by the new queueworker implementation. Also - the race condition wasn't actually in the queueworker - it was in the fbimporter script itself (which is what I was working on when I discovered it). There is still a theoretical possibility of a race condition in the original queueworker - but after the rewrite and after fixing the fbimporter, I removed all the mechanisms to prevent a race and tried as best I could to run it through its paces and couldn't cause it to mess up. So, I think the new implementation with the race prevention code should be as close to bullet proof as possible.

Limited distribution - but not private?

Wed, 02 Jan 2019 17:52:07 +0000

I was thinking through some of the pitfalls of modern "social media" and began to consider the "all or nothingness" of the medium.

I really like the "privacy group" feature of Hubzilla - both for organizing contacts as well as holding somewhat private conversations. But I almost wish there was some sort of "in-between". Something like a PUBLIC post that isn't sent to my entire friends list, but is intentionally only shared with a subset.

Kind of like being in a group of friends who meet at the bar and talk - others can over hear - but they aren't seated at the table and aren't forced to hear.

The idea is that I don't care if others see and interact with the post (so I don't mind that it's "public"), but for various reasons, I may not want to shove it in the face of some of my contacts. If they "happen across" it somehow, I don't have a problem with them or anyone else interacting with it, I just don't want to force them to do so. I want it to appear on my "public" channel - but I don't want it to be automatically sent to my entire friends list.

As it is, I see only 2 options - fully public - or fully private. Is a "middle ground" possible?

Perhaps there is a way to do this already that I'm not seeing or perhaps the conversation has been had and I am unaware. Thoughts?

!!Hubzilla Development
!!Hubzilla Support Forum

show all 12 Comments

Mario Vavti

Wed, 09 Jan 2019 10:04:11 +0000

Something that goes in this direction is the nofed addon...

Giacomino

Wed, 09 Jan 2019 13:43:25 +0000

OT
Your posts and comments are coming with "days late"

. Does it only happen to me?

Aksel Kiesling @Hz

Wed, 09 Jan 2019 13:52:45 +0000

Like that thought/idea.

And post is 7 days late.

Giacomino

Wed, 09 Jan 2019 14:02:20 +0000

I want it to appear on my "public" channel - but I don't want it to be automatically sent to my entire friends list.

that's the point.

well, for what i know , i don't think you can do it in this moment....

❦ CRIME LICH ❦

Wed, 09 Jan 2019 15:22:45 +0000

i use a forum for that

Wed, 09 Jan 2019 16:22:08 +0000

Re: lateness - yeah - there was a problem on my hub and messages weren't being sent properly.

As to the OP:

I think I've figured out how I'm going to approach this. It will be an addon. The thought is to have a switch in the PRIVACY GROUP to indicate whether the members in that group should be sent public posts. The option will default to ON (yes - send public messages to members of this privacy group). Since the posts are public - they will be viewable by anyone - but if a connection ONLY appears in a privacy group where the option is OFF, they will not be sent a copy of the original message. If they are tagged or happen across the message and comment, there is already logic that will add them to the list of recipients of future updates.

I'm running into a UI problem as I think about it, though. Namely, what if I want to somehow add a group that wouldn't normally receive public posts, but I want to include them specifically on a specific post. It might be possible with @tagging, but I haven't gotten that far in my research yet. Currently my thinking is, it will not be possible - which is not really my preferred answer - But the problem isn't really TECHNICAL (it is definitely possible to do), I just don't know how to present it in a non-confusing way to users.

Wed, 09 Jan 2019 17:09:07 +0000

A forum works for some circumstances - but because of the way the social graph is built, it doesn't work in every case. I have quite a number of people who have connected with me because of my dev work on Hubzilla who certainly aren't interested in receiving messages I send as a pastor. I can set up different channels for different personal contexts and for such widely different contexts, that's what I do.

But even within each of those channels, it isn't that I always want to keep the conversation "private" so others don't know it's going on - rather - I don't want to target them specifically. If they find out and become interested, I have no problem with them seeing the message.

An example might be, if I want a group that targets the people who have commit access to the repository with a note about some updates - more or less to let them know they are there. There's nothing inherently "private" about that conversation - and to maintain openness of development, I have no desire to prohibit others from knowing I've done it. It may even be a bugfix for something I know several people are having trouble with, so I want to include them on the conversation. But most people don't care and don't need to know about it. So why does it need to show up on the feed of someone who doesn't run an instance and wouldn't know a git pull from a utility pole.

I guess I want a way to respect the timelines of the people my message reaches and only send them messages I think they'll care about, but I don't want to (necessarily) restrict them from seeing other messages nor do I want to make them sign up to a bunch of different channels in order to have access to everything they might be interested in that I talk about.

FB does this to a greater or lesser extent - but we all know they manipulate the feed for their own eyeball-entrapping, advertising selling reasons (or to promote their own political agenda - depending on who you talk with).

But as far as I can tell, right now Hubzilla and its relatives force us to SHARD ourselves - or to keep things either in a strictly public space or private space. There is no concept of "shared space" with built in ways to respect the fact that the space is shared and so only target a subset of contacts while still remaining out in the open and public.

For example, a pub/bar is a shared space. You don't go into a and find everybody standing on their chairs shouting so that everyone in the place must listen to their conversation (ok, some people do that, but not most people, LOL). Rather, people sit in groups - mostly intending their statements to be heard by those seated nearby - but also knowing (or even desiring) that they may be overheard by others who will join the conversation because they are interested.

Maybe one of the people in the original group tags Johnny saying, "Hey, Johnny, come here - this conversation will interest you!." And then Fred hears that Johnny was invited to the conversation and tags along. Or, maybe Johnny happens to be ordering a drink at the bar and overhears the topic and decides to chime in.

But if Johnny and Frank are just on the other side of the bar catching a cold one after a day at work - even though we're in the same room, there's no reason I need to speak in such a way that Johnny is forced to hear and process the conversation I'm having with the group I'm in.

This is the type of thing I'd like to create. Something that promotes the way that people naturally communicate in shared spaces - Not forcing people to engage or even see/hear me unless I think they're interested, but at the same time, not having the conversation in a PRIVATE space where ONLY those who began in the conversation can ever participate unless a new conversation is created.

Wed, 09 Jan 2019 19:25:06 +0000

Just wanted to interject that forums were designed as a shared space. I also envisioned sourced channels as a way to implement something like this. It's a mechanism for creating derivative streams/channels. The major difference between this and what you are discussing is that such mechanisms use a different channel and you're trying to achieve the same kinds of results using one channel.

Anyway, yes - a limited group is probably the best mechanism for what you propose. But it will require a fair bit of work to get around our assumption that privacy groups are private. I'm thinking the most obvious implementation might be to toggle the ACL selector into a delivery selector - deliver only to these channels/groups. The first hurdle you'll encounter is how to store this information so as not to conflict with the ACL. This is solvable but possibly messy since we didn't plan for it. The next issue you'll find is that Zot delivery (and Diaspora iirc) sends public posts via a different mechanism than private posts so the delivery logic will need to modified accordingly.

Then adding people to the original conversation will raise its head. We simply have no existing model for doing this. Perhaps when creating the deliver-to list one can scan the conversation and add anybody who reacted but wasn't included originally.

Wed, 09 Jan 2019 20:19:52 +0000 last edited: Wed, 09 Jan 2019 20:25:49 +0000

@Mike Macgirvin - My implementation thought was to create a filter/hook in include/items.php::collect_recipients() for items that have no allow/deny cid/gid that would be called before the existing select statement. The existing select just grabs all the entries from abook and creates the recipient list from that (filtering for remote network public policy settings). Basically the idea is, If the public_item_recipient_filter (or whatever the name ends up) returns a recipient list, skip the existing logic and use the provided list otherwise, use the default logic.

collect_recipieints already adds any authors of other posts in the thread - so I think reactions will get included by default in this method. My only question is about mention tags... I haven't fully traced that down yet.

The info will probably need to be stored in pconfig - since iconfig/sharable get's sent with the item to recipients and not just clones - but pconfig will only go to clones. Probably a json list of gid/cid info originally included.

In terms of the ACL selector - one thought I had was a toggle "mark post as public/private" switch - and then the form either sent back allow/deny cid/gid info (if "private") or sendto cid/gid info (if public). And then, maybe a toggle in the settings of each privacy group that allows the user to choose whether or not to "Send public posts to this group by default" - that would end up the default entries selected in the the sendto field for public posts.

I'm thinking by using a filter in collect_recipients() I'll avoid needing to mess with delivery logic myself.

I've been thinking about this for a week or so - and I haven't found anything that's a "gotcha" to this approach - and it actually seems (I say again SEEMS) pretty straight forward and non-intrusive. But, of course, you know where the dragon dens are better than I.

Mario Vavti

Wed, 09 Jan 2019 20:32:36 +0000

I think this will be hard to implement in the UI (without getting a lot of too complicated complains).

Wed, 09 Jan 2019 20:34:31 +0000

*THAT* is actually my biggest concern.

Wed, 09 Jan 2019 20:37:39 +0000

It's also why I think it should be an addon rather than an out-of-the-box type thing. The existing split between privacy groups and security settings still catches me up mentally (being somewhat of a techie and working with this for 18 months or so!). Adding a 3rd layer (distribution vs. private conversation) to the mix out the gate is just asking for trouble.

That said, what do you think of the "public/private" message toggle and the shared use selector?

MR 1451 for SQL syntax in "experimental queue processing"

Mon, 31 Dec 2018 17:45:17 +0000

This was noted in another thread a day or so ago - but I think it may have gotten lost as things moved down the page.

If you use the EXPERIMENTAL worker queue which helps even out usage spikes, the SQL as originally written causes a problem with POSTGRESQL. The MySQL parser seems to handle it OK.

I've submitte MR 1451 with a rewrite of the problematic SQL queries using a subselect rather than a raw update .... limit 1.

Here is the Merge Request - since I don't have a POSTGRES install to test with - it'd be helpful if someone who does could check and make sure it doesn't choke like the old version does.

https://framagit.org/hubzilla/core/merge_requests/1451

!!Hubzilla Development
@Mario Vavti
@Max Kostikov

poVoq

Sun, 30 Dec 2018 07:05:45 +0000

!Hubzilla Development
#Matrix (a federated chat system similar to XMPP) is apparently looking into #nomadic identities for 2019:

https://youtu.be/thT8B_AHwVg

In the end he is asking for proposals how to implement it... I guess he is unaware of #Zot6? Might be a good idea for someone from the Hubzilla dev team to get in touch?

poVoq

Sun, 30 Dec 2018 06:46:17 +0000

!Hubzilla Development
Now that #OpenID-Connect #OAuth2 is supported, I have been looking into better ways to link accounts to a #XMPP server (I have it working with a multi step work around using Nextcloud, but that is far from ideal). So this looks like an option:
https://github.com/rwth-acis/prosody-auth-OIDC

However, I currently nither have #Prosody installed on my server, nor can I judge if this plugin still works with the latest version of Prosody.
I hope there are others interested is this that could help with testing this? Or are there other options I am unaware off?

poVoq

Sun, 30 Dec 2018 06:50:39 +0000

https://modules.prosody.im/mod_auth_oauthbearer.html

Might also work and seems to be from the same author as the #XMPP software that is integrated in the Hubzilla app.

An(arch)druid Wraidd

Tue, 08 Jan 2019 16:17:21 +0000

Why would you not use a directory service for this? This is the kind of thing LDAP is for.

Andrew

Fri, 28 Dec 2018 12:48:22 +0000

The landing page for Hubzilla ⟩⟩ Start and the Got Zot? intro article are broken. It looks like the CSS files are not getting loaded.

https://start.hubzilla.org/page/admin/start

Mario Vavti

Sat, 29 Dec 2018 08:04:56 +0000

@h.ear.t | tobias?

hEARt PhoniX

Sun, 30 Dec 2018 12:44:10 +0000

I am having a look. Have not been logged in as admin for aaages as my other accounts are admin too. Therefore I have not touched the webpage.

hEARt PhoniX

Sun, 30 Dec 2018 14:01:43 +0000

OK, the layout was switched to none-layout. I Switched it back to simple-article-layout.
I had not logged in as admin for aaages (as you know I do have my own account promoted to admin), so this must have happened through an update.

Zotlabs|Hubzilla Development

hubzilla-development@zotlabs.org