New MR's - static files - and queueworker updates

Mon, 15 Jul 2019 05:31:44 +0000

The following MR's have been submitted:

QUEUEWORKER:

1) https://framagit.org/hubzilla/addons/merge_requests/270
2) https://framagit.org/hubzilla/core/merge_requests/1692

A number of fixes:

Add Summon() hook/handler

Fix process management count

a) Delay adding a new queueworker process slightly to make sure database is updated (avoid race)

b) Add processes dynamically until max process count is reached (previously, the queueworker would prevent new processes if there were too many, but it wouldn't add processes if there was a lot of work to do. Now, as long as there is work to do, the queueworker will kick off new processes until the maximum number of workers is reached)

With these updates, I think we are very much approaching a stable working queueworker. This version has been stress tested using a mass-photo import tool that imports photos from a ZIP archive. With the new Summon() hook and the more robust process count management, things run consistently and reliably and avoid system crashes and fires off the background processes without timing out. Basically, things can be completely DUMPED into the queue rapidly without any need for internal timeouts in front-end processes which would be required otherwise.

This should be helpful for other tools such as content importing and copying files/data from other sites which can suffer from web server timeout issues.

SERVE STATIC FILES

https://framagit.org/hubzilla/core/merge_requests/1692

if the query variable q= is fed the location of a file with the following extensions and that file exists, serve it directly from the file system:

jpg,jpeg,gif,png,ico,css,js,htm,html,map,ttf,woff,woff2,svg

In most cases, the web server should be configured to intercept these files and a request should never actually get to us. In some specialized circumstances, there may be a need to have the application handle these files.

Note: Because they are static files, they are set to allow caching under the same terms as the published web server configuration recommendations.

!!zotlabs.org | Hubzilla Development

show all 31 Comments

Mario Vavti

Mon, 15 Jul 2019 08:28:42 +0000

It seems in its current state the queueworker addon does not deliver anything...

Mario Vavti

Mon, 15 Jul 2019 10:44:27 +0000

I stand corrected... It does deliver but with quite some delay allthough the queue is almost empty...

Mon, 15 Jul 2019 11:58:51 +0000

It may be because of the updates to process limiting. We may need to boost the default minimum number of threads since previously it would exceed the limit in some circumstances.

Can you try setting maxworkers to 6 or 8 and see if it is more immediate?

Mario Vavti

Mon, 15 Jul 2019 13:08:43 +0000

It seems to me that nothing will be delivered immediately anymore even if the queue is empty...

Mon, 15 Jul 2019 13:09:27 +0000

A quick guess is that this behavior is related to the "race" condition that allowed the process limits to be exceeded when a large number of items were summoned virtually at the same time.  With these updates, each DELIVER now stands in line to be processed in order with only maxqueueworkers working on the line no matter what.  Previously, because of how quickly the DELIVER requests were sent to summon(), it was possible that they would be kicked off without counting against the maximum number of workers.

One of the next things is adding prioritization to the work queue.  Perhaps part of that needs to be some sort of "override maxqueueworkers" priority that allows certain tasks (like "Deliver") bypass the work queue entirely.  I don't like that solution, but it may be a first step.

Another option is somehow have the queueworker monitor deliveries and reprioritize the delivery requests and try to deliver to the nodes that respond fastest first - or just check the queue and if there is a message in the queue already for that node, skip the attempt or at least requeue it with a much lower priority for processing.  That way non-responsive or slow nodes won't hold up delivery to nodes that are "healthier".

Any thoughts on the above proposals definitely welcome as I haven't settled on an approach (in fact, this message itself is the "evolution" of thought on the matter of prioritization.  My initial thought was to prioritize based only on the task (standard system processes have a higher priority than, for example, "ADDON" background processes.  But the slow delivery issue you identified causes me to think there needs to be some further nuance to the approach.

One thing to keep in mind is that the queueworker addon is basically a mechanism to create a queue to handle background processes and provide a mechanism to limit the total number of php processes for the application.  Without it, basically background processing is an "all-you-can-eat" buffet.  Now we're limiting the number of processes to extremely low default value (4).  So in some ways some of these things may feel like going from Gigabit Ethernet down to 28.8 modem speeds.

Because of some of the other work I'm doing (eg., the FBImport tool), and how badly some background processes can crush a machine, I'm hesitant to increase the default and instead would prefer to optimize a bit.  We haven't (yet) run into many conditions where the load brings a machine to its knees, but I actually have a number of them on my long-range planning board that have that potential - or could simply result in a problem based on the sheer number of backgrounded processes which may execute quickly, but would overwhelm based on quantity alone (DELIVER is a good example of this type of process).  "Throw more hardware at the problem" is one way to deal with it, of course, but I prefer smart use of existing resources first - thus - a queue to even out the workload.

Mario Vavti

Mon, 15 Jul 2019 13:10:05 +0000

I can do some more test with higher maxworker settings tomorrow...

Mon, 15 Jul 2019 13:16:38 +0000

I just realized, there are also a couple of internal "settling" timers that probably should be tweaked downward as well. Currently each queueworker process will dequeue and process an item once every 2 seconds. With a large list of recipients and a small number of workers, that could be problematic. Those should probably be adjusted down to at most a tenth of a second. I'll submit an MR to do that.

Mario Vavti

Mon, 15 Jul 2019 13:22:53 +0000

I just did a quick test and no matter how many queueworkers, delivery will always only start at the next cron run (when Zotlabs/Daemon/Master.php is called).

Mon, 15 Jul 2019 17:59:57 +0000

Can you outline your test methodology? Just so I can see it and try to trace it down. Usually I can mentally walk through and make an educated guess where to start looking. In this case, I don't see in my mind where to start looking.

Also, I submitted another MR to reduce the timing between tasks so that (when it's working properly) it won't introduce unnecessary delay - but it also won't be subject to the race condition I ran into in my testing.

Mario Vavti

Mon, 15 Jul 2019 18:17:29 +0000

What i did was to wait for the poller to run. Then make a post. The post was always only delivered when the poller ran next time.

Mon, 15 Jul 2019 21:40:48 +0000

I think I found it and fixed it in the latest MR (queueworker-improvements2) -- When the task got passed from Summon to Release, it was marked as a duplicate and ignored, so Release never kicked off any queue processors. Since the CRON bypasses Summon and just kicks off Release, it picked it up and worked from there.

Mario Vavti

Tue, 16 Jul 2019 17:15:31 +0000

Unfortunately i am still seeing the same behaviour...

Tue, 16 Jul 2019 18:07:09 +0000

Are you sure you're not exceeding some other limit that immediately queues things? I just threw the code onto another box to test and I'm getting both the "Summon handled by hook" and "Release handeled by hook" messages immediately. Is, perhaps, your load average somehow high or is your message queue larger than your "Queue Threshhold" setting? Or, possibly, were the latest revisions not actually running on the box you're testing on (I only ask because I've been absolutely certain I was looking at box A when testing only to find out I was looking at box B). Also, I'm assuming you have applied the patches to /Zotlabs/Daemon/Master.php also.

I'm just grasping at straws to figure out what it could be. I absolutely detest giving the "works for me" response, but in this case, "works for me". BUT.... there is one likely difference ---> I applied the patches and tested on a box running MASTER and not DEV. I mention it in case there is some other difference between MASTER and DEV that may be playing a role.

Mario Vavti

Tue, 16 Jul 2019 18:33:38 +0000

Hm... Nope, my queue is at 30 while threshold is at 3000 and if i disable the addon, a new post is delivered within some seconds... I do have federation addons enabled but that should not make a difference afaik...

Tue, 16 Jul 2019 18:44:20 +0000

I know you're thorough and careful about testing - so I'm doubtful that you've missed anything - I'm just at a loss to explain it.

If you have debugging on - do you see both the "Summon handled by hook" and "Release handled by hook" messages right after you post a message?

Tue, 16 Jul 2019 18:49:41 +0000

Oh! Interesting..... Sorry, I am actually seeing the same behavior. Ok.... Saw debug message (1) and interpreted it as debug message (2).... sorry for the confusion.

Tue, 16 Jul 2019 19:37:10 +0000

Ok, So digging in --- queueworker is properly calling Zotlabs\Daemon\Deliver.php::run().  It has the correct arguments and passes the correct data.

But when Deliver.php::run() does the "select * from outq..." there is no entry in the outq table at that point for some reason.  So deliver terminates, finding nothing to do.

When the POLLER runs, the entry is in outq, and so, all works as it should.

Clearly queueworker is not interfering with it getting entered in outq, but somehow, it is messing up the timing and ordering of events for whatever reason.

I need to dig deeper, but that's the actual cause.  Now, WHY the entry isn't found in outq is the question.

Mario Vavti

Tue, 16 Jul 2019 20:54:19 +0000 last edited: Tue, 16 Jul 2019 20:57:57 +0000

As far as i see, the dataformat is not the same.

The release argv with the traditional worker looks like this

[
    [0] => Deliver
    [1] => befc3c895f5417d0576276741d45dbb04b95f2fed0f58ffe93163fcb30a2f754
    [2] => 5df3a31b5ec28c409c3c53afceb82658c39ee9cb8970f2af
    [3] => 847431b0ca8549552f43e7eb5f7c1e74505efb711bd83fdb
]

while with the queueworker addon it looks like this

[
    [0] => Deliver
    [1] => [
        [0] => befc3c895f5417d0576276741d45dbb04b95f2fed0f58ffe93163fcb30a2f754
        [1] => 5df3a31b5ec28c409c3c53afceb82658c39ee9cb8970f2af
        [2] => 847431b0ca8549552f43e7eb5f7c1e74505efb711bd83fdb
    ]
]

Wed, 17 Jul 2019 02:54:29 +0000

And there it is... Thanks.

proc_run() uses flatten_array_recursive().

New MR submitted - thanks!

poVoq

Wed, 23 Oct 2019 20:16:34 +0000

@M. Dent Did you ever notice that the QueryWorker addon (at least in the current Master branch) does not work with RSS feed reposts?

Wed, 23 Oct 2019 22:12:20 +0000

Hadn't noticed that - you mean if you have a channel subscribed to an RSS feed that is supposed to republish the feed material? (Just trying to clarify what you mean by "RSS feed reposts").

I'll try to take a look in the next day or two.

poVoq

Thu, 24 Oct 2019 12:48:05 +0000

@M. Dent yes I run a few of such "planets" and all of them stopped relaying rss feeds as soon as I enabled the query worker addon but otherwise worked fine. After disabling the queryworker addon they immediatly stared relaying RSS feeds again.

Fri, 25 Oct 2019 10:47:20 +0000

Very strange. I will see if I can trace down a cause.

poVoq

Mon, 11 Nov 2019 14:10:13 +0000

@M. Dent Did you manage to fix this so that it can go into the new Hubzilla release?

Mon, 11 Nov 2019 15:35:33 +0000

Thanks for the reminder! (No, I got distracted with a bunch of other stuff... sorry... But, I will try to make sure it's found before 4.6 finishes the release cycle.)

Mon, 11 Nov 2019 18:59:29 +0000

@poVoq -

Are all the hubs having this trouble running 4.4.1? (I should have looked/asked before... but it didn't dawn on me)

There was a commit to DEV back in August that should resolve this issue.

Return in the case of further processing (!1711) · Merge Requests · hubzilla / core

Queueworker expects dispatched work to return in order to continue processing. Killing the thread is counter productive.

poVoq

Mon, 11 Nov 2019 19:02:04 +0000

@M. Dent yes latest stable release. Good to know that it is already fixed in dev. Looking forward to upgrade and reenabling it soon then :)

Mario Vavti

Mon, 11 Nov 2019 19:26:41 +0000

@poVoq iirc this fix is included in 4.4.1

Mon, 11 Nov 2019 20:45:32 +0000

@Mario Vavti  -

I thought it may have been also.  It WAS really close, but it didn't make it.

I just double checked the datestamps.  The underlying commit was created August 19, the MR was opened August 20, and was merged by Max on September 10.   4.4.1 was released August 16.

Mon, 11 Nov 2019 20:50:29 +0000 last edited: Mon, 11 Nov 2019 20:51:44 +0000

We use new queueworker with Zotum and and I don’t see critical problems maybe except that sometimes the number of working workers goes beyond an established limits. Don't sure why and when this happens.

Mon, 11 Nov 2019 22:06:24 +0000

The system assumes that if a process entry in the database is older than X - the process was killed or died before finishing, but is no longer active. Because outside forces may kill a process, and in some cases the process won't be able to report back what happened, there is not much that can be done. The database entries are just deleted after the timeout.

Because of this, there is a rare instance where some of the longer running processes (eg. RSS feed processing) will take longer than the timeout, but won't actually have died, and so the limit will be exceeded.

This is kept to a minimum as much as possible by checking whether a queue thread process has existed longer than the timeout before starting another task. But the current task itnis processing will be allowed to finish first.