View Issue Details

IDProjectCategoryView StatusLast Update
0025497mantisbtemailpublic2019-02-24 06:49
ReporterChrisG Assigned To 
PrioritynormalSeverityminorReproducibilityhave not tried
Status newResolutionopen 
Summary0025497: Optimize send_emails.php performance
Description

send_emails.php is called up as a Cronjob, meaning on most servers it will not benefit from Opcode caching.

According to my profiler, it is loading up 76 PHP files and taking around 8 seconds to do this. If it is running every minute, that is a very significant overhead on the server.

Now, the 8 second time figure may be exaggerated based on some server performance issues I am currently chasing, but it's still a lot of work on a routine background task that is just sending emails.

I realize to get this down it will require some heavy restructuring of how the bootup process is working within Mantis.

TagsNo tags attached.

Activities

atrol

atrol

2019-02-19 18:13

developer   ~0061534

I assume that missing opcode caching is not the root cause of the performance issue.
As a first step, check if database table mantis_email_table is empty, contains just a few entries or contains lots of entries.

Independent from that, you could try if performance is better when using the default

$g_email_send_using_cronjob = OFF;
ChrisG

ChrisG

2019-02-19 18:25

reporter   ~0061535

Last edited: 2019-02-19 18:26

For 0 emails in that table, all the following .php files have to be parsed (I just hacked the code to spit out a list)...

bug_api.php
gpc_api.php
php_api.php
rss_api.php
tag_api.php
date_api.php
file_api.php
form_api.php
html_api.php
http_api.php
icon_api.php
lang_api.php
ldap_api.php
news_api.php
user_api.php
email_api.php
error_api.php
event_api.php
print_api.php
access_api.php
config_api.php
crypto_api.php
events_inc.php
filter_api.php
helper_api.php
layout_api.php
plugin_api.php
string_api.php
tokens_api.php
bugnote_api.php
columns_api.php
history_api.php
logging_api.php
mention_api.php
prepare_api.php
profile_api.php
project_api.php
session_api.php
utility_api.php
version_api.php
antispam_api.php
category_api.php
collapse_api.php
compress_api.php
constant_inc.php
database_api.php
api_token_api.php
user_pref_api.php
email_queue_api.php
filter_form_api.php
sponsorship_api.php
bug_revision_api.php
current_user_api.php
custom_field_api.php
last_visited_api.php
relationship_api.php
authentication_api.php
custom_function_api.php
bug_group_action_api.php
filter_constants_inc.php
project_hierarchy_api.php

So that's the problem really. With opcode caching it would be less of a problem, but even with, that's a lot of code to call up every minute.

The problem with sending out emails in realtime is if there's a lot, you can get a real lag problem. That's why I had to enable the mail queue.

ChrisG

ChrisG

2019-02-19 18:35

reporter   ~0061536

Here's a hackerish patch...

--- a/tracker/core.php
+++ b/tracker/core.php
@@ -282,6 +282,11 @@ if( !defined( 'MANTIS_MAINTENANCE_MODE' ) ) {
        }
 }

+# Performance optimization for sending emails
+if (basename($argv[0]) == 'send_emails.php') {
+       if (db_result( db_query( 'SELECT COUNT(*) FROM mantis_email_table' ) ) == 0) exit(); // Nothing to do
+}
+
 # Register global shutdown function
 shutdown_functions_register();

I suppose it would not hurt to merge something like that, given the alternatives are leaving the inefficiency or heavily restructuring the code.

cproensa

cproensa

2019-02-19 20:11

developer   ~0061537

I'm not denying your point, but:

How many files are parsed for a typical page view from a user?
How many page views do you expect per minute?
Regarding number/size of php files being parsed by the email job, is that any more significative than one user view per minute?

ChrisG

ChrisG

2019-02-19 20:25

reporter   ~0061539

Right, yes. There is some nuance to this. It shouldn't hurt a server if an opcode cache is there. If an opcode cache is there, it can well be the case that parsing that much PHP and doing that many disk I/O operations every minute could cause very spiky behaviour, especially with a low number of cores and/or an unimpressive HDD with poor disk caching. So my point about opcode caches not being configured for CLI is key to the understanding of priority here. It really can be very surprising how much lack of an opcode cache can change things.

Regarding opcode caching not being on - I can say for Zend Opcache by default it is totally disabled for CLI (opcache.enable_cli=0), and is configured to use shared memory which only works for a memory-resident PHP context (like mod_php). It can be configured to be enabled on CLI, and to use a disk-based backend (MMAP), but that's not the default and usually won't be configured.

atrol

atrol

2019-02-20 02:56

developer   ~0061543

that's a lot of code to call up every minute.

What's the reason to run it every minute?
5 Minutes are recommended and should be fast enough for typical use cases.

The problem with sending out emails in realtime is if there's a lot, you can get a real lag problem.

Did you try with latest version, or is it an observation when using older MantisBT versions?
I am asking as there has been some conceptual change some while ago.
E-Mails are sent in shutdown function, so they shouldn't block UI.
Of course, this change has not been documented.

Concerning

if there's a lot

There are hardly any use cases where a lot emails are sent as a result of a single page request.
So maybe $g_email_send_using_cronjob = OFF; is an option for you to have good performance without any tweaks.

76 PHP files and taking around 8 seconds to do this.

This is what I see when running latest MantisBT using PHP 7.0.32 on Ubuntu 16.04, using a pretty cheap hosted virtual system.

time php ./scripts/send_emails.php
Sending emails...
Done.

real    0m0.125s
user    0m0.102s
sys     0m0.020s

Your system can hardly be 64 times slower than my test system, assuming you are not using a Raspberry Pi 1 like this user 0025121.

Don't you think there could be some configuration issue on your server?

To avoid comparing apples to oranges, can you try with MantisBT 2.19.0 or latest code from master branch?
I am especially asking, as we had a major version upgrade of the underlying PHPMailer component, see 0024990.
The old PHPMailer 5.2 did require just PHP >=5.3.3, but AFAIK PHP > 7.0 was never officially supported, as there was PHPMailer 6.0 at this time.

ChrisG

ChrisG

2019-02-20 17:17

reporter   ~0061544

Last edited: 2019-02-20 17:18

Thanks for the help guys :).

"What's the reason to run it every minute?" - I like the responsiveness, but I probably didn't even think about that or properly read the docs TBH lol.

It is correct to point out my server has some issues. That's why I went down the rabbithole of installing tideways-xhprof on there and having everything benchmarked over a day or so. Basically I think my machine was doing a lot of page swapping which was slowing processes, and disk reads, down - it had a decent amount of free RAM, but I think Linux wanted to keep that RAM free by preemptively moving less important stuff in and out of swap. I did a tonne of optimization, but reconfiguring the FastCGI settings and reducing the overhead of some other processes has helped most.

I have now cleared that issue, and this is what I am currently getting:

real    0m0.379s
user    0m0.266s
sys 0m0.052s

So, far better. A little worse than what you're getting, but certainly very tolerable.

However...

It is a reality some people are going to be on crappy server situations. Over the years I've seen countless very stressed servers provided by webhosts.

Imagine a user with:

  • Mechanical HDD
  • No opcode caching
  • No effective disk caching
  • HDD is currently in the process of page swapping
  • HDD read head is having to jump all over the place due to competing processes needing to read
  • And dealing with 30 other web requests and a handful of other background Cron processes at the same time, on a shared or over-stretched server

That's more the reality for some people.

Now, to be honest, that's more of an argument to optimize the requests that are coming in thick and fast, rather than something running just every 1 or 5 minutes.
But at the same time, I don't want to see any very minor script sitting around for long because there's an overhead in that (e.g. RAM, in an environment where RAM may already be overstretched - or making the HDD read head jump about even further). The web requests may be cleanly queued, Cron requests not so much, and they are spiky.

I realize this isn't a major or pressing issue, but if someone has the time I do think it's worth looking at at some point.

Regarding register_shutdown_function...

This can be problematic for a few reasons:
1) It will hold up an Apache instance (if non-threaded), or it will hold up a fast CGI instance, until the operation is finished.
2) Depending on various factors, it may keep the connection open (i.e. no Connection: Close header) until the operation is finished - so even if the page has painted, it may show as still 'loading'
3) I have around 6 people getting every tracker notification, so that's at least 6 emails for each new issue or issue reply.

TBH I don't want to start tinkering with my configuration again as I just lost 3 days unexpectedly wrestling my server back under control and am seriously behind - sorry about that. It may well be that the speed your sending out these emails now is fast enough that it's barely noticeable even when sending dozens. I'm happy with things as they are right now with the Cron script.

atrol

atrol

2019-02-24 06:49

developer   ~0061563

I went down the rabbithole of installing tideways-xhprof

From my experience with other profiling tools I expect this added a lot of overhead and influenced performance a lot more than any other topic we discussed before.