PHP8: Benchmarking the Symfony Demo App with and without JIT
I was recently asked whether JIT will bring a big improvement to our PHP applications and I decided to put that to the test. So here's a short test of PHP 8 performance with different configurations.
The Test Setup
- Symfony Demo Application (Blog with Symfony 5 and Doctrine / SQLite)
- PHP 8 beta 3 - for more infos see my setup guide
- Using PHP internal Web Server via
symfony serve - Tests done using Apache Benchmark
Symfony with Doctrine is pretty much our go-to setup for most applications, so the demo application should give me a pretty good idea of what impact PHP 8 will have on our applications.
The Results
PHP 8 performance benchmark without OPCache
Let's first establish a baseline. PHP8 with out-of-the-box configuration with opcache disabled:
$ ab -n 100 -c 20 -l localhost/en/blog/
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, www.zeustech.net
Licensed to The Apache Software Foundation, www.apache.org
Document Path: /en/blog/
Document Length: Variable
Concurrency Level: 20
Time taken for tests: 18.510 seconds
Complete requests: 100
Failed requests: 0
Total transferred: 8194800 bytes
HTML transferred: 8153300 bytes
Requests per second: 5.40 [#/sec] (mean)
Time per request: 3701.971 [ms] (mean)
Time per request: 185.099 [ms] (mean, across all concurrent requests)
Transfer rate: 432.35 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 3 5 4.4 3 19
Processing: 193 3308 915.8 3652 3826
Waiting: 191 3308 915.9 3651 3825
Total: 197 3313 913.3 3654 3829
Percentage of the requests served within a certain time (ms)
50% 3654
66% 3736
75% 3771
80% 3778
90% 3802
95% 3816
98% 3828
99% 3829
100% 3829 (longest request)
Seems pretty slow to me - 5.4 requests per second
PHP 8 performance benchmark with OPCache
Let's set the following configuration options to enable OPCache and restart the symfony app:
zend_extension=opcache.so
opcache.enable=1
opcache.enable_cli=1
Let's test again:
$ ab -n 100 -c 20 -l localhost/en/blog/
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, www.zeustech.net
Licensed to The Apache Software Foundation, www.apache.org
Document Path: /en/blog/
Document Length: Variable
Concurrency Level: 20
Time taken for tests: 3.674 seconds
Complete requests: 100
Failed requests: 0
Total transferred: 8194800 bytes
HTML transferred: 8153300 bytes
Requests per second: 27.22 [#/sec] (mean)
Time per request: 734.707 [ms] (mean)
Time per request: 36.735 [ms] (mean, across all concurrent requests)
Transfer rate: 2178.48 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 2 5 3.7 3 14
Processing: 51 609 151.3 668 697
Waiting: 51 609 151.3 668 697
Total: 58 614 149.0 671 711
Percentage of the requests served within a certain time (ms)
50% 671
66% 675
75% 681
80% 685
90% 688
95% 689
98% 691
99% 711
100% 711 (longest request)
Ok, that's a lot faster - we are now at 27.22 requests per second
PHP 8 performance benchmark with OPCache and JIT
To answer the original question on whether or not JIT will help our applications, I then added JIT with the following settings in php.ini:
opcache.jit_buffer_size=100M
opcache.jit=1235
(You can look up the config reference and defaults at https://wiki.php.net/rfc/jit#phpini_defaults )
Now, let's benchmark again:
$ ab -n 100 -c 20 -l localhost/en/blog/
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, www.zeustech.net
Licensed to The Apache Software Foundation, www.apache.org
Document Path: /en/blog/
Document Length: Variable
Concurrency Level: 20
Time taken for tests: 3.756 seconds
Complete requests: 100
Failed requests: 0
Total transferred: 8194800 bytes
HTML transferred: 8153300 bytes
Requests per second: 26.63 [#/sec] (mean)
Time per request: 751.111 [ms] (mean)
Time per request: 37.556 [ms] (mean, across all concurrent requests)
Transfer rate: 2130.91 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 2 4 3.6 3 15
Processing: 64 612 148.7 673 692
Waiting: 63 612 148.7 673 692
Total: 71 616 146.2 675 706
Percentage of the requests served within a certain time (ms)
50% 675
66% 680
75% 682
80% 683
90% 686
95% 687
98% 687
99% 706
100% 706 (longest request)
Huh, that's actually a bit slower than without JIT configuration. But to be honest, I wasn't expecting much of a difference (see below). We are now at 26.63 requests per second, which seems like more or less the same result as before.
Comparison, Result, Conclusion
| Configuration | Requests |
|---|---|
| PHP8 without OPCache | 5.4 rq/s |
| PHP8 with OPCache | 27.2 rq/s |
| PHP8 with OPCache + JIT | 26.6 rq/s |
The point is, if you look in the announcements and comments about what the JIT does you can pretty much guess why this is kind of the expected result. The JIT compiler improves performance for CPU-intensive work (for example calculating Mandelbrots) but not so much for the "default" kind of applications. We usually don't do all that many calculations.
Upshot: We won't see an immediate improvement of our default use cases, but there are two things I'd like to point out before you go dismissing the JIT out of hand:
- it has just been introduced and will most likely be improved further in the future, so it might improve our setups after all
- the CPU computation performance improvements mean that PHP might now be able to compete in a field it did not compete in before - we can now probably use PHP in areas where we would have had to switch to a different language for decent performance before now
All in all, enabling JIT does not seem to harm our applications and might even help in high computational load scenarios.
Sidenote: Comparison with PHP 7.4
I also did the same benchmark on the same Symfony app with PHP 7.4: there is no notable difference in the performance between the two versions.