Arjun Roy 94ab9eb9b2
net-zerocopy: Defer vm zap unless actually needed. ...
Zapping pages is required only if we are calling vm_insert_page into a
region where pages had previously been mapped. Receive zerocopy allows
reusing such regions, and hitherto called zap_page_range() before
calling vm_insert_page() in that range.
zap_page_range() can also be triggered from userspace with
madvise(MADV_DONTNEED). If userspace is configured to call this before
reusing a segment, or if there was nothing mapped at this virtual
address to begin with, we can avoid calling zap_page_range() under the
socket lock. That said, if userspace does not do that, then we are
still responsible for calling zap_page_range().
This patch adds a flag that the user can use to hint to the kernel
that a zap is not required. If the flag is not set, or if an older
user application does not have a flags field at all, then the kernel
calls zap_page_range as before. Also, if the flag is set but a zap is
still required, the kernel performs that zap as necessary. Thus
incorrectly indicating that a zap can be avoided does not change the
correctness of operation. It also increases the batchsize for
vm_insert_pages and prefetches the page struct for the batch since
we're about to bump the refcount.
An alternative mechanism could be to not have a flag, assume by
default a zap is not needed, and fall back to zapping if needed.
However, this would harm performance for older applications for which
a zap is necessary, and thus we implement it with an explicit flag
so newer applications can opt in.
When using RPC-style traffic with medium sized (tens of KB) RPCs, this
change yields an efficency improvement of about 30% for QPS/CPU usage.
Signed-off-by: Arjun Roy <arjunroy@google.com >
Signed-off-by: Eric Dumazet <edumazet@google.com >
Signed-off-by: Soheil Hassas Yeganeh <soheil@google.com >
Signed-off-by: Jakub Kicinski <kuba@kernel.org >
2020-12-04 13:40:53 -08:00
2020-07-23 09:35:36 +02:00
2020-07-15 20:34:11 -07:00
2020-11-20 12:05:14 +01:00
2019-06-05 10:55:30 -04:00
2020-01-21 11:41:36 +01:00
2020-08-22 10:53:12 +01:00
2019-09-07 17:44:25 +02:00
2020-05-28 11:22:14 +02:00
2020-10-31 11:55:38 +01:00
2019-10-04 14:25:17 -07:00
2020-05-04 11:30:15 -05:00
2019-10-04 14:25:17 -07:00
2019-10-04 14:25:17 -07:00
2019-09-10 09:26:33 -04:00
2020-07-21 22:05:32 -07:00
2019-06-24 19:23:46 +02:00
2020-06-15 16:03:38 +01:00
2020-10-03 17:28:45 -07:00
2020-08-12 10:57:59 -07:00
2019-12-18 18:07:31 +01:00
2019-06-26 20:14:13 -04:00
2019-04-25 19:33:34 +02:00
2020-07-13 17:01:44 -07:00
2020-07-21 11:21:44 -04:00
2020-08-12 10:58:00 -07:00
2019-03-07 18:32:01 -08:00
2020-06-26 10:36:30 +02:00
2020-06-26 10:37:11 +02:00
2020-07-25 07:38:21 -06:00
2019-03-07 18:32:01 -08:00
2020-10-09 12:47:02 -06:00
2020-12-03 17:38:21 -08:00
2019-07-25 11:05:10 +02:00
2020-01-10 17:20:07 +01:00
2020-10-07 12:13:23 +02:00
2020-07-27 12:55:43 +02:00
2020-11-20 09:43:29 +01:00
2020-09-01 19:38:33 -06:00
2020-10-02 17:58:52 -06:00
2019-10-07 07:55:17 -03:00
2019-10-01 17:19:41 -03:00
2020-10-29 18:39:43 -07:00
2019-10-09 22:31:14 -04:00
2020-06-17 00:07:38 +02:00
2019-07-16 19:23:23 -07:00
2019-09-25 17:51:39 -07:00
2020-09-17 18:46:04 +02:00
2019-08-02 14:44:02 +10:00
2019-11-15 14:38:29 +01:00
2019-10-28 13:41:01 -07:00
2020-11-24 13:04:04 -08:00
2020-05-04 11:30:15 -05:00
2020-10-01 15:03:40 -04:00
2020-04-27 16:29:41 +05:30
2019-12-17 21:37:40 +05:30
2020-09-10 09:10:06 +02:00
2019-03-20 21:11:22 -04:00
2020-09-04 12:46:07 +01:00
2020-07-19 19:20:22 -07:00
2020-09-22 16:45:34 -07:00
2020-09-19 16:54:35 -07:00
2020-07-27 23:24:00 +02:00
2020-06-24 21:34:11 +02:00
2020-05-14 16:44:25 +02:00
2020-05-12 19:34:57 +03:00
2020-05-12 19:34:52 +03:00
2020-06-05 16:19:28 -07:00
2020-04-20 15:45:41 -06:00
2019-03-27 13:30:07 -07:00
2020-07-06 21:37:08 -07:00
2020-05-28 22:09:47 -04:00
2020-09-22 06:48:44 -07:00
2019-08-12 19:33:50 -07:00
2020-09-18 15:17:40 +02:00
2019-11-05 18:20:55 -08:00
2020-10-03 14:18:29 -07:00
2020-06-05 20:19:20 +02:00
2020-10-28 15:46:47 +01:00
2019-09-04 12:43:54 +02:00
2020-08-27 08:01:47 -07:00
2019-12-11 15:31:52 +01:00
2020-07-27 12:20:40 -07:00
2020-07-23 17:55:20 +00:00
2020-07-19 19:20:22 -07:00
2020-10-31 13:16:02 -07:00
2020-08-05 19:02:07 +05:30
2020-09-25 17:48:52 +10:00
2020-05-04 11:30:15 -05:00
2019-12-26 13:09:37 -08:00
2020-10-30 17:49:19 -07:00
2020-10-29 18:39:43 -07:00
2020-12-03 08:21:29 -08:00
2020-03-16 01:42:31 -07:00
2020-11-09 16:41:40 -08:00
2020-08-22 12:44:37 -07:00
2019-03-21 13:19:15 -07:00
2019-04-19 13:58:17 -07:00
2020-05-01 12:24:40 -07:00
2020-07-13 15:32:56 -07:00
2020-04-20 12:43:24 -07:00
2020-07-24 17:12:41 -07:00
2020-08-27 07:55:59 -07:00
2020-09-03 15:17:28 -07:00
2020-09-08 16:31:14 -07:00
2019-12-13 15:00:36 -08:00
2020-09-30 20:32:33 -06:00
2020-10-01 14:52:46 +02:00
2019-07-04 02:29:49 +02:00
2019-07-25 11:05:10 +02:00
2020-09-15 09:57:45 -05:00
2020-03-29 22:30:57 -07:00
2020-07-27 16:34:12 +03:00
2019-08-22 17:24:51 +02:00
2020-01-04 13:55:09 -08:00
2019-03-05 21:07:14 -08:00
2019-09-08 15:37:04 +02:00
2020-05-19 15:19:06 +01:00
2020-08-31 14:40:29 -04:00
2019-07-02 18:56:01 +02:00
2020-10-21 17:36:32 -04:00
2020-09-29 13:26:36 -07:00
2019-03-07 18:31:59 -08:00
2019-04-22 13:02:53 -04:00
2020-05-21 08:20:35 -06:00
2020-03-29 22:30:57 -07:00
2020-05-27 11:10:05 +02:00
2020-08-12 10:57:59 -07:00
2020-07-19 18:05:49 -07:00
2020-03-20 09:01:16 +01:00
2019-06-21 16:45:38 -04:00
2020-08-18 15:44:44 +02:00
2020-09-25 14:23:27 +02:00
2020-05-06 17:45:45 -07:00
2020-09-15 22:13:31 +10:00
2020-08-27 16:06:47 -04:00
2020-07-09 12:38:41 -07:00
2019-02-21 13:05:05 -08:00
2020-09-10 12:25:51 -07:00
2020-11-25 13:33:35 -08:00
2019-11-15 14:38:29 +01:00
2020-07-25 19:34:47 -06:00
2020-06-24 14:36:33 -07:00
2020-05-04 11:30:15 -05:00
2020-03-29 22:32:36 -07:00
2020-10-14 20:28:05 -07:00
2020-10-09 20:22:32 -07:00
2020-05-22 14:00:38 -07:00
2020-10-16 09:28:43 -04:00
2020-07-13 17:52:46 -04:00
2019-04-25 14:18:14 -04:00
2020-10-02 09:37:41 -04:00
2019-07-12 11:05:40 -07:00
2020-09-22 13:58:40 +02:00
2020-11-11 08:39:13 +01:00
2019-11-06 06:17:38 +09:00
2020-01-18 09:19:18 -05:00
2020-11-27 11:03:06 -08:00
2020-10-21 09:58:39 -05:00
2020-04-02 17:57:10 +01:00
2020-10-19 19:39:22 +02:00
2019-10-02 20:32:27 -06:00
2020-09-04 12:34:50 +02:00
2020-07-24 15:23:31 -07:00
2020-06-29 17:08:28 -07:00
2019-07-30 20:34:34 +02:00
2019-12-01 14:00:59 -08:00
2019-10-23 17:23:47 +02:00
2020-09-04 12:46:07 +01:00
2020-05-21 17:04:07 -07:00
2019-04-16 23:05:21 +02:00
2020-04-30 15:19:33 +10:00
2020-07-19 19:22:56 -07:00
2019-07-16 19:23:24 -07:00
2020-01-07 16:07:01 -05:00
2020-07-10 14:50:51 +02:00
2020-06-25 16:26:25 -07:00
2020-08-04 20:16:37 -07:00
2019-11-15 14:38:29 +01:00
2020-03-29 22:30:57 -07:00
2020-03-29 22:30:57 -07:00
2020-05-11 16:04:35 +02:00
2020-11-25 12:34:44 -08:00
2020-10-05 16:35:57 +01:00
2019-12-04 19:44:12 -08:00
2020-02-12 17:57:51 -05:00
2020-10-30 15:24:11 -07:00
2020-07-14 16:29:42 -07:00
2019-11-04 07:11:31 -07:00
2020-08-03 17:57:40 -07:00
2020-12-04 13:30:50 -08:00
2019-11-15 14:38:29 +01:00
2020-07-06 14:06:08 +02:00
2020-03-24 15:59:34 -07:00
2019-11-15 14:38:29 +01:00
2020-12-01 17:56:13 -08:00
2020-11-09 15:34:44 -08:00
2020-02-27 18:50:19 -08:00
2019-08-03 11:02:46 -07:00
2019-04-19 14:07:40 -07:00
2020-12-03 10:03:14 -08:00
2020-02-21 11:22:15 -08:00
2020-01-15 11:00:39 -06:00
2020-01-31 10:30:39 -08:00
2020-07-28 22:25:30 -04:00
2019-12-18 18:07:31 +01:00
2020-12-04 13:40:53 -08:00
2020-08-21 08:55:13 +02:00
2020-07-07 15:55:21 +02:00
2019-12-18 18:07:33 +01:00
2020-02-21 11:22:15 -08:00
2019-12-18 18:07:33 +01:00
2019-10-04 14:25:17 -07:00
2020-09-18 13:58:37 -07:00
2020-09-18 13:58:37 -07:00
2020-11-27 14:32:37 -08:00
2020-08-12 10:57:59 -07:00
2019-12-09 09:59:07 +01:00
2020-03-29 23:29:08 +02:00
2019-05-22 10:36:35 -07:00
2019-08-14 16:52:13 +02:00
2020-04-07 10:43:39 -07:00
2019-11-15 14:38:29 +01:00
2020-09-14 15:39:42 +02:00
2020-09-26 10:22:29 +02:00
2020-09-26 10:22:29 +02:00
2020-07-10 13:45:32 +02:00
2020-07-10 13:45:32 +02:00
2020-06-03 11:27:43 +02:00
2020-10-07 14:23:44 -06:00
2020-10-12 11:41:02 -06:00
2020-10-23 11:55:28 -04:00
2020-10-23 11:55:28 -04:00
2020-10-04 12:19:12 +02:00
2020-08-05 09:30:19 -04:00
2020-08-05 09:30:20 -04:00
2020-08-05 11:08:39 -04:00
2020-08-03 16:11:42 -04:00
2020-08-05 11:08:39 -04:00
2020-08-05 11:08:39 -04:00
2020-09-10 11:39:22 +02:00
2020-08-18 14:14:25 +02:00
2020-06-04 15:36:52 -04:00
2020-08-05 11:08:40 -04:00
2020-08-05 11:08:40 -04:00
2020-08-05 11:08:40 -04:00
2020-09-10 10:05:58 +02:00
2020-08-05 11:08:41 -04:00
2020-09-10 10:05:58 +02:00
2020-08-05 11:08:40 -04:00
2020-06-02 02:45:13 -04:00
2020-08-05 11:08:40 -04:00
2019-12-11 15:01:23 -08:00
2019-07-25 11:05:10 +02:00
2019-08-01 21:49:46 +02:00
2020-05-19 15:19:06 +01:00
2019-12-16 19:22:22 -08:00
2020-07-31 09:24:23 +02:00
2019-07-25 11:05:10 +02:00
2020-06-12 13:23:34 -04:00
2020-07-13 15:32:56 -07:00
2020-06-24 07:51:01 +02:00
94ab9eb9b2