diff --git a/docs/cn/bthread.md b/docs/cn/bthread.md index f0de58bcd7..809050986c 100644 --- a/docs/cn/bthread.md +++ b/docs/cn/bthread.md @@ -1,3 +1,5 @@ +[English version](../en/bthread.md) + [bthread](https://github.com/apache/brpc/tree/master/src/bthread)是brpc使用的M:N线程库，目的是在提高程序的并发度的同时，降低编码难度，并在核数日益增多的CPU上提供更好的scalability和cache locality。”M:N“是指M个bthread会映射至N个pthread，一般M远大于N。由于linux当下的pthread实现([NPTL](http://en.wikipedia.org/wiki/Native_POSIX_Thread_Library))是1:1的，M个bthread也相当于映射至N个[LWP](http://en.wikipedia.org/wiki/Light-weight_process)。bthread的前身是Distributed Process(DP)中的fiber，一个N:1的合作式线程库，等价于event-loop库，但写的是同步代码。 # Goals diff --git a/docs/cn/bthread_id.md b/docs/cn/bthread_id.md index d6ba8a2c68..34f432972a 100644 --- a/docs/cn/bthread_id.md +++ b/docs/cn/bthread_id.md @@ -1,3 +1,5 @@ +[English version](../en/bthread_id.md) + bthread_id是一个特殊的同步结构，它可以互斥RPC过程中的不同环节，也可以O(1)时间内找到RPC上下文(即Controller)。注意，这里我们谈论的是bthread_id_t，不是bthread_t（bthread的tid），这个名字起的确实不太好，容易混淆。 具体来说，bthread_id解决的问题有： diff --git a/docs/cn/bthread_or_not.md b/docs/cn/bthread_or_not.md index 2b71c90dd7..0ba33d5f93 100644 --- a/docs/cn/bthread_or_not.md +++ b/docs/cn/bthread_or_not.md @@ -1,3 +1,5 @@ +[English version](../en/bthread_or_not.md) + brpc提供了[异步接口](client.md#异步访问)，所以一个常见的问题是：我应该用异步接口还是bthread？ 短回答：延时不高时你应该先用简单易懂的同步接口，不行的话用异步接口，只有在需要多核并行计算时才用bthread。 diff --git a/docs/cn/bthread_tagged_task_group.md b/docs/cn/bthread_tagged_task_group.md index 027bd4eb9e..038eaead0b 100644 --- a/docs/cn/bthread_tagged_task_group.md +++ b/docs/cn/bthread_tagged_task_group.md @@ -1,3 +1,4 @@ +[English version](../en/bthread_tagged_task_group.md) # Bthread tagged task group diff --git a/docs/cn/bthread_tracer.md b/docs/cn/bthread_tracer.md index bab09ce40c..f92b356d9e 100644 --- a/docs/cn/bthread_tracer.md +++ b/docs/cn/bthread_tracer.md @@ -1,3 +1,5 @@ +[English version](../en/bthread_tracer.md) + gdb（ptrace）+ gdb_bthread_stack.py主要的缺点是要慢和阻塞进程，需要一种高效的追踪bthread调用栈的方法。 bRPC框架的协作式用户态协程无法像Golang内建的抢占式协程一样实现高效的STW（Stop the World），框架也无法干预用户逻辑的执行，所以要追踪bthread调用栈是比较困难的。 diff --git a/docs/cn/execution_queue.md b/docs/cn/execution_queue.md index 32d4d2d43b..7814741d90 100644 --- a/docs/cn/execution_queue.md +++ b/docs/cn/execution_queue.md @@ -1,3 +1,5 @@ +[English version](../en/execution_queue.md) + # 概述 类似于kylin的ExecMan, [ExecutionQueue](https://github.com/apache/brpc/blob/master/src/bthread/execution_queue.h)提供了异步串行执行的功能。ExecutionQueue的相关技术最早使用在RPC中实现[多线程向同一个fd写数据](io.md#发消息). 在r31345之后加入到bthread。 ExecutionQueue 提供了如下基本功能: diff --git a/docs/en/bthread.md b/docs/en/bthread.md index 7c627f1b5b..9b8dd51c0d 100644 --- a/docs/en/bthread.md +++ b/docs/en/bthread.md @@ -1,7 +1,67 @@ -# uthread +[中文版](../cn/bthread.md) -This document has not yet been translated into English. +# bthread -Please refer to the [Chinese version](../cn/bthread.md) for the full content. +[bthread](https://github.com/apache/brpc/tree/master/src/bthread) is the M:N threading library used by brpc. The goal is to raise concurrency, keep coding simple, and get better scalability and cache locality on CPUs with more and more cores. "M:N" means M bthreads are mapped onto N pthreads, and M is usually much larger than N. Because Linux pthread ([NPTL](http://en.wikipedia.org/wiki/Native_POSIX_Thread_Library)) is 1:1, those M bthreads are also mapped onto N [LWPs](http://en.wikipedia.org/wiki/Light-weight_process). bthread grew out of the fiber in Distributed Process (DP), an N:1 cooperative threading library. That model is equivalent to an event-loop library, except that users write synchronous code. -Contributions to translate this document are welcome. See [TRANSLATING](TRANSLATING) for guidelines. +# Goals + +- Users keep the synchronous programming style, can create a bthread in a few hundred nanoseconds, and can synchronize with a variety of primitives. +- Every bthread API is callable from a pthread and has reasonable behavior. Code that uses bthread APIs can run correctly inside a pthread. +- Make full use of multiple cores. +- Better cache locality; NUMA support is a plus. + +# Non-goals + +- Provide a pthread-compatible ABI that works just by linking. **Rejected because**: bthread has no priorities and is not suitable for every workload. Silent replacement by linking would make users pick up bthread without knowing it, and cause bugs. +- Intercept every possibly blocking glibc function and syscall so that they block the bthread instead of the system thread. **Rejected because**: + - Blocking a bthread may switch the underlying system thread, so functions that depend on system TLS have undefined behavior. + - Mixing them with functions that block pthreads can deadlock. + - These hooks are usually slower, because they often need extra syscalls such as epoll. The same coverage is more useful for an N:1 cooperative library (fiber): the hook itself is slower, but without it the whole system thread blocks and every fiber stalls. +- Patch the kernel so that pthread can switch quickly on the same core. **Rejected because**: with a large number of pthreads, per-thread resources are diluted and thread-local caches (for example tcmalloc) work poorly. A separate bthread library does not have this problem, because it still maps onto a small number of pthreads. A large part of bthread's speedup over pthread comes from concentrating thread resources. Portability also matters: bthread prefers pure userland code. + +# FAQ + +##### Q: Is bthread a coroutine? + +No. "Coroutine" here means an N:1 threading library: all coroutines run in one system thread. Compute power is equivalent to an event-loop library. Because they never leave that thread, switches need no syscall (about 100ns–200ns) and cache-coherence cost is small. The cost is that coroutines cannot use multiple cores well, and the code must be non-blocking, otherwise every coroutine stalls. That makes them a good fit for IO servers whose run time is deterministic, such as an HTTP server. Carefully tuned, they can reach very high throughput. Most online services at Baidu do not have deterministic run time, and a search is often built by dozens of people. One slow function stalls every coroutine. Event loops have the same problem: one blocking callback stalls the whole loop. ub**a**server (note the **a**, not ubserver) was Baidu's attempt at an async framework of several parallel event loops. In practice it behaved poorly: a slightly slow log in a callback, a hiccup talking to Redis, or a bit more compute caused waiting requests to time out in bulk. The framework never caught on. + +bthread is an M:N threading library. One stalled bthread does not stall the others. The two key techniques are work-stealing scheduling and butex. The former schedules bthreads onto more cores quickly; the latter lets bthreads and pthreads wait for and wake each other. Neither is needed by a coroutine. See [threading overview](threading_overview.md) for more on threading models. + +##### Q: Should I create lots of bthreads in my program? + +No. Unless you need to [run some code concurrently inside one RPC](bthread_or_not.md), do not call bthread APIs directly. Leave that to brpc. + +##### Q: How do bthreads map onto pthread workers? + +A pthread worker runs exactly one bthread at a time. When the current bthread suspends, the worker first tries to pop a ready bthread from its local runqueue. If that is empty, it steals a ready bthread from a random other worker. If that also fails, it sleeps and is woken when a new ready bthread appears. + +##### Q: Can a bthread call blocking pthread or system functions? + +Yes. That only blocks the current pthread worker. Other pthread workers are unaffected. + +##### Q: Does one blocked bthread affect other bthreads? + +No. If the bthread blocks on a bthread API, it yields the current pthread worker to other bthreads. If it blocks on a pthread API or a system function, ready bthreads on that worker are stolen by idle pthread workers. + +##### Q: Can pthread code call bthread APIs? + +Yes. A bthread API called from a bthread affects the current bthread; called from a pthread, it affects the current pthread. Code that uses bthread APIs can run directly in a pthread. + +##### Q: If many bthreads call blocking pthread or system functions, does that hurt RPC? + +Yes. For example, with 8 pthread workers, if 8 bthreads all call `usleep()`, RPC code that handles network IO cannot run for a while. As long as the block is not too long, this usually **does not matter much**: the workers are all busy, and queuing is about the only option left. +In brpc you can raise the worker count to mitigate this: on the server set [ServerOptions.num_threads](server.md#number-of-worker-pthreads) or [-bthread_concurrency](flags.md); on the client set [-bthread_concurrency](flags.md). + +Is there a way to avoid this completely? + +- Dynamically adding workers is the obvious idea, but it often fails in practice. When many workers block at once, they are often waiting for the same resource (for example the same lock). Adding workers may only add more waiters. +- Split IO threads and worker threads? IO threads would only send and receive, so a fully blocked worker pool would not stall IO. An extra hop does not relieve congestion. If every worker is stuck, the program is still stuck; the stall just moves from the socket buffer to the queue between IO threads and workers. In other words, IO threads that keep running while workers are stuck may be doing useless work. That is what **does not matter much** above really means. Another cost is that every request jumps from an IO thread to a worker, adding a context switch. On a busy machine that switch is sometimes not scheduled promptly, which lengthens the tail latency. +- A practical fix is to [limit max concurrency](server.md#limit-concurrency). If the number of in-flight requests stays below the worker count, "all workers blocked" does not happen. +- Another fix: when blocked workers pass a threshold (for example 6 of 8), stop running user code in place and throw it into a separate thread pool. Even if all user code blocks, a few workers remain to handle RPC IO. bthread mode does not have this mechanism today, but a similar one is implemented when [pthread mode](server.md#pthread-mode) is on. Is that "useless work" while user code is fully blocked, as above? Possibly. The mechanism is more about avoiding a rare deadlock: all user code holds a pthread mutex that must be unlocked in an RPC callback; if every worker is blocked, nothing can run that callback and the process deadlocks. Most RPC implementations have this latent issue, but it is rare in practice. Do not issue RPCs while holding a lock, and you can avoid it. + +##### Q: Will bthread have [Channel](https://gobyexample.com/channels)? + +No. A channel models a relationship between two points, while many real problems are many-to-many. The natural channel solution is then: one role owns a thing or a resource, and every other thread sends commands to that role over a channel. Give the program N roles, each doing its job, and the program runs in an orderly way. So using channels implies splitting the program into roles. Channels are intuitive, but they cost extra context switches. Nothing finishes until the callee is scheduled, processes the message, and replies. No amount of cache-locality tuning removes that cost. Another reality: channel-heavy code is hard to write. Business consistency often binds resources together, so one role wears several hats, cannot do two things at once, and work has priorities. Interrupts, early exits, and resumes make the final code very complex. + +What we usually need is a buffered channel that acts as a queue with ordered execution. bthread provides [ExecutionQueue](execution_queue.md) for that. diff --git a/docs/en/bthread_id.md b/docs/en/bthread_id.md index ff67747ff0..52dd12d80a 100644 --- a/docs/en/bthread_id.md +++ b/docs/en/bthread_id.md @@ -1,7 +1,40 @@ -# uthread id +[中文版](../cn/bthread_id.md) -This document has not yet been translated into English. +# bthread_id -Please refer to the [Chinese version](../cn/bthread_id.md) for the full content. +`bthread_id` is a special synchronization structure. It can serialize different steps of an RPC, and it can find the RPC context (the Controller) in O(1) time. We are talking about `bthread_id_t` here, not `bthread_t` (the bthread tid). The name is unfortunate and easy to confuse. -Contributions to translate this document are welcome. See [TRANSLATING](TRANSLATING) for guidelines. +Concretely, `bthread_id` solves: + +- The response arriving while the request is still being sent, so response handling races with send code. +- A timer firing immediately after it is set, so timeout handling races with send code. +- Several responses from retries arriving at the same time and racing with each other. +- Finding the RPC context from a `correlation_id` in O(1) time, without a global hash map from `correlation_id` to context. +- Cancelling an RPC. + +Those bugs show up widely in other RPC frameworks. Here is how brpc uses `bthread_id` to close them. + +A `bthread_id` has two parts: a user-visible 64-bit id, and a hidden `bthread::Id` struct. User APIs all operate on the id. Mapping from id to struct is the same as [other structures](memory_management.md) in brpc: 32 bits are the pool offset, 32 bits are the version. The former locates in O(1); the latter avoids ABA. + +The `bthread_id` API is not small: + +- create +- lock +- unlock +- unlock_and_destroy +- join +- error + +The extra APIs exist to cover different flows. + +- Send request: `bthread_id_create` → `bthread_id_lock` → … register timer and send RPC … → `bthread_id_unlock` +- Receive response: `bthread_id_lock` → … process response → `bthread_id_unlock_and_destroy` +- Error handling: timeout / socket fail → `bthread_id_error` → run the `on_error` callback (which takes the lock). Then either: + - Retry / backup request: register timer and send RPC again → `bthread_id_unlock` + - Cannot retry, final failure: `bthread_id_unlock_and_destroy` +- Wait synchronously for the RPC to finish: `bthread_id_join` + +To cut waiting, `bthread_id` has a few extra mechanisms: + +- When an error arrives and the id is already locked, the error is pushed onto a pending queue and `bthread_id_error` returns immediately. On `bthread_id_unlock`, pending work is taken off the queue and run. +- When the RPC finishes and there is a user callback, first call `bthread_id_about_to_destroy` so in-flight `bthread_id_lock` waiters fail immediately, then run the user callback (which may take a long, unpredictable time), and finally `bthread_id_unlock_and_destroy`. diff --git a/docs/en/bthread_or_not.md b/docs/en/bthread_or_not.md index 2450c3c04a..52b0178cfc 100644 --- a/docs/en/bthread_or_not.md +++ b/docs/en/bthread_or_not.md @@ -1,7 +1,66 @@ -# uthread or not +[中文版](../cn/bthread_or_not.md) -This document has not yet been translated into English. +# bthread or not -Please refer to the [Chinese version](../cn/bthread_or_not.md) for the full content. +brpc provides an [asynchronous API](client.md#asynchronous-call), so a common question is: should I use the async API or bthread? -Contributions to translate this document are welcome. See [TRANSLATING](TRANSLATING) for guidelines. +Short answer: when latency is not high, start with the simple synchronous API. If that is not enough, use the async API. Use bthread only when you need parallel compute across cores. + +# Sync or async + +Async means replacing blocking with callbacks: wherever you would block, you get a callback. Callbacks work well in JavaScript and are widely accepted there, but that is a different kind of callback from what server code needs. The difference is not [lambda](https://en.wikipedia.org/wiki/Anonymous_function) or [future](https://en.wikipedia.org/wiki/Futures_and_promises); it is that JavaScript is single-threaded. Drop those callbacks into a multi-threaded program and few of them would even run — too much contention. Single-threaded sync and multi-threaded sync are completely different. Can a service look similar: several threads, each an independent event loop? Yes. ub**a**server (note the **a**) did that, and the result was poor. Turning blocking into callbacks is not simple. When the block sits inside a loop, a branch, or a deep helper, the rewrite is especially hard, and a lot of legacy or third-party code cannot be rewritten at all. Unavoidable blocking then delays every other callback on that thread, traffic times out, and the server misses its performance target. If someone says "I want to turn our sync code into a pile of callbacks that nobody else understands, and it might even be slower", most people will say no. Do not be sold by async evangelism written for programs that are async top to bottom and ignore multi-threading. That is not the code you have to write. + +Async in brpc is not single-threaded async. The callback runs on a different thread from the caller, so you get multi-core scalability, but you must deal with multi-threading. You can block inside the callback; as long as there are enough threads, overall server performance is fine. Async code is still hard to write, which is why we provide [combo channels](combo_channel.md): by composing channels you declare complex access patterns without sweating every detail. + +When latency is short and QPS is not high, we still recommend the sync API. That is also why bthread exists: keep synchronous code and still improve interactive performance. + +**Choosing sync or async**: compute `qps * latency` (latency in seconds). If the result is on the same order as the number of CPU cores, use sync; otherwise use async. + +Examples: + +- qps = 2000, latency = 10ms, result = 2000 * 0.01s = 20. Same order as a typical 32-core machine → sync. +- qps = 100, latency = 5s, result = 100 * 5s = 500. Not the same order as core count → async. +- qps = 500, latency = 100ms, result = 500 * 0.1s = 50. Roughly the same order → sync is OK. If latency keeps growing, consider async. + +The formula is the average number of in-flight requests (try proving it). It is comparable to thread count and CPU cores. When it is much larger than the core count, most operations are not burning CPU; they are parking a lot of threads. Async then saves thread resources (stack memory) in a visible way. When the value is at or below the core count, the thread-resource savings from async are small, and simple sync code matters more. + +# Async or bthread + +With bthread you can even implement async yourself. Take "semi-sync" as an example. In brpc you have several options: + +- Start several async RPCs and Join them one by one. Join blocks until the RPC finishes. (This is only for comparison with bthread. In real code we recommend [ParallelChannel](combo_channel.md#parallelchannel) instead of joining by hand.) +- Start several bthreads, each doing a sync RPC, then join the bthreads. + +Which is faster? The first. The second pays for creating bthreads, and those bthreads stay blocked for the whole RPC and cannot be used for anything else. + +**If you only need concurrent RPCs, do not use bthread.** + +Parallel compute is a different story. bthread makes it easy to build a tree of parallel work and use multiple cores. If a search has three stages that can run in parallel, start two bthreads for two stages, run the third in place, then join the two bthreads: + +```c++ +bool search() { + ... + bthread th1, th2; + if (bthread_start_background(&th1, nullptr, part1, part1_args) != 0) { + LOG(ERROR) << "Fail to create bthread for part1"; + return false; + } + if (bthread_start_background(&th2, nullptr, part2, part2_args) != 0) { + LOG(ERROR) << "Fail to create bthread for part2"; + return false; + } + part3(part3_args); + bthread_join(th1); + bthread_join(th2); + return true; +} +``` + +Notes: + +- You could start three bthreads and join all of them, but that costs one extra thread resource compared with running one stage in place. +- There is a delay from creating a bthread to running it (scheduling delay). On a machine that is not very busy, the median is about 3 microseconds, 90% finish within 10 microseconds, and 99.99% within 30 microseconds. Two consequences: + - The payoff is clear when the compute takes more than 1ms. If the work finishes in a few microseconds, bthread is not worth it. + - Run the slowest stage in place. Then even if the bthread stages are delayed by a few microseconds, they may still finish first, and the delay disappears. Joining an already-finished bthread returns immediately, with no context-switch cost. + +If you need something like a thread pool that runs one class of jobs, you can also replace the pool with bthreads. If job order matters, use bthread's [ExecutionQueue](execution_queue.md). diff --git a/docs/en/bthread_tagged_task_group.md b/docs/en/bthread_tagged_task_group.md index b509b31e33..a80e64f13c 100644 --- a/docs/en/bthread_tagged_task_group.md +++ b/docs/en/bthread_tagged_task_group.md @@ -1,7 +1,58 @@ -# uthread tagged task group +[中文版](../cn/bthread_tagged_task_group.md) -This document has not yet been translated into English. +# Bthread tagged task group -Please refer to the [Chinese version](../cn/bthread_tagged_task_group.md) for the full content. +Many applications need to isolate thread resources. For example a service has a control plane and a data plane, and heavy data-plane traffic should not starve the control plane. Or a service has several disks, and threads serving different disks should not interfere with each other. Tagging bthread task groups splits the bthread worker pool by tag so that groups do not affect one another. -Contributions to translate this document are welcome. See [TRANSLATING](TRANSLATING) for guidelines. +Tagging is per server. Put services of different groups on different servers; those servers listen on different ports. Some background or timer jobs have no service at all and still need their own pool. You can give those jobs a dedicated tag, and you control that pool's concurrency yourself. On top of that you can add policies such as pinning a tag to a NUMA node or setting thread-local variables. + +The implementation creates several worker groups at the bthread layer; each group runs the same logic as before. The bthread API adds a `tag` field on `bthread_attr_t`. The RPC layer adds `bthread_tag` on `brpc::ServerOptions` so a server can pick which worker group it runs on. + +# Usage + +`example/bthread_tag_echo_c++` has a sample. Start the server and clients separately. The server splits workers into 3 tags. `FLAGS_tag1` and `FLAGS_tag2` tag different servers. The remaining tag is for background jobs. + +```bash +# Server +./echo_server -task_group_ntags 3 -tag1 0 -tag2 1 -bthread_concurrency 20 -bthread_min_concurrency 8 -event_dispatcher_num 1 + +# Clients +./echo_client -dummy_port 8888 -server "0.0.0.0:8002" -use_bthread true +./echo_client -dummy_port 8889 -server "0.0.0.0:8003" -use_bthread true +``` + +`FLAGS_bthread_concurrency` is the total number of threads. `FLAGS_bthread_min_concurrency` is the lower bound across all groups. `FLAGS_event_dispatcher_num` is the number of event dispatchers in one group. `FLAGS_bthread_current_tag` is the tag whose size you are about to change, and `FLAGS_bthread_concurrency_by_tag` sets that group's thread count. + +Ordinary bthreads do not need to set `bthread_attr_t.tag`; they run in the current tag context. To run a bthread on another tag, set `bthread_attr_t.tag` to that value. That costs some performance and should be avoided on the hot path. + +Q: How do I change a group's thread count at runtime? + +A: You can size each group more freely for your service. At startup the pool is initialized from `bthread_concurrency`. If you set `bthread_min_concurrency`, that value is used instead. For a server, `num_threads` is the worker count of that tag. Change a group's size with `FLAGS_bthread_current_tag` and `FLAGS_bthread_concurrency_by_tag`. If those are unset (tagging is off, default `BTHREAD_TAG_INVALID`), `num_threads` means the total worker count across all groups. + +Q: How do groups relate to each other? + +A: They are independent thread pools and event dispatchers. They do not interact. + +Q: Can I synchronize bthreads across groups? + +A: Yes. Each bthread keeps its own tag. After it suspends and runs again, it continues on that tag's pool. + +Q: On which group does a client send and receive RPC messages? + +A: It depends on the client context. If the client is not on any tag, tag 0 is used; otherwise the current tag is used. + +Q: How do I bind a group's threads to specific CPUs? + +A: `int bthread_set_tagged_worker_startfn(void (*start_fn)(bthread_tag_t))` runs initialization on a group. You can implement CPU binding there, using the `tag` argument to pin different groups to different CPUs. + +# Monitoring + +Metrics split by tag today: thread count, thread usage, `bthread_count`, and connection info. + +Thread usage: ![img](../images/bthread_tagged_worker_usage.png) + +Dynamically changing tag 1's thread count: + +Set tag 1: ![img](../images/bthread_tagged_increment_tag1.png) + +Set all tags: ![img](../images/bthread_tagged_increment_all.png) diff --git a/docs/en/bthread_tracer.md b/docs/en/bthread_tracer.md index 6df0d8d742..35a4cd056a 100644 --- a/docs/en/bthread_tracer.md +++ b/docs/en/bthread_tracer.md @@ -1,7 +1,223 @@ -# uthread tracer +[中文版](../cn/bthread_tracer.md) -This document has not yet been translated into English. +gdb (`ptrace`) plus `gdb_bthread_stack.py` is slow and blocks the process. We need a cheaper way to trace a bthread call stack. -Please refer to the [Chinese version](../cn/bthread_tracer.md) for the full content. +brpc's cooperative userland threads cannot do an efficient STW (Stop The World) the way Go's preemptive goroutines can, and the framework cannot interrupt user logic. Tracing a bthread stack is therefore hard. -Contributions to translate this document are welcome. See [TRANSLATING](TRANSLATING) for guidelines. +Online tracing has to solve: + +1. Tracing a suspended bthread's call stack. +2. Tracing a running bthread's call stack. + +# bthread status model + +The current bthread status model: + +![bthread status model](../images/bthread_status_model.svg) + +# Design + +## Core idea + +To solve those two problems, this design implements STB (Stop The Bthread). While a bthread stack is being traced, the status must not move into a state that the current tracing method does not support. STB has two modes: context tracing and signal tracing. + +### Context tracing + +Context tracing covers suspended bthreads. A suspended stack is stable. Using the context saved in `TaskMeta.stack` (on x86_64 the important registers are mainly RIP, RSP, RBP), a library that can unwind a given context walks the stack. A suspended bthread may wake at any time, run (including `jump_stack`), and then the stack keeps changing. An unstable context cannot be unwound, so scheduling is intercepted before `jump_stack` and the bthread only continues after tracing finishes. Context tracing therefore supports the ready and suspended states. + +### Signal tracing + +Signal tracing covers running bthreads. A running bthread is unstable, so `TaskMeta.stack` cannot be used. Instead a signal interrupts the bthread and the signal handler unwinds the stack. Signals bring two issues: + +1. Async-signal-safety. +2. Signal tracing does not support `jump_stack`. Unwinding needs register state, and `jump_stack` mutates registers, so interrupting `jump_stack` is unsafe. Scheduling is intercepted before `jump_stack` and the bthread only suspends after tracing finishes. + +So this mode only supports the running state. + +### Summary + +`jump_stack` is on every path that suspends or runs a bthread, and it is the STB intercept point. STB splits states into three groups: + +1. Context-tracing states: ready, suspended. +2. Signal-tracing states: running. +3. Unsupported states. Neither method can unwind during `jump_stack`. Scheduling is intercepted before `jump_stack` and continues only after tracing finishes. + +### Flow + +After STB, two intercept states are added on top of the original model: about-to-run and about-to-suspend. + +![bthread STB status model](../images/bthread_stb_model.svg) + +STB flow: + +1. When TaskTracer (the STB module) receives a trace request, it marks tracing in progress. When tracing finishes, it marks completion and signals bthreads that may be in about-to-run or about-to-suspend. TaskTracer then branches on status: +- created, ready but no stack yet, destroyed: finish immediately. +- suspended, ready: context tracing. +- running: signal tracing. +- about-to-run, about-to-suspend: spin until the bthread moves to the next state (suspended or running), then continue. + +2. While TaskTracer is tracing, the bthread also branches on status: +- created, ready but no stack yet, ready: nothing extra. +- suspended, running: notify TaskTracer to continue. +- about-to-run, about-to-suspend, destroyed: wait on a condition variable until TaskTracer finishes. After that, TaskTracer wakes the bthread to continue `jump_stack`. + +# Usage + +1. Install libunwind and abseil-cpp. **Note: libunwind must be built from source. Do not use the distro package `libunwind-dev` / `libunwind-devel`**, or you will hit the crash in [Known issue: libunwind and libgcc_s `_Unwind_*` symbol conflict](#known-issue-libunwind-and-libgcc_s-_unwind_-symbol-conflict). Bazel builds can skip this step and use the libunwind version maintained in the brpc repo. +2. Pass `--with-bthread-tracer` to `config_brpc.sh`, or `-DWITH_BTHREAD_TRACER=ON` to cmake, or `--define with_bthread_tracer=true` to bazel (Bzlmod). +3. Hit the builtin service `http://ip:port/bthreads/?st=1`, or call `bthread::stack_trace()` in code. +4. To trace a pthread, call `bthread::init_for_pthread_stack_trace()` on that pthread to get a fake `bthread_t`, then use step 3. + +Example output: + +```shell +#0 0x00007fdbbed500b5 __clock_gettime_2 +#1 0x000000000041f2b6 butil::cpuwide_time_ns() +#2 0x000000000041f289 butil::cpuwide_time_us() +#3 0x000000000041f1b9 butil::EveryManyUS::operator bool() +#4 0x0000000000413289 (anonymous namespace)::spin_and_log() +#5 0x00007fdbbfa58dc0 bthread::TaskGroup::task_runner() +``` + +# Known issues + +## Known issue: libunwind and libgcc_s `_Unwind_*` symbol conflict + +### Symptom + +With bthread tracer enabled, you may see an occasional segfault on `bthread_exit` / `pthread_exit` or on a C++ exception path, with a stack like: + +```text +#0 0x0000000000000000 in ?? () +#1 0x00007fa2b5d6458a in _ULx86_64_dwarf_find_proc_info () + from /root/.cache/bazel/_bazel_root/743b333b2429a1dbd390ef66b59c771d/execroot/_main/bazel-out/k8-fastbuild/bin/test/../_solib_k8/libexternal_Slibunwind~_Slibunwind.so +#2 0x00007fa2b5d6668d in fetch_proc_info () + from /root/.cache/bazel/_bazel_root/743b333b2429a1dbd390ef66b59c771d/execroot/_main/bazel-out/k8-fastbuild/bin/test/../_solib_k8/libexternal_Slibunwind~_Slibunwind.so +#3 0x00007fa2b5d681a1 in _ULx86_64_dwarf_make_proc_info () + from /root/.cache/bazel/_bazel_root/743b333b2429a1dbd390ef66b59c771d/execroot/_main/bazel-out/k8-fastbuild/bin/test/../_solib_k8/libexternal_Slibunwind~_Slibunwind.so +#4 0x00007fa2b5d70cfd in _ULx86_64_get_proc_info () + from /root/.cache/bazel/_bazel_root/743b333b2429a1dbd390ef66b59c771d/execroot/_main/bazel-out/k8-fastbuild/bin/test/../_solib_k8/libexternal_Slibunwind~_Slibunwind.so +#5 0x00007fa2b5d6c775 in __libunwind_Unwind_GetLanguageSpecificData () + from /root/.cache/bazel/_bazel_root/743b333b2429a1dbd390ef66b59c771d/execroot/_main/bazel-out/k8-fastbuild/bin/test/../_solib_k8/libexternal_Slibunwind~_Slibunwind.so +#6 0x00007fa2b503c6df in __gxx_personality_v0 () from /lib/x86_64-linux-gnu/libstdc++.so.6 +#7 0x00007fa2b5452ce5 in ?? () from /lib/x86_64-linux-gnu/libgcc_s.so.1 +#8 0x00007fa2b54533c0 in _Unwind_ForcedUnwind () from /lib/x86_64-linux-gnu/libgcc_s.so.1 +#9 0x00007fa2b4ca57a4 in __GI___pthread_unwind (buf=) at ./nptl/unwind.c:130 +#10 0x00007fa2b4c9dd22 in __do_cancel () at ../sysdeps/nptl/pthreadP.h:271 +#11 __GI___pthread_exit (value=0x0) at ./nptl/pthread_exit.c:36 +#12 0x0000000000000000 in ?? () +``` + +### Root cause + +libunwind's `src/unwind/*.c` implements GCC's `_Unwind_*` ABI compatibility layer (`_Unwind_GetLanguageSpecificData`, `_Unwind_ForcedUnwind`, `_Unwind_Resume`, and so on), which exports the same global symbols as `libgcc_s.so.1`. When libunwind is linked as a **shared library** and appears before `libgcc_s.so.1` in the final binary's `DT_NEEDED` list, the runtime linker `ld.so` resolves `_Unwind_*` calls from `pthread_exit` / exception handling to libunwind's DWARF implementation. That implementation needs an internal context that bRPC has not initialized on the `pthread_exit` path, so you get a null-pointer access. + +This is an ELF **runtime symbol-resolution-order** issue. It is independent of the compiler (GCC / Clang). Clang's default runtime is also `libstdc++ + libgcc_s`, and it hits the same crash. + +### Workaround + +> **Important: do not use the distro libunwind** (for example `apt install libunwind-dev`, `yum install libunwind-devel`). Most distro `libunwind.so` builds still export `_Unwind_*` in the dynamic symbol table, which triggers the crash in this section. +> +> You must use **libunwind built from source**. Upstream `./configure` + `make` hides `_Unwind_*` as local via `-Wl,--version-script` by default, so they are not exported and the conflict goes away. + +Recommended approach per build system: + +| Build | Recommendation | +|---|---| +| `config_brpc.sh` + `make` | Build and install libunwind from source, then pass its include and lib dirs to `config_brpc.sh` | +| `cmake` | Build and install libunwind from source, then pass its include and lib dirs to `cmake` | +| `bazel` (Bzlmod) | Use the libunwind version maintained in the brpc repo | + +### make (config_brpc.sh) + +Build and install libunwind into a private prefix (do not pollute the system), then point `config_brpc.sh` at that prefix. + +```bash +# 1) Build libunwind from source (v1.8.1 or newer recommended) +git clone https://github.com/libunwind/libunwind.git +cd libunwind && git checkout tags/v1.8.1 +mkdir -p /opt/libunwind +autoreconf -i +./configure --prefix=/opt/libunwind +make -j$(nproc) && make install +cd .. + +# 2) Make config_brpc.sh use headers and libs under /opt/libunwind +# (do not let it pick up the system libunwind-dev) +cd brpc +sh config_brpc.sh \ + --with-bthread-tracer \ + --headers="/opt/libunwind/include /usr/include" \ + --libs="/opt/libunwind/lib /usr/lib /usr/lib64" +make -j$(nproc) +``` + +After the build, confirm `libunwind.so` does not export `_Unwind_*`: + +```bash +nm -D /opt/libunwind/lib/libunwind.so | grep ' T _Unwind_' \ + && echo "WARN: _Unwind_* exported" \ + || echo "OK: _Unwind_* hidden" +``` + +### cmake + +[`CMakeLists.txt`](../../CMakeLists.txt) looks up libunwind with `find_library(... NAMES unwind unwind-x86_64)`. Build libunwind from source into a private prefix as in the make section, then prefer that prefix with `CMAKE_PREFIX_PATH`: + +```bash +# 1) Build libunwind from source (same as the make section) + +# 2) Make cmake search headers and libs under /opt/libunwind first +cd brpc +mkdir build && cd build +cmake -DWITH_BTHREAD_TRACER=ON \ + -DCMAKE_PREFIX_PATH=/opt/libunwind \ + .. +make -j$(nproc) +``` + +> Tip: if `libunwind-dev` is already installed, `find_library` may still prefer `/usr/lib`. Pass +> `-DLIBUNWIND_LIB=/opt/libunwind/lib/libunwind.so -DLIBUNWIND_X86_64_LIB=/opt/libunwind/lib/libunwind-x86_64.so -DLIBUNWIND_INCLUDE_PATH=/opt/libunwind/include` +> to force the self-built copy. + +### bazel (Bzlmod) + +The brpc repo already maintains a Bzlmod overlay for libunwind under [`registry/modules/libunwind/`](../../registry/modules/libunwind/), used via `--registry=https://github.com/apache/brpc/registry` in [`.bazelrc`](../../.bazelrc). The version uses a `.brpc-no-unwind` suffix (for example `1.8.3.brpc-no-unwind`) to distinguish it from the same base version on BCR. The overlay adds a switch: + +``` +--define libunwind_hide_unwind_symbols=true +``` + +When on, libunwind's `src/unwind/*.c` (the GCC `_Unwind_*` compatibility layer) is not compiled, matching upstream autoconf's default. bRPC only uses libunwind's native `unw_*` API (`unw_getcontext`, `unw_init_local`, `unw_step`, and so on) and does not need the `_Unwind_*` layer, so the switch is safe. + +`.bazelrc` already turns this on for the `build:test` / `test` configs: + +``` +build:test --define libunwind_hide_unwind_symbols=true +test --define libunwind_hide_unwind_symbols=true +``` + +Add `--define=with_bthread_tracer=true` as in usage step 2: + +```bash +# Tests: the test config in .bazelrc already includes the hide switch +bazel test //test:bthread_unittest + +# Non-test (production) builds need both defines +bazel build --define=with_bthread_tracer=true \ + --define=libunwind_hide_unwind_symbols=true \ + //... +``` + +> **Note**: a production build that sets only `--define=with_bthread_tracer=true` and omits `--define=libunwind_hide_unwind_symbols=true` can crash on `pthread_exit` / exception paths. + +After the build, confirm the libunwind shared library does not export `_Unwind_*`: + +```bash +nm -D bazel-bin/external/_solib_*/libexternal*libunwind*.so 2>/dev/null \ + | grep ' T _Unwind_' || echo "OK: no _Unwind_* exported by libunwind.so" +``` + +# Related flags + +- `signal_trace_timeout_ms`: timeout for signal tracing, default 50ms. diff --git a/docs/en/execution_queue.md b/docs/en/execution_queue.md index 194251f15a..fcd8cd1663 100644 --- a/docs/en/execution_queue.md +++ b/docs/en/execution_queue.md @@ -1,7 +1,219 @@ -# execution queue +[中文版](../cn/execution_queue.md) -This document has not yet been translated into English. +# Overview -Please refer to the [Chinese version](../cn/execution_queue.md) for the full content. +Like kylin's ExecMan, [ExecutionQueue](https://github.com/apache/brpc/blob/master/src/bthread/execution_queue.h) runs tasks asynchronously and serially. The technique was first used in RPC to [write to the same fd from multiple threads](io.md#sending-messages). It was added to bthread after r31345. ExecutionQueue provides: -Contributions to translate this document are welcome. See [TRANSLATING](TRANSLATING) for guidelines. +- Async ordered execution: tasks run on a separate thread, strictly in submit order. +- Multi producer: several threads can submit to one ExecutionQueue at the same time. +- Cancel a submitted task +- Stop +- High-priority tasks that jump the queue + +Main differences from ExecMan: + +- Submit is [wait-free](https://en.wikipedia.org/wiki/Non-blocking_algorithm#Wait-freedom). ExecMan uses a lock, so when the machine is busy, one process being descheduled can block every thread. ExecutionQueue does not. +- Batching: the consumer can process submitted tasks in batches for better locality. After ExecMan finishes one AsyncClient's AsyncContext, the next task is often another client's, so the CPU cache keeps bouncing between those resources. +- The execute function is not pinned to a fixed thread. ExecMan hashes AsyncClient onto a fixed worker. Different ExecutionQueues are independent; with enough threads, every non-idle queue can run at once. With too few threads, ExecutionQueue cannot guarantee fairness — then you need to add bthread workers to raise overall capacity. +- The consumer runs as a bthread, so you can use bthread sync primitives without blocking a pthread. In ExecMan you should avoid primitives that are likely to block. + +# Background + +In multi-core programming, [message passing](https://en.wikipedia.org/wiki/Message_passing) is a common way to remove races. Logic is split by the resources it depends on into independent actors. Each actor owns its resource. Changing a resource becomes a message to that actor. The actor (usually in another context) applies the command, then either wakes the caller (sync) or submits to the next actor (async). + +![img](http://web.mit.edu/6.005/www/fa14/classes/20-queues-locks/figures/producer-consumer.png) + +# ExecutionQueue vs mutex + +Both ExecutionQueue and mutex can remove races among threads. Compared with a mutex, ExecutionQueue has these advantages: + +- Roles are clear, the idea is simple, and you do not have to reason about lock problems (such as deadlock). +- Task order is guaranteed; mutex wakeup order is not. +- Every thread is doing useful work; nobody waits. +- Under load or stalls, batching gives higher overall throughput. + +The drawbacks are equally real: + +- One flow's code is often scattered, so it is harder to read and maintain. +- To raise concurrency, one job is often pipelined across several ExecutionQueues, which hops between cores and pays extra scheduling and cache-sync cost. When the critical section is tiny, that cost is not negligible. +- Operating on several resources atomically gets harder. With mutexes you can lock several of them; with ExecutionQueue you need an extra dispatch queue. +- Everything is single-threaded on that queue, so a slow task blocks every other task on the same ExecutionQueue. +- Flow control is harder. A queue that caches too many tasks can use too much memory. + +Ignoring performance and complexity, any system can theoretically use only mutexes or only ExecutionQueues to remove races. For a complex system, pick per scenario: + +- If the critical section is tiny and contention is light, prefer a mutex, then use the [contention profiler](contention_profiler.md) to see whether it becomes a bottleneck. +- If you need ordered execution, or contention you cannot remove but can batch for throughput, choose ExecutionQueue. + +There is no universal multi-thread model. Combine profiling with the actual workload and balance complexity against performance. + +**One extra note**: an uncontended Linux mutex lock/unlock is only a few atomic instructions, and the cost is negligible in most cases. + +# Usage + +### Implement the execute function + +``` +// Iterate over the given tasks +// +// Example: +// +// #include +// +// int demo_execute(void* meta, TaskIterator& iter) { +// if (iter.is_queue_stopped()) { +// // destroy meta and related resources +// return 0; +// } +// for (; iter; ++iter) { +// // do_something(meta, *iter) +// // or do_something(meta, iter->a_member_of_T) +// } +// return 0; +// } +template +class TaskIterator; +``` + +### Start an ExecutionQueue + +``` +struct ExecutionQueueOptions { + ExecutionQueueOptions(); + + // Execute in resident pthread instead of bthread. default: false. + bool use_pthread; + + // Attribute of the bthread which execute runs on. default: BTHREAD_ATTR_NORMAL + // Bthread will be used when executor = nullptr and use_pthread == false. + bthread_attr_t bthread_attr; + + // Executor that tasks run on. default: nullptr + // Note that TaskOptions.in_place_if_possible = false will not work, if implementation of + // Executor is in-place(synchronous). + Executor * executor; +}; + +// Start a ExecutionQueue. If |options| is nullptr, the queue will be created with +// default options. +// Returns 0 on success, errno otherwise +// NOTE: type |T| can be non-POD but must be copy-constructible +template +int execution_queue_start( + ExecutionQueueId* id, + const ExecutionQueueOptions* options, + int (*execute)(void* meta, TaskIterator& iter), + void* meta); +``` + +The return value is a 64-bit id, a [weak reference](https://en.wikipedia.org/wiki/Weak_reference) to the ExecutionQueue instance. You can locate the queue wait-free in O(1). You can copy the id freely, even send it in an RPC as a handle to a remote resource. +You must keep `meta` alive until the ExecutionQueue has really stopped. + +### Stop an ExecutionQueue + +``` +// Stop the ExecutionQueue. +// After this function is called: +// - All the following calls to execution_queue_execute would fail immediately. +// - The executor will call |execute| with TaskIterator::is_queue_stopped() being +// true exactly once when all the pending tasks have been executed, and after +// this point it's ok to release the resource referenced by |meta|. +// Returns 0 on success, errno othrwise +template +int execution_queue_stop(ExecutionQueueId id); + +// Wait until the the stop task (Iterator::is_queue_stopped() returns true) has +// been executed +template +int execution_queue_join(ExecutionQueueId id); +``` + +`stop` and `join` can be called more than once and still behave reasonably. `stop` can be called at any time without worrying about thread safety. + +Like `close` on an fd, if `stop` is never called, the resource leaks forever. + +Safe time to free `meta`: when `execute` sees `iter.is_queue_stopped() == true`, or after `join` returns. Do not double-free. + +### Submit a task + +``` +struct TaskOptions { + TaskOptions(); + TaskOptions(bool high_priority, bool in_place_if_possible); + + // Executor would execute high-priority tasks in the FIFO order but before + // all pending normal-priority tasks. + // NOTE: We don't guarantee any kind of real-time as there might be tasks still + // in process which are uninterruptible. + // + // Default: false + bool high_priority; + + // If |in_place_if_possible| is true, execution_queue_execute would call + // execute immediately instead of starting a bthread if possible + // + // Note: Running callbacks in place might cause the dead lock issue, you + // should be very careful turning this flag on. + // + // Default: false + bool in_place_if_possible; +}; + +const static TaskOptions TASK_OPTIONS_NORMAL = TaskOptions(/*high_priority=*/ false, /*in_place_if_possible=*/ false); +const static TaskOptions TASK_OPTIONS_URGENT = TaskOptions(/*high_priority=*/ true, /*in_place_if_possible=*/ false); +const static TaskOptions TASK_OPTIONS_INPLACE = TaskOptions(/*high_priority=*/ false, /*in_place_if_possible=*/ true); + +// Thread-safe and Wait-free. +// Execute a task with defaut TaskOptions (normal task); +template +int execution_queue_execute(ExecutionQueueId id, + typename butil::add_const_reference::type task); + +// Thread-safe and Wait-free. +// Execute a task with options. e.g +// bthread::execution_queue_execute(queue, task, &bthread::TASK_OPTIONS_URGENT) +// If |options| is nullptr, we will use default options (normal task) +// If |handle| is not nullptr, we will assign it with the handler of this task. +template +int execution_queue_execute(ExecutionQueueId id, + typename butil::add_const_reference::type task, + const TaskOptions* options); +template +int execution_queue_execute(ExecutionQueueId id, + typename butil::add_const_reference::type task, + const TaskOptions* options, + TaskHandle* handle); + +template +int execution_queue_execute(ExecutionQueueId id, + T&& task); + +template +int execution_queue_execute(ExecutionQueueId id, + T&& task, + const TaskOptions* options); + +template +int execution_queue_execute(ExecutionQueueId id, + T&& task, + const TaskOptions* options, + TaskHandle* handle); + +``` + +High-priority tasks also run **strictly in submit order**, unlike ExecMan, where `QueueExecEmergent` AsyncContext order is undefined. That also means you cannot jump ahead of an already-submitted high-priority task. + +`in_place_if_possible` skips one thread schedule and cache sync when there is no contention. It can deadlock or recurse too deep (for example endless ping-pong). Turn it on only if your code does not have those problems. + +### Cancel a submitted task + +``` +/// [Thread safe and ABA free] Cancel the corresponding task. +// Returns: +// -1: The task was executed or h is an invalid handle +// 0: Success +// 1: The task is executing +int execution_queue_cancel(const TaskHandle& h); +``` + +A non-zero return only means ExecutionQueue has already handed the task to `execute`. The real logic may still cache that task in another container, so it does not mean the logical task is done. You have to guarantee that in your own code. diff --git a/docs/en/overview.md b/docs/en/overview.md index 0a704ddc6b..858daac9da 100644 --- a/docs/en/overview.md +++ b/docs/en/overview.md @@ -85,7 +85,7 @@ Although almost all RPC implementations claim that they're "high-performant", th * Reading and parsing requests from different clients is fully parallelized and users don't need to distinguish between "IO-threads" and "Processing-threads". Other implementations probably have "IO-threads" and "Processing-threads" and hash file descriptors(fd) into IO-threads. When a IO-thread handles one of its fds, other fds in the thread can't be handled. If a message is large, other fds are significantly delayed. Although different IO-threads run in parallel, you won't have many IO-threads since they don't have too much to do generally except reading/parsing from fds. If you have 10 IO-threads, one fd may affect 10% of all fds, which is unacceptable to industrial online services (requiring 99.99% availability). The problem will be worse when fds are distributed unevenly across IO-threads (unfortunately common), or the service is multi-tenancy (common in cloud services). In brpc, reading from different fds is parallelized and even processing different messages from one fd is parallelized as well. Parsing a large message does not block other messages from the same fd, not to mention other fds. More details can be found [here](io.md#receiving-messages). * Writing into one fd and multiple fds is highly concurrent. When multiple threads write into the same fd (common for multiplexed connections), the first thread directly writes in-place and other threads submit their write requests in [wait-free](https://en.wikipedia.org/wiki/Non-blocking_algorithm#Wait-freedom) manner. One fd can be written into 5,000,000 16-byte messages per second by a couple of highly-contended threads. More details can be found [here](io.md#sending-messages). -* Minimal locks. High-QPS services can utilize all CPU power on the machine. For example, [creating bthreads](../cn/memory_management.md) for processing requests, [setting up timeout](../cn/timer_keeping.md), [finding RPC contexts](../cn/bthread_id.md) according to response, [recording performance counters](bvar.md) are all highly concurrent. Users see very few contentions (via [contention profiler](../cn/contention_profiler.md)) caused by RPC framework even if the service runs at 500,000+ QPS. -* Server adjusts thread number according to load. Traditional implementations set number of threads according to latency to avoid limiting the throughput. brpc creates a new [bthread](../cn/bthread.md) for each request and ends the bthread when the request is done, which automatically adjusts thread number according to load. +* Minimal locks. High-QPS services can utilize all CPU power on the machine. For example, [creating bthreads](../cn/memory_management.md) for processing requests, [setting up timeout](../cn/timer_keeping.md), [finding RPC contexts](bthread_id.md) according to response, [recording performance counters](bvar.md) are all highly concurrent. Users see very few contentions (via [contention profiler](../cn/contention_profiler.md)) caused by RPC framework even if the service runs at 500,000+ QPS. +* Server adjusts thread number according to load. Traditional implementations set number of threads according to latency to avoid limiting the throughput. brpc creates a new [bthread](bthread.md) for each request and ends the bthread when the request is done, which automatically adjusts thread number according to load. Check [benchmark](../cn/benchmark.md) for a comparison between brpc and other implementations.