Vehicle Researcher
59e83e4c27
b8382bbb steerLimitTimer should be car dependent
9a229687 add pa0 temp to ThermalData
f6f0f60e Add stock Fcw to carState
b608683f no l/r distinction for LDW
555f48d6 Add ldw alert
8e8b4a4a Remove plusFrame socket in favor of UiLayoutState
3410325c log stock AEB events
2219f2bd Add warning about not using cython version of sec_since_boot
8f1a5122 for legacy-testing reasons, better to define the used percent instead of avail
e86d9545 adding low memory event
ad238340 remove TODO
d0962b34 log mem available and cpu perc in thermald
3b753be9 Implement error handling and exceptions (#18)
a7d5bb76 add explicit dependencies on services.h
1ba64677 fix linter
c7d215b6 Added communityFeatureDisallowed event
492140a5 Added communityFeature bit detection to CarParams
266a5fed log Panda fault types
347a8661 Switch from polling on FIFOs to signal (#12)
e25bba77 no need to double build the objects
fe43a994 20Hz for radar time step is very standard
2aabf1ee Added radar time step to car params
e8ae9086 Generate capnp for java
57126a23 cereal_shared
da655cd3 Add uptime to health
f6a8e394 add test with multiple subscribers
84b3af53 comment out the debugging
4b9c942a added power save state to health packet
66be3708 run python unittest in ci
52c6db87 Run scons in CI (#14)
9414615b do need it, but only for arm
2856c37c remove gnustl_shared
7f05ee64 fix apks
e3a6bded Revert "no more makefiles"
487fbd06 don't rely on BASEDIR, and add zmq library
223e37a5 no more makefiles
da2ed115 don't link the wrong one
fe9fe2a2 scons builds the python lib now
2f81135e err, it can't build services.h
57b03f8b now we shouldn't need that yaml crap everywhere
f8e53277 bridge builds with services.h
2b0cb608 noOutput safety mode is now called silent
83880d51 add msgq tests
bcad1848 msgq: dont block when fifo does not exists
b4b26782 Default to zmq
473e2912 fix compilation in docker
30aaaddc msgq: try again when no timeout on poll but also no message
c4f2ad53 msgq: make sure read_fifos is initalized so we dont close random fds
4e513a85 msgq: dont clean up uninitialized sockerts
c008b630 also remove the fifo from disk
ef64eb27 MSGQ stability improvements when opening and closing lots of queues
e147abcc Revert "Revert "deprecate irpwr""
932dc32e Revert "deprecate irpwr"
a6844150 disengage
ec27e18c capnpc also generated the header files
ee52ab9e deprecate irpwr
301c74c8 Merge branch 'master' of github.com:commaai/cereal
6da7d55a add front frame
a5944eb4 add conflate parameter for SubSocket::create
ca8df170 Add fault status to health
ef4ded06 add conflate support in SubSocket constructor
7fd314af update scons build file
93d814e4 add saturated flags to indi and lqr logs
50302fee add steeringRateLimited to car.capnp
05e3513d add msgq readme
a6759a95 faster make
94b73778 Add struct to log FW version
64ce0b5f add scons build
dc9ad18a add debug print statement on SIGINT
4a612698 Merge pull request #10 from commaai/msgq
4873449a use recv one or none after poll
a054864b default to msgq
fbc4a4cf oops bad number
5067cf4c add meta
cbd02865 fix export prefix and make shared library world readable
c2730541 add c exports for jni usage
e77f41ef zmq already sets the errno correctly
3196cf69 Fix service list path in bridge
d35515a2 add all msgq files, but dont use as default
a68a38fa Don't delete context from python side only
bd46c225 Revert "zmq_ctx_term is blocking"
a1fc26b8 zmq_ctx_term is blocking
09021820 remote address support
21a35361 only delete subsocket when created by same object
34df7351 remove extra underscore from __dealloc__
c8748f86 fix internal refs
79b2fbf7 fixups
23ad2563 import messaging and services
git-subtree-dir: cereal
git-subtree-split: b8382bbb2b8156f2f1d7e1c1b42b46c54d85761f
old-commit-hash: e3b21173a5
3.7 KiB
MSGQ: A lock free single producer multi consumer message queue
What is MSGQ?
MSGQ is a system to pass messages from a single producer to multiple consumers. All the consumers need to be able to receive all the messages. It is designed to be a high performance replacement for ZMQ-like SUB/PUB patterns. It uses a ring buffer in shared memory to efficiently read and write data. Each read requires a copy. Writing can be done without a copy, as long as the size of the data is known in advance.
Storage
The storage for the queue consists of an area of metadata, and the actual buffer. The metadata contains:
- A counter to the number of readers that are active
- A pointer to the head of the queue for writing. From now on referred to as write pointer
- A cycle counter for the writer. This counter is incremented when the writer wraps around
- N pointers, pointing to the current read position for all the readers. From now on referred to as read pointer
- N counters, counting the number of cycles for all the readers
- N booleans, indicating validity for all the readers. From now on referred to as validity flag
The counter and the pointer are both 32 bit values, packed into 64 bit so they can be read and written atomically.
The data buffer is a ring buffer. All messages are prefixed by an 8 byte size field, followed by the data. A size of -1 indicates a wrap-around, and means the next message is stored at the beginning of the buffer.
Writing
Writing involves the following steps:
- Check if the area that is to be written overlaps with any of the read pointers, mark those readers as invalid by clearing the validity flag.
- Write the message
- Increase the write pointer by the size of the message
In case there is not enough space at the end of the buffer, a special empty message with a prefix of -1 is written. The cycle counter is incremented by one. In this case step 1 will check there are no read pointers pointing to the remainder of the buffer. Then another write cycle will start with the actual message.
There always needs to be 8 bytes of empty space at the end of the buffer. By doing this there is always space to write the -1.
Reset reader
When the reader is lagging too much behind the read pointer becomes invalid and no longer points to the beginning of a valid message. To reset a reader to the current write pointer, the following steps are performed:
- Set valid flag
- Set read cycle counter to that of the writer
- Set read pointer to write pointer
Reading
Reading involves the following steps:
- Read the size field at the current read pointer
- Read the validity flag
- Copy the data out of the buffer
- Increase the read pointer by the size of the message
- Check the validity flag again
Before starting the copy, the valid flag is checked. This is to prevent a race condition where the size prefix was invalid, and the read could read outside of the buffer. Make sure that step 1 and 2 are not reordered by your compiler or CPU.
If a writer overwrites the data while it's being copied out, the data will be invalid. Therefore the validity flag is also checked after reading it. The order of step 4 and 5 does not matter.
If at steps 2 or 5 the validity flag is not set, the reader is reset. Any data that was already read is discarded. After the reader is reset, the reading starts from the beginning.
If a message with size -1 is encountered, step 3 and 4 are replaced by increasing the cycle counter and setting the read pointer to the beginning of the buffer. After that another read is performed.