Heterogeneous Inter-Processor Communication (HIPC)

Introduction

Arm Zena CSS contains Armv9-A (Primary Compute) and Armv8-R AArch64 (Safety Island) heterogeneous processing elements, which share data via the Message Handling Unit (MHUv3) and shared Static Random Access Memory (SRAM). The MHUv3 is a mailbox controller used for signal transmission and the shared memory is used for data exchange.

On Arm Zena CSS platform, Zephyr is deployed on Safety Island Cluster 1 (SI CL1) in RD-Aspen CFG2 variant.

HIPC communication is implemented between the Primary Compute and Safety Island CL1 in RD-Aspen CFG2 variant.

HIPC Baremetal Architecture

Remote Processor Messaging protocol

Remote Processor Messaging (RPMsg) is a messaging protocol enabling heterogeneous communication, which can be used by Linux as well as Real Time Operating Systems.

In Linux, the RPMsg framework is implemented on top of the Virtio-RPMsg bus and remoteproc framework. The Virtio-RPMsg implementation is generic and based on Virtio Vring to transmit/receive messages to/from the remote CPU over shared memory.

On the Safety Island side, Zephyr has imported OpenAMP as an external module. The OpenAMP library implements the RPMsg backend based on Virtio, which is compatible with the upstream Linux remoteproc and RPMsg components. You can use this library along with the Zephyr kernel or Zephyr applications to behave as an RPMsg backend service for communication with the Primary Compute.

Virtual network device over RPMsg

RPMsg offers a range of user APIs for RPMsg endpoints to send and receive messages. These APIs are suitable for simple inter-processor communication. However, many current user applications are not built on RPMsg APIs. Instead, they use BSD sockets for IPC. The reason for this is that BSD sockets can abstract the difference between inter-processor communication and intra-processor communication. This makes it possible for applications to be more versatile and portable. In response to the needs of such applications, a virtual network device based on RPMsg has been added to the Reference Software Stack.

On the Safety Island side, a network device is created over an RPMsg endpoint with a specific service name. The RPMsg endpoint sends a Name Service message to the Primary Compute to announce its existence. The message is then processed by the RPMsg bus, which creates an RPMsg endpoint and a corresponding network device. When this is done, the virtual network devices establish network communication.

On the Primary Compute side, RPMsg frames must be copied to the Socket Buffer (skb) utilized by the Network Stack. However, if the traffic exceeds the performance limit, the Socket Buffer might get dropped during processing for congestion control or by the protocol layers. In such cases, the network statistics will show an increased dropped packet counter.

In the above diagram, Primary Compute and Safety Island CL1 use shared memory for message exchange. They use the MHUv3 device for signaling when new messages are available. The size of the shared memory is 512KB. The shared memory instance has a resource table (128KB), Vring 0, Vring 1 (128KB each), and an RPMsg Virtio Buffer (128KB) used to send and receive information between the Primary Compute and the Safety Island CL1.

On the Primary Compute, the Safety Island remoteproc driver and RPMsg-based virtual interface driver are integrated to communicate with the Safety Island CL1. The RPMsg-net driver on the Primary Compute and Veth-RPMsg on the Safety Island CL1 implement the virtual ethernet device that is the basis for communication between the Primary Compute and Safety Island CL1.

Safety Island remoteproc driver

The remoteproc framework allows different platforms/architectures to control (power on/off, load firmware) remote processors while abstracting the hardware differences, so the entire driver does not need to be duplicated. The remoteproc platform driver is added to the software stack to provide support for communication between Primary Compute and Safety Island CL1.

In RD-Aspen, Linux running in the Primary Compute regards the Safety Island CL1 as its remote processor. The Safety Island CL1 behaves as an independent entity and has its own resources to establish the connection to the Primary Compute.

The Safety Island CL1 is not booted by the Primary Compute processor. Therefore, the initial status of the Safety Island CL1 in the driver is RPROC_DETACHED , which means the Safety Island CL1 has been booted independently from the Primary Compute processor. This driver implements the notification handler using an MHUv3-based mailbox, which notifies other cores when new messages are sent to the virtual queue.

The resource table, Vring 0, Vring 1, and RPMsg Virtio Buffer memory regions are defined in the Linux device tree bindings of the Safety Island CL1 (see Device tree). The remoteproc driver reads the device tree node of the Safety Island CL1 and adds it to the remoteproc framework. The Safety Island CL1 has its own resource table, Vring 0 , Vring 1, and RPMsg Virtio Buffer, which serve as the foundation for communication.

Memory map

The AP shared SRAM used by the Primary Compute and Safety Island CL1 for inter-processor data transfer has the following memory regions: resource table, Vring 0, Vring 1, and Virtio Buffer.

Safety Island Cluster 1:

rsc_table : Used to share resource information between Primary Compute and Safety Island CL1

shared_data : Used for data transfer between Primary Compute and Safety Island CL1

Primary Compute:

si_cl1_rproc_rsctbl : Used to share resource information between Primary Compute and Safety Island CL1

si_cl1_vdev0vring0 : Primary Compute vring, used to pass messages from Safety Island CL1 to Primary Compute

si_cl1_vdev0vring1 : Safety Island CL1 vring, used to pass messages from Primary Compute to Safety Island CL1

si_cl1_vdev0buffer : Used for data transfer between Primary Compute and Safety Island CL1

Network topology

VLAN

Open vSwitch is used to create a virtual switch that connects all the network interfaces of the Primary Compute.

VLAN is a concept standardized by IEEE 802.1Q. It is used to partition a switch into multiple logical switches. The VLAN tag has a value from 0 to 4095 stored in the packet header. Usually 0 means that the packet is untagged, but some values are reserved.

On a switch, using VLAN-tagged traffic makes sure that a packet tagged with a certain VLAN identifier reaches only ports that are configured to manage the traffic tagged with that identifier (tag).

The traffic between the Primary Compute and Safety Island CL1 uses VLAN 200.

The diagram below shows the network topology for the Baremetal Architecture. The ethsi1 interface is an RPMsg-based virtual interface that connects the Primary Compute to Safety Island CL1. The ovsbr0 bridge is the Open vSwitch network switch that carries untagged traffic. Communication between the Primary Compute and Safety Island CL1 is managed through the brsi1 VLAN-tagged switch, which carries VLAN-tagged traffic between ovsbr0 and the ethsi1 interface.

User space applications on the Primary Compute can communicate with Safety Island CL1 via brsi1.

Fig. 48 HIPC Network Topology - Baremetal

There are some limitations for HIPC. For more information, see the Limitations section in the release notes.

Device tree

In Linux, a remoteproc binding is defined for Safety Island CL1. It includes MHUv3 transmit and receive channels for signaling, along with memory regions used for data exchange between the Primary Compute and Safety Island CL1.

In Zephyr, there is an overlay device tree for the network over RPMsg application, which also defines the MHUv3 channels and device memory regions.