commit 222235fa1c8052829cacc3f706f1517f57ba23bec78519ce0c103b627882bc54
| author | 斟酌 鵬兄 <tgckpg@gmail.com> |
|---|
| date | 2026-05-10T23:56:55Z |
|---|
| subject | Added cmm docs |
|---|
commit 222235fa1c8052829cacc3f706f1517f57ba23bec78519ce0c103b627882bc54
Author: 斟酌 鵬兄 <tgckpg@gmail.com>
Date: 2026-05-10T23:56:55Z
Added cmm docs
---
README.md | 8 ++
docs/cmm.md | 320 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 328 insertions(+)
diff --git a/README.md b/README.md
index b27373b..46b3834 100644
--- a/README.md
+++ b/README.md
@@ -143,6 +143,14 @@ Tested worker node upgrade chain:
---
+## CMM
+
+Based on vendor's ASK
+
+See:
+
+- [Running cmm](docs/cmm.md)
+
## Current status
This project is usable for experimenting with a single control-plane device image, but it is still a development project.
diff --git a/docs/cmm.md b/docs/cmm.md
new file mode 100644
index 0000000..7596d49
--- /dev/null
+++ b/docs/cmm.md
@@ -0,0 +1,320 @@
+# CMM integration for monok8s
+
+This document describes how monok8s runs the vendor Connection Manager daemon (`cmm`) from [ASK](https://github.com/we-are-mono/ASK/) on Kubernetes nodes.
+
+`cmm` is part of the NXP/ASK hardware-offload stack. In the vendor layout it is normally started as a boot-time service, together with the `cdx` kernel module and `dpa_app`. monok8s intentionally does **not** follow that model. Kubernetes has priority: the node should boot, kubelet should come up, CNI should configure networking, and only then should the CMM stack start from a DaemonSet.
+
+## Startup model
+
+The intended startup order is:
+
+1. The node boots.
+2. `kubelet` starts.
+3. CNI is configured.
+4. The `cmm` DaemonSet starts on the node.
+5. The DaemonSet prepares the DPA/CDX runtime and starts `cmm` in the foreground.
+
+This is different from the vendor flow, where CMM-related components are treated as host services started early during boot. That flow is a poor fit for monok8s because CNI and Kubernetes-owned networking must win any ordering conflict.
+
+## Local changes from vendor ASK
+
+monok8s carries a small set of patches so the ASK CMM stack behaves correctly inside a Kubernetes pod.
+
+### `cmm`
+
+The `cmm` daemon is patched to:
+
+- run in the foreground, so it can be supervised directly by Kubernetes;
+- log to stdout/stderr, so logs are visible through `kubectl logs`;
+- avoid exiting when it sees CNI-managed conntrack entries it does not understand.
+
+The last item is important. On a Kubernetes node, conntrack is not exclusively owned by CMM. CNI, kubelet, host networking, and ordinary pods can all create conntrack entries. CMM must tolerate that environment.
+
+### `cdx` kernel module
+
+The `cdx` module is patched so loading the module does **not** automatically start `dpa_app`.
+
+In monok8s, module loading and DPA configuration are separate steps. This avoids doing device configuration too early, before Kubernetes networking is ready.
+
+### `dpa_app`
+
+`dpa_app` is patched so the XML config paths can be supplied through environment variables:
+
+- `CDX_CFG_FILE`
+- `CDX_PCD_FILE`
+- `CDX_PDL_FILE`
+- `CDX_SP_FILE`
+
+This lets the DaemonSet select different XML files per node, board, or port layout without rebuilding the image.
+
+## Patch locations
+
+The relevant patch sets are under:
+
+```text
+patches/ask/upstream/libnetfilter-conntrack
+patches/ask/cmm
+patches/ask/cdx
+patches/ask/dpa
+```
+
+Other ASK patches in the tree are mostly kernel-porting work for the target NXP LSDK kernel, including the 6.18-based kernel used by monok8s.
+
+## Installation
+
+CMM is **not installed by default**. Install it explicitly after the node-control components are available.
+
+With `MKS_ENABLE_NODE_CONTROL` enabled, generate and apply the CMM manifests with:
+
+```sh
+kubectl -n mono-system exec -it ds/node-agent -- ctl create cmm | kubectl apply -f -
+```
+
+This creates the CMM DaemonSet and the supporting objects required to run it on each matching node.
+
+Check that the pod is running:
+
+```sh
+kubectl -n mono-system get pods -l app.kubernetes.io/name=cmm -owide
+```
+
+View logs with:
+
+```sh
+kubectl -n mono-system logs ds/cmm -f
+```
+
+If the DaemonSet name or labels change, inspect the generated YAML from `ctl create cmm` and use the actual object names.
+
+## Accessing the CMM CLI
+
+The CMM CLI is exposed from the pod for debugging and manual inspection. The DaemonSet uses `hostNetwork: true`, but the safest access method is still `kubectl port-forward`; it avoids exposing the CLI beyond your local machine.
+
+First find a CMM pod:
+
+```sh
+kubectl -n mono-system get pods -l app.kubernetes.io/name=cmm
+```
+
+Then forward a local port to the CMM CLI port inside the pod. Kubernetes port-forward syntax is `LOCAL_PORT:REMOTE_PORT`.
+
+For example, if CMM listens on port `12345` in the pod:
+
+```sh
+kubectl -n mono-system port-forward pod/cmm-xxxxx 12345:2103
+```
+
+In another terminal, connect to the local forwarded port:
+
+```sh
+telnet 127.0.0.1 12345
+```
+
+Use `telnet` for this CLI. Plain `ncat` can show leading garbage characters or behave badly with the login prompt because the CMM CLI behaves like a telnet-style interactive console rather than a clean raw TCP protocol.
+
+Default login, if unchanged by the generated config, is usually:
+
+```text
+Username: admin
+Password: admin
+```
+
+Do not expose this port through a Service or LoadBalancer unless you have added proper access control. Treat the CMM CLI as an operator/debug interface.
+
+## Configuration
+
+`ctl create cmm` emits a default configuration suitable for the expected monok8s hardware layout. You can override the generated YAML before applying it.
+
+The vendor's original `fastforward` config is preserved in the image as a reference file, but monok8s uses its own runtime config. Keep those roles separate:
+
+- vendor reference config: useful for comparison and debugging;
+- monok8s runtime config: the config actually consumed by the DaemonSet.
+
+A clear filename for the preserved vendor file is:
+
+```text
+fastforward.vendor.orig
+```
+
+That name is less project-specific than `fastforward.ask.orig` and makes the intent obvious: it is the original vendor-provided config, not the active config.
+
+## Multi-node configuration
+
+If all nodes have the same board and port layout, one shared CMM/DPA config is enough.
+
+If nodes have different port layouts, use node-specific XML config. The recommended pattern is:
+
+1. Mount all supported configs into the CMM pod.
+2. Pass the Kubernetes node name into the pod.
+3. Run a small wrapper script before `dpa_app`.
+4. The wrapper selects the XML files for the current node and exports the corresponding `CDX_*` environment variables.
+5. The wrapper then execs the normal DPA initialization script.
+
+Example DaemonSet fragment:
+
+```yaml
+spec:
+ template:
+ spec:
+ initContainers:
+ - name: dpa-app
+ image: localhost/monok8s/cmm:dev
+ imagePullPolicy: Never
+ command:
+ - /node-config/select-dpa-config.sh
+ env:
+ - name: NODE_NAME
+ valueFrom:
+ fieldRef:
+ fieldPath: spec.nodeName
+ volumeMounts:
+ - name: node-config
+ mountPath: /node-config
+ readOnly: true
+ - name: dpa-configs
+ mountPath: /etc/monok8s/dpa-configs
+ readOnly: true
+ volumes:
+ - name: node-config
+ configMap:
+ name: cmm-node-config-wrapper
+ defaultMode: 0755
+ - name: dpa-configs
+ configMap:
+ name: cmm-dpa-configs
+```
+
+Example wrapper:
+
+```sh
+#!/bin/sh
+set -eu
+
+CONFIG_DIR="/etc/monok8s/dpa-configs/${NODE_NAME}"
+
+if [ ! -d "${CONFIG_DIR}" ]; then
+ echo "missing DPA config directory for node ${NODE_NAME}: ${CONFIG_DIR}" >&2
+ exit 1
+fi
+
+export CDX_CFG_FILE="${CONFIG_DIR}/cdx_cfg.xml"
+export CDX_PCD_FILE="${CONFIG_DIR}/cdx_pcd.xml"
+export CDX_PDL_FILE="${CONFIG_DIR}/hxs_pdl_v3.xml"
+export CDX_SP_FILE="${CONFIG_DIR}/cdx_sp.xml"
+
+exec /bin/init_dpa.sh
+```
+
+The ConfigMap layout should then look like this conceptually:
+
+```text
+/etc/monok8s/dpa-configs/
+ node-a/
+ cdx_cfg.xml
+ cdx_pcd.xml
+ hxs_pdl_v3.xml
+ cdx_sp.xml
+ node-b/
+ cdx_cfg.xml
+ cdx_pcd.xml
+ hxs_pdl_v3.xml
+ cdx_sp.xml
+```
+
+For production, prefer a naming scheme based on stable node labels or hardware profiles rather than raw node names if multiple nodes share the same layout. Raw node names are fine for early bring-up, but they do not scale well.
+
+## Operational notes
+
+### CMM and Kubernetes conntrack
+
+Do not assume CMM owns the full conntrack table. Kubernetes nodes contain conntrack entries from:
+
+- CNI traffic;
+- kubelet;
+- host-network pods;
+- service routing;
+- node-local traffic;
+- ordinary workloads.
+
+CMM must tolerate unknown entries. If it exits because it encountered a CNI or Kubernetes conntrack entry, that is a bug in the integration layer, not an operator error.
+
+### `hostNetwork: true`
+
+The CMM pod uses `hostNetwork: true` because it needs to interact with host networking and hardware-offload state. This also means any port bound by the pod may be bound in the host network namespace.
+
+For the CLI, prefer `kubectl port-forward` anyway. It gives you a controlled local tunnel and avoids accidentally publishing the CLI on the node network.
+
+### `CMM_MAX_CONNECTIONS`
+
+The default value:
+
+```sh
+CMM_MAX_CONNECTIONS="${CMM_MAX_CONNECTIONS:-131072}"
+```
+
+uses `131072`, which is `128 * 1024`. It is a power-of-two-sized default commonly used for connection-table limits. Treat it as a capacity default, not a magic correctness value.
+
+Lower it if memory pressure is a concern. Raise it only if the hardware, memory budget, and expected traffic justify it.
+
+## Troubleshooting
+
+### The CMM pod is not running
+
+Check the DaemonSet and pod events:
+
+```sh
+kubectl -n mono-system get ds cmm -oyaml
+kubectl -n mono-system describe pod -l app.kubernetes.io/name=cmm
+```
+
+Then check logs:
+
+```sh
+kubectl -n mono-system logs ds/cmm --all-containers=true --tail=200
+```
+
+### The CLI shows strange characters with `ncat`
+
+Use `telnet` instead:
+
+```sh
+telnet 127.0.0.1 2103
+```
+
+The CMM CLI behaves like a telnet-style console. `ncat --telnet` may still not behave exactly like traditional telnet for this CLI.
+
+### Port-forward connects to the wrong port
+
+Remember the syntax:
+
+```text
+LOCAL_PORT:REMOTE_PORT
+```
+
+So this command:
+
+```sh
+kubectl -n mono-system port-forward pod/cmm-xxxxx 12345:2103
+```
+
+means:
+
+```text
+127.0.0.1:12345 on your workstation -> port 2103 inside the pod
+```
+
+Connect to `127.0.0.1:12345`, not `127.0.0.1:2103`.
+
+### `dpa_app` uses the wrong XML files
+
+Confirm the environment seen by the init container or wrapper:
+
+```sh
+kubectl -n mono-system logs pod/cmm-xxxxx -c dpa-app
+```
+
+The wrapper should print enough information to identify the selected config directory and XML paths. If it does not, add explicit logging before `exec /bin/init_dpa.sh`.
+
+## Recommended policy
+
+Keep CMM optional. It is hardware-specific, operationally sharp, and not required for a generic Kubernetes node. The base monok8s node should boot and join the cluster without CMM. Enable CMM only on hardware profiles where the ASK offload stack is expected and tested.