infamy: Always send logs from DUTs to remote syslog in tests

[mattiaswal]

Description

When tests fail in the CI it is often impossible to debug it after the run, to solve this i suggest to always send the syslog as remote syslog

Store the logs per test and per DUT, to be able to be able to trace errors from CI.

1. test-config always setup a syslog to the test PC (test container) over the mgmt interface (IPv6)
2. Test runs
3. Infamy "collect" and package the logs
4. Repeat

Example content:

logs/
   test1/
      dut1/
         kern.log
         messages
         syslog
      dut2/
         kern.log
         messages
         syslog
      dut3/
         kern.log
         messages
         syslog
   test2/
      dut1/
         kern.log
         messages
         syslog

Another thing you could log, without overhead, is the operational state gotten (for each test) hook into get_data() (or what it is called) when getting operational, save it to the test directory. To ease debug.

then compress it to logs.tar.gz, if running in CI, publish it as an artifact.

Additional Information

No response

General Information

Anyone can help out by sponsoring development of new features or contributing pull requests.
Please use this issue for discussions related to the feature.

[troglobit]

Notes from brainstorming session

• Log from DUTs to multicast group?
• Log from DUTs using *.* @MCGROUP ; RFC5424,iface=eth0
• Test container runs syslogd -H -n to prevent resolving the DUT's IP
• We need Add RPC for injecting log messages to a device #1639
• Infamy can switch the test log symlink (/path/to/<test>) when a new test starts

Idea for /etc/syslog.d/dut.conf in test container:

*.*                           -/path/to/<test>/log/full
:source, isequal, "fe80:..."
*.*                           -/path/to/<test>/log/dutN.log

Log markers from Infamy (follow-up from #1639)

#1639 adds the infix-syslog:log RPC and an Infamy helper, target.log(), so tests can stamp start/stop/step markers in a DUT's log over the management API. The RPC carries the full RFC 5424 header, including structured data, which is the natural place for test name, step, and DUT role when the logs are collected per test and per DUT as proposed here.

Two things to settle when this issue is picked up:

1. SD-ID. RFC 5424 private structured-data ids have the form name@enterprise-number. KernelKit has no IANA Private Enterprise Number yet, see Register an IANA PEN for KernelKit org. #1640. The rpc_log test uses the RFC 5612 documentation number, test@32473, as a placeholder. Once a PEN is registered, pick the real id, e.g. infamy@PEN, and use it in the Infamy helper for all markers.
2. MSGID convention. Fix a small vocabulary for markers (test-start, test-stop, step) so the collected logs can be filtered on the msgid property filter, both on the DUT and on the test server side.