The agent Troubleshooting
Troubleshooting
Start in the console. A machine's page shows its state, and the installer on the machine tells you why it stopped. Everything below comes from what the console and the installer actually say.
A machine is Installing or Stuck
A machine is Installing from the moment it enrolls until its first heartbeat, then Active. If it enrolled and no heartbeat arrives within 15 minutes, it becomes Stuck. The console says Waiting for its first heartbeat for Installing and No heartbeat since enrolment for Stuck.
Enrolling is not the same as reporting. The machine was enrolled, so the token is spent, but the agent has not yet reached the console. Check, in this order:
-
Is the agent running? On the machine, run the agent's
statuscommand. It prints the machine's identity, server, bundle hash, last heartbeat and health.WindowsmacOSLinux -
Can the machine reach the console? The agent calls the console's address, the one in the install command.
-
Read the log. See "Logs on the machine" below.
To start again, create a new install command and follow "Reinstall" below.
States at a glance
| State | What it means | What you can do |
|---|---|---|
| Installing | Enrolled, no heartbeat yet | Wait up to 15 minutes. Suspend or Delete are available. |
| Stuck | Enrolled, never checked in after 15 minutes | Check the agent as above. Suspend or Delete are available. |
| Active | The agent has checked in | Nothing |
| Suspended | The agent captures and enforces nothing | Resume, or Delete |
| Uninstalled | The agent was removed on the machine with its removal key | Delete. The record is kept for its history |
| Archived | A Stuck or Uninstalled machine, hidden from the default list after 30 days | Delete |
Choose a view in the machines list to see the machines in a state.
The installer stopped
The installer prints a box that names what stopped it, what to do next, and whether your token is still valid. Most stops happen before the token is spent, and say Your token is unspent.
| The installer says | What it means and what to do |
|---|---|
| This PowerShell is not elevated | Open PowerShell with Run as administrator and run the same command. |
| No systemd | The Linux installer registers a systemd unit and this machine does not run systemd. Install on a machine that does. |
| An agent is already installed on this machine | The installer will not replace it without being told to. Re-run with -Replace (Windows) or --replace (macOS and Linux), together with the machine's removal key. |
| Removing the existing agent needs a removal key | Your enrollment token does not give authority to remove an agent. Issue a removal key and add it to the command. |
| The existing agent was not removed | The reason is printed in the box. Re-running is safe. |
| The previous agent's Windows service is still registered | Windows only. The previous agent's service was not removed. Follow the box. |
| This machine cannot run the agent as a service | On macOS and Linux, the agent must sit in a directory only root can write. Use the default install directory. |
| No agent release to install | No build is published for this operating system. A platform admin has to publish a build pack and make it the default, then you run the same command again. |
| Enrollment failed with exit code … | The agent's own message is printed above the box. Whether your token was spent depends on that message. |
| Service registration failed with exit code … | Windows only. The service did not register. |
If enrollment itself is refused with enrollment credential is not recognised, expired, revoked, or already used, the token was used once, is older than 24 hours, or its profile was revoked. Create a new install command. A reusable secret behaves the same way once its machine count or expiry is used up. See Enrolling many machines.
At the end of a good install, a box titled Installed lists anything that still needs a person under Needs attention: no tray icon, no tool hooks, or a certificate that is not trusted yet. A machine whose profile includes TLS interception reads Degraded until it trusts the interception certificate.
Logs on the machine
The agent writes a log only when it runs as the installed service. An agent
that you start by hand logs to its own console instead. The log is in a logs
folder inside the agent's data folder, and it rotates daily.
| Platform | Log folder |
|---|---|
| Windows | %ProgramData%\Oxelone\agent\logs |
| macOS | /Library/Application Support/Oxelone/agent/logs |
| Linux | /var/lib/oxelone/agent/logs |
On macOS and Linux only root can read the folder.
Raise the log level
Open the machine and go to Configuration › Log level. The levels are
error, warn, info, debug and trace. By default the machine follows
its profile. Choose Set on this machine to override it.
A person at the machine can also set the OXEL_LOG environment variable,
which overrides every other setting.
A full tracing directive in OXEL_LOG, such as ureq_proto=trace, writes
raw network bytes, including the compiled policy, into the log. While it is
set, the log is customer content, and a diagnostics bundle collected then
carries it to the server. Unset the variable and delete the logs folder
before anyone collects diagnostics. A plain level such as debug does not do
this.
Diagnostics
Open the machine and choose its Diagnostics tab. It has four sections.
| Section | What it does |
|---|---|
| Round trip | Test round trip has the agent measure a real call to the server. It answers on the machine's next heartbeat. It reports the network time, and the control-channel time: how long the command waited for a heartbeat. |
| Collect diagnostics | Collect diagnostics asks the agent to send its local logs. The bundle arrives after its next heartbeat. |
| Bundles | The bundles received. Open one to see the collection, the log and the counters. |
| Crashes | The last crash the agent reported: when, on which thread, and the error. |
Running a round trip or collecting diagnostics needs the fleet:manage
permission. Neither is offered for a machine whose agent is gone, such as an
Uninstalled or Archived one.
What a bundle holds
A bundle is deliberately small:
- Counters: names with numbers or yes/no values, one level deep. No text.
- A log tail: the most recent part of the log. The bundle is capped at 256 KiB, so a long log is cut to its tail, and the bundle says so.
- The collection: when it was received and collected, the agent version, the last error as a category, and the policy bundle the machine had.
A machine keeps its ten most recent bundles. Reading a bundle's log is recorded, with who looked. Delete this bundle? is permanent and recorded.
If a bundle says The agent found no log files on the machine, the agent was not running as the installed service, because it writes a log file only as one.
Remove or reinstall
Remove an agent
Open the machine, then Lifecycle (under More), and choose Issue key. The removal key belongs to that one machine. It is shown once, and issuing a new key invalidates the one before it. Run the uninstall command on the machine itself, as an administrator. Deployment has the commands.
The console never tells an agent to remove itself. Removal happens on the machine, with the key.
If the console that the agent enrolled against no longer has any record of the
machine, because its database was rebuilt, there is nowhere to issue a key
from. The installer and uninstaller take --orphaned for that. It is not
authority by itself: the agent asks the server in its own configuration and
removes itself only if that server answers that it has no record of the
machine. If the server cannot be reached, or answers anything else, nothing is
removed.
Reinstall
An agent already on the machine stops the installer, so you remove or replace it:
- To replace it in one step, re-run the install command with the removal key
and
-Replaceor--replace. The previous agent is removed first, and the new enrollment is linked to the machine it replaces. - To start clean, uninstall with the key, Delete the old machine, and install again.
Suspend, resume and delete
Suspend stops the agent acting on the machine and puts back what it changed. It is reversible: Resume restores it, and the agent does so on its next heartbeat, within a minute. Use it instead of removing an agent when you want a pause.
Delete is the final step, and is offered for a machine that is Suspended, Stuck, Installing, Uninstalled or Archived. It removes the machine from the console. If its agent comes back online, the agent removes itself. It cannot be undone, and the audit trail keeps who deleted it and when. A reinstall on that computer enrolls as a new machine, which can name the deleted one as the machine it replaces.
Changing a machine's lifecycle needs the fleet:manage permission.
Still stuck
Use New ticket in the account menu to tell Oxelone about a Bug or a
Question. Copy diagnostics in the Help menu puts what the ticket would
carry about the page you are on on the clipboard. Read any log text before you
paste it into a ticket: a log can hold customer content if OXEL_LOG was set.
See Tickets.