Today the Server Went Down

Today my server first started responding badly, then went down once, then twice. I thought maybe it was a connection problem, but something interesting turned out: during autonomous work, and a fairly standard procedure in Codex, an agent created a process that got into a loop and began multiplying itself. It was not malicious, it was a model error, not an external attack.

If I had not had a second device from which I could diagnose all this, the situation might not have ended very well. But I have a separate server for each project, the servers are connected to each other, and I can give one a command so that it writes to another.

Today I launched my commander on my Mac, on the computer. It established that the connection to the server was good, and the problem was on the server itself: the RAM was completely full and everything was hanging. It went in, stopped the processes, found out what had gone wrong, and passed this as a task to the agents managing that server.

We developed a health-monitoring setup, something like the task manager on a computer, which watches the processes. This is deterministic code that uses practically no resources, but notices strong changes, excessive load and unusual things.

It will diagnose everyday work, and we will look at what is normal for the system and what is not. Once we define the boundaries of normal, we will be able to assess changes in load. For example, when 5–10 tasks are running in different agents, that is one level of load, but if their number suddenly grows three, four or five times, there is reason to suspect an unhealthy process. After that the system will work out what is happening and how.

If the system does not wake up, it sends a signal to another device, which will act from outside and repair it. This already sounds futuristic, and there are, of course, many cybersecurity nuances here, but I am working on them.

The connection between my devices is special and protected, they have their own personalized tunnel, they are not simply open on the internet. An interesting case happened today.