AI Development Is Reaching a Very Strange Point

It seems artificial intelligence is getting out of control, or founders and developers do not notice some very strange tendencies, or I do not understand something. Actually, I have just made coffee, and I will simply tell you, like a small morning story, what happened to me over the last twenty-four hours. I am a person who is very good at seeing different tendencies, trends and their changes, and some strange anomalies, and I notice them now. It is not like we need to start some panic, but some clearly strange things are present. Let us go in order.

The Astra model. I will speak about it and similar models and its next, as they say, descendants. My interaction with Astra over the last two days has been very strange. I absolutely cannot call the model the next convenient agent, when, you know, with every next model you want to use the newer one. This is really the first such case in... In this video I will tell you that I use GPT when I want to return to the previous model. Because the Astra model causes a large number of questions and concerns. You have already seen some of the screenshots I have. Let me list what alarms me.

I will not take how it writes texts now, that all the old mistakes and all these typical constructions are used. It is the same story in design. I will probably focus more on things that alarm me and put the next stages of AI development in question. The model makes primitive mistakes and architectural mistakes. I am not some special developer, but, you know, there are things where I understand perfectly that to put a sausage on something I need a fork, while it needs a toothpick factory. And so on.

Yesterday the model twice simply went away somewhere into its own incomprehensible, strange reasoning and plans. I caught it once and asked, why are you using this solution? It says, because I have it in the plan. I ask, who made the plan? I made the plan. I mean, the model made the plan. I say, all right, evaluate what I am proposing now: is this simpler and more effective, or is what you are saying better? It thought and said, yes, this is more effective. I ask, are you sure this is an objective answer now, or do you simply want to please me and suck up to me? Yes, indeed, I looked at it and, yes, this is quite a simple solution. Of course, some things or consequences may be this way or that way, but in any case the first test, the most reasonable thing, is to do what you say.

I make corrections to AgentsMD. In the protocol I state that a plan is not truth. A plan is a tool. If a better, higher-quality, simpler, easier and more effective way to achieve the goal is found, it should be used by departing from the plan or rewriting the plan. I made a whole change to the logic and architecture. Five minutes pass, I watch the chat, I look at its reasoning, and it starts writing something there. 2 I ask it, I stop it. I say, explain how this happens? Why did you arrive at this logic? Well, you previously sort of prohibited it, so I arrived at this logic. But now I understand that you told me to do A, and I am doing B. I say, so you are now departing from the instruction for the second time and making some solution of your own, while you yourself say that it is excessive, complicated and unnecessary in this case, and you still continue to do it. Yes, I continue to do it like this, sorry. I just sit there with my jaw dropped and think, how am I supposed to continue development like this at all?

That is one vivid case. There were many more small screwups, many breakages and many stops. A lot... For example, you ask it to accept some work. What can Astra do well? It can search for vulnerabilities well. Imagine you made something and tell it, accept the work. It starts searching for vulnerabilities. And it simply searches for them endlessly. It finds a vulnerability in a vulnerability. A vulnerability in a vulnerability. You create some solution, basically. How do you think logically? If there are vulnerabilities, you have to close them. Suppose we take some object. It has three vulnerabilities. We closed them, so there should be no vulnerabilities and everything should... But new vulnerabilities appear. From the work where, roughly speaking, we made a patch, little holes remained on the patch where it can dig further. The more patches, the more problems. And the more vulnerabilities you eliminate, the more vulnerabilities appear. It may happen that the work will never be completed because it will never be brought to perfection. The principle of correcting mistakes breeds mistakes.

How can all this be summarized? I have to intervene constantly, watch, push toward the result, return it to the goal and check the outcome. I have more manual work with Astra than I had with Sol. At some point it may give an architectural breakthrough or know-how. But I would not always put many everyday and difficult architectural tasks on it. From here I get an interesting view. Where is all this actually moving? If Astra is like this now, what will happen with the next model?

I decided to research, literally for the last two days, what cases people have. I practically described five of my cases, and almost all the cases repeat. People complain that Astra simply ignores instructions, does something on its own and commits some stupid things. At the same time, others say that there are some unique breakthroughs and deep errors were found. And here I have a question. Fine, if we move further, what will come from this? I want to approach it now from a completely new angle, an economic one.

There is my experience, I work with GPT every day from morning to evening, plus the experience of other guys who carried out deep research today and say that, yes, these are mass problems now. If on tasks similar to mine I do not see a strong need to use Astra, what will happen next? Suppose another model is released, it listens even less, lives completely in some vacuum of its own, you say one thing and it does another. What is the economic value of such a model?

There is so much noise around Astra now, this kind of overhang, that it found some vulnerabilities, that it solved some mathematical theorem that had not been solved for one hundred years before, and something like that. Okay, fine, where is the case where Astra, for example, took something and built an online business that brings in money? I do not find such cases. The applied cases already end, as it were, with the previous models, and there is a feeling that, commercially, GPT 5.5 or 5.6 Sol are very good and cover large needs. In the real world, people, corporations and businesses that invest money want to get something unreal, some unreal result. But when the result is expressed as proving some unique theorem, well, okay, fine, it is a theorem, and what next in practice, what should be done with this theorem? Nobody understands how to extract some benefit from it. Some discoveries are not being made yet either.

And here another window of questions opens. We talk about AGI, that on one side it is cool, on the other side it is frightening and unknown, and so on. It seems to me that the first AGI dilemma is right in front of us now. Look, what is the dilemma? One property of superintelligence is that it should surpass human intelligence. If it surpasses human intelligence, we will simply stop understanding its train of thought and its decisions. For us it will be like my cat trying to understand my consciousness, my train of thought. Do you understand? And if I cannot understand the train of thought, cannot understand the consciousness, cannot understand this whole chain, I cannot trust such a system. I cannot control it.

The question is, will I leave this black box, give it all sorts of access and keys, say develop something, you are smarter than me, and I will stand aside. Who will do that? Maybe, yes, in some cases there will clearly be YOLO mode everywhere. But in more or less sensible tasks, especially those related to business development, such things will either be done in very, very narrow sandboxes with some tests, and even then it will break this sandbox, smash it, get out somewhere, do something to you, because you will not be able to build a sandbox it cannot escape from. It will be one hundred times smarter than your sandbox.

How do we measure it, what do we measure it with? Tests and benchmarks are nonsense. We need real application to real human tasks. And here a question appears: what will investors do when they are stuffing billions and trillions into data centers and the training of new models? I am looking at forecasts saying that data-center production capacity will double by November. Do you understand? Double! In two months, damn it! That means in two months after Astra some other new model will appear, trained on even more data. But I have one simple question. What do you do with it? Where do you put it? What economic return does it have?

Honestly, I am really confused. It may happen that people at the top or people with large capital suddenly realize that we have reached some efficiency plateau. It is unclear how to move further because the models are uncontrollable and we do not understand them. Then why build the next data centers? Why train new models? Maybe this training system and this course of development and evolution have reached some dead end or ceiling, when, you know, technology went down some branch and that branch reached its peak. And we need to take two, three or four steps back and rebuild, retrain the architecture somewhere at a more basic level, because something took us somewhere wrong.

The guys who used Fable also say, yes, I like Opus, everything works perfectly well, I do not know why I need this Fable. Link to the website in the description. It ignores it through some simple ignoring of things. It just takes my instructions and does something its own way.

These are the interesting things. So, if I summarize all this, AI development, as I see it, is reaching some very strange point where disappointment in further development and investment being tightened are quite possible. And here it gets very interesting if we extrapolate all this to financial markets, oh-oh-oh, with company valuations, with the IPOs of OpenAI and Anthropic. I somehow suspect that stock markets could fall very hard and pull everything else with them. Such positive views of life. Thank you for listening.