Nestor Arocha

Celery (python library) limitations

Thu, 06 Aug 2026 00:00:00 +0100

celery is a python library to manage some concurrent workloads.

Celery tasks are called from regular python code. They go through a queue and get executed in a separate worker process. The results are sent back to a database available to the caller. Its simplest form looks like this:

result = task.delay(arg1, arg2, kwarg1='x', kwarg2='y')

with celery managing everything if configured properly. In celery it is possible to specify more complicated structures using workflow primitives.

groups execute tasks in parallel. chords call a final one when all done

chains execute tasks using the output of an entry as the input of the next one, returning the final output

map apply the same function to a set of inputs, and return the outputs of each of them

The limitation

Celery is one of the many options for concurrent execution in python. It is a bit more complicated that using threads or subprocess, but resistant to the main process restarting. Unlike asyncio it allows more than 1 cpu at a time. It is less complicated that using a database or stream processing where guarantees of execution are stronger.

But there is a weakness that after certain scale becomes problematic: There is no permanent task storage over time.

When a celery task runs, the task gets inserted in the queue.
When a worker is ready to run the task (or wants to prefetch it), the task gets removed from the queue.

So for example, if the worker gets killed but the task isn’t finished, what happens? The worker is responsible to put the task back in the queue. In most circumstances it will be handled fine, but under pressure, the task can receive a SIGKILL or produce a exotic signal like SIGSEGV. Celery won’t handle this so the task will be lost forever.

It is possible to use acks_late as a property of the task. This informs celery not to acknowledge the task until it is fully done. The problem then is about running the task multiple times in case of failure. This is the exactly once problem that kafka in particular aims to solve, although any database with transactional guarantees offers the same.

There is a complication to this limitation: Remember the workflows I mentioned before? They live as tasks in the same system. So they are either in the queue or in the workers’ memory. Any complicated workflow will inevitably disappear and executed partially or get executed multiple times in these situations.

Alternatives

Some people suggest to use temporal.io . I think the lock in is too unpalatable so rolling out something with redis, postgres or kafka that has better guarantees could do.

This depends on the complexity of the workflows though. For very complicated workflows something like https://argoproj.github.io/workflows/ could work best.

tags:#python

I agree with codeberg's LLM contribution decision

Wed, 05 Aug 2026 00:00:00 +0100

Recently codeberg passed two motions regarding LLM usage:

https://blog.codeberg.org/protecting-our-floss-commons-from-llms.html

The first vote is about codeberg not using the hosted code to train LLMs. The second one is the one I want to focus on; it prohibits vibe coded projects on the platform.

They mention in their post the “digital divide”. Large cloud companies are the only ones able to afford the infrastructure to handle all the costs associated with using LLM and being part of the 2026. Smaller actors can’t afford to exist on their own and instead they have to rent from the cloud providers.

I strongly resonate with this divide as I perceive it constantly. For example, I don’t think it would be possible for any startup to create github today if it didn’t exist. The amount of noise of slop projects would make too hard to serve legitimate users while keeping it free or with a harmless “pay for private repos” that github used to have.

So codeberg had the option to try to accommodate to these forces and become bigger and more corporate-like, or to stay small and serving the people that want the community above the market or energy forces.

The ecological case

Creating and modifying software involves effort. Developers have to put the thought to understand and translate the requirements into whatever needs to exist.

Cloud availability has accelerated the conversion from thought and reading into literal computer effort. I am talking about pipelines, deployments and other verification tools. Some of the effort has shifted to a computer running in the cloud, reducing the amount of human effort. Simply push the code and the pipeline will run.

LLMs are a big step in the same direction. A set of prompts and documents (a specification) plus a lot of energy and materials in a data centre produce plausible code. Leaving aside how good the code is, it reduces the amount of human effort to generate code. I wrote on this trade off recently:

[[2026-03-15-the-llm-code-bet]]

Both in the cloud revolution and in the AI revolution, the human cost of creating software goes down. Jevon’s paradox seems at play in here as these changes have increased the amount of code produced.

At the same time, LLMs have reduced the effect of trust and human networks in the process. The proof of work is not there. So I believe that people are less trusting of new projects unless they come with some guarantee in the form of robust tests, verification or guardrails.

The new ways of work require more energy to generate code, more energy to verify the code and also produce more software in general. It makes sense if energy is very cheap.

The bill please

So who is paying for this extra energy cost? I would say originally the investor class paid for it, looking for future profits. As LLMs improved, working people are starting to pay directly by losing their jobs. We are also losing by losing access to personal computing.

Energy wise the extra demands come from other sectors of investment. All the capital that go into data centres is not available for something else, like infrastructure, health care or education.

Ultimately the capital cost reflects the cost to nature so we are all losing out on that one, although we call it externalities.

This is why I think codeberg is right not to be part of this extra cost dynamic, and wait until we find the amount of LLM usage that actually increases productivity without increasing energy or human collaboration costs.

P.S.: I haven’t mentioned the likely bubble bursting nor that the multiple wars around oil will likely impact the price of energy, making LLM software development fragile. Maybe another day.

tags:#llm

On Conway's law

Mon, 13 Jul 2026 00:00:00 +0100

Conway’s law is a well known “law” of systems. It is an observation really:

Organizations which design systems (in the broad sense used here) are constrained to produce designs which are copies of the communication structures of these organizations.

https://en.wikipedia.org/wiki/Conway's_law

Martin Fowler also wrote similarly:

The key thing to remember about Conways Law is that the modular decomposition of a system and the decomposition of the development organization must be done together. This isn’t just at the beginning, evolution of the architecture and reorganizing the human organization must go hand-in-hand throughout the life of an enterprise.

https://martinfowler.com/bliki/ConwaysLaw.html

I would distill it as “codified systems follow, at best, the structure of the system that produces it”. This matches my personal experience.

Companies that do understand Conway’s law try to modify both the existing code structure and the organisational structure, slowly nudging both towards the desired structure. Or allow the teams to be flexible to nudge the system structure towards the company goals.

Companies that do not understand Conway’s law produce a structure that does not reflect the existing codebase structure, and instead focus on the expected business structure.

This creates a divergence between the code structure, its ownership and s responsibilities that translate into debt and constant wasted resources of dealing with the ever growing divergence

The mirroring hypothesis

Recently I came across “Exploring the Duality between Product and Organizational Architectures: A Test of the “Mirroring” Hypothesis” Alan MacCormack, John Rusnak, Carliss Baldwin explore the vailidity of the mirroring hypothesis

The mirroring hypothesis I interpret as a weaker version of conway’s law:

The mirroring hypothesis predicts that these different organizational forms will produce products with distinctly different architectures. Specifically, loosely-coupled organizations will develop more modular designs than tightly-coupled organizations.

It refers to the “modularity” or “coupling” aspect of conway’s law rather than the particular

They found strong evidence to support the mirroring hypothesis. I also believe this is a strong arguments for self organised units, similar to what agile proposes. Fred Emery coined the two design principles.

DP1 and DP2

DP1 and DP2 are that two design principles that Fred Emery attributed to all forms of organisation.

https://ingbrief.wordpress.com/2012/03/02/genotypical-organization-design-principles-dp1-and-dp2-merrelyn-emery/

DP1 is the redundancy of parts. So many parts do the same thing to preserve the organisation. Think of managers referring to the same work and autocracy.

DP2 is the redundancy of functions. Each component of the organisation has the ability to do the same thing and they collaborate. It is democratic.

There is also the Laissez-faire which refers to organisations that fail to follow any of these arrangements.

I believe that Conway’s law, the agile formulation of acting as an independent cell with redundant functions and DP2 structures refer to the same thing. They are all about keeping an eye on the structure to fix things structurally, and enable the modularity that resilient structures exhibit.

They are politics so they are hard to optimise for. Without team or business structure considerations, rigid authoritarian or chaotic outcomes dominate the company which are counterproductive in different ways.

tags:#architecture #design #software

The whale fall

Tue, 07 Jul 2026 00:00:00 +0100

Whales can live longer than a human. They follow Kleiber’s law, so bigger whales will consume less energy per unit of mass, which is corelated with longer lifespan. Bigger whales live longer on average.

But even if they lived thousands of years, they are not immortal. When they die, gravity takes over. Buoyancy doesn’t work if the air inside them escapes. If they are far enough from the coast, they will inevitably fall to the bottom of the sea.

https://en.wikipedia.org/wiki/Whale_fall

But life imposes it cycles everywhere with its ecological logic. Various organisms populate the remains as they provide a new opportunity to growth.

A mental experiment

With this carcass on the ocean floor in mind, I want to propose a mental experiment: Imagine you are one of these microorganisms that happened to find one of these carcasses. Big carcasses are great opportunities to grow compared to small animals, so this is a good place to be.

After the organic material is all processed, there is a certain amount of population that the carcass can support by acting as inorganic coral. It is not a lot though.

On the negative side, this is a very remote carcass. It is also releasing radioactive particles whenever is processed. Releasing these particles will reduce the chances of survival as it will contaminate the whole ocean including the rest of the carcass.

So here are some ideas on how to go about it:

Grow as fast as you can until the carcass depletes, then deal with the crash later plus the radioactive particles
Invest on resilience by stopping growth and fighting competing organisms. Note that this only works temporarily as a growing organism will eventually use more resources to take over
Grow as little as possible or not at all but do not let others take over, seeking for a low energy homeostasis
Leave entirely renouncing to the available growth

What would you do?

The real life thing

I could write that because of this year unprecedented heat waves, climate catastrophe is around the corner. I feel like this has been true for my whole life, and accelerated considerably. It will still be true many years in the future.

As small organisms part of human society, we have some choices or at least tendencies to act one way or another.

Some people see growth as a goal and something to strive for, so there is no drawback to it even if the crash is bigger later
Some people will actively invest on local resiliency. This only works if the extra resources cannot be retaken, so timing is critical
Some people will not invest on either, and try to live within their means. Other people that grow too much can make this unstable too
Some people withdraw entirely and refuse to be part of the system. They move to somewhere far away where they can be self sufficient, though other’s growth can make this unstable over long term too

Any choice that isn’t growth requires good timing and common restrain. Is it possible to choose with competition? What would you do?

tags:#ecology #climatechange

Measuring import time in python

Sun, 05 Jul 2026 00:00:00 +0100

In python a code file can do arbitrary stuff. They can connect to a server, calculate something complicated or have enormous literals to parse.

When projects get big, this is a problem because the import time compounds and most people and code dislike imports inside function or class scopes.

lazy imports appeared in python 3.15 and allows deferring the actual import while keeping them at the top of the file.

In both cases deferring the import introduces arbitrary waits that are not predictable. They also can affect the state of the program by patching, forking or performing any other side effect operation that depends on order.

Sometimes it is better to measure the import time and make the changes so the expensive code runs in the appropriate context.

The post 3.8 way

Python now includes import time in its standard library.

https://stackoverflow.com/questions/34755728/profile-python-import-times

This method requires no code changes and tuna allows to see the visualise the output without.

The custom or pre 3.7 way

Back in the day with python 3.6 patching the easiest way to get the information was to patch the import system

from dataclasses import dataclass
import inspect
import sys
import importlib.machinery
import time

@dataclass
class ImportDuration:
    name:str
    parent: str
    duration: float

    @property
    def calculated_duration(self):
        return max(0.0, self.duration - sum(x.duration for x in self.children))

    @property
    def children(self):
        return [node for node in IMPORT_DURATIONS.values() if node.parent == self.name]

IMPORT_DURATIONS = {}

def _genrate_loader_details():
    """Returns a list of file-based module loaders.
    """
    extensions = importlib.machinery.ExtensionFileLoader, importlib.machinery.EXTENSION_SUFFIXES
    source = MyLoader, importlib.machinery.SOURCE_SUFFIXES
    bytecode = importlib.machinery.SourcelessFileLoader, importlib.machinery.BYTECODE_SUFFIXES
    return [extensions, source, bytecode]

class MyLoader(importlib.machinery.SourceFileLoader):
    def exec_module(self, module):
        start = time.time()
        result = super().exec_module(module)
        duration = time.time()-start
        module_file = module.__file__
        if module_file not in IMPORT_DURATIONS:
            stack = inspect.stack()[4]
            parent_filename = stack.filename
            IMPORT_DURATIONS[module_file] = ImportDuration(module_file, parent_filename, duration)
        return result

sys.path_hooks[1] = importlib.machinery.FileFinder.path_hook(*_genrate_loader_details())


if __name__ == "__main__":
    import cogapp # any import that we want to measure
    from pprint import pprint
    pprint(sorted([(x.name, x.calculated_duration) for x
                   in IMPORT_DURATIONS.values()],
                  key=lambda y: y[1]))

The main idea is:

Reimplement SourceFileLoader by adding the timing code around the actual loading. docs
Tell the FileFinder to use the new loader for loading python files using .path_hook docs
Run the actual code
Inspect the data structure where the timing information is stored

I don’t remember where I got this code nor how much did I have to fine tune it, but I remember getting the signatures like the loader_details being hard.

This method has the advantage of being fully customisable so for example it can run for imports after forking, or raise an exception if an import goes above certain duration.

tags:#python

Postgres: Long transactions and Access exclusive locks

Thu, 02 Jul 2026 00:00:00 +0100

I would like to share a postgres story that has happened to me a few times.

The story has the following ingredients:

An overly centralised Entity/Relationship model
Long database transactions to aid data consistency
Schema migration of tables
Poor observability of transactions

1. Centralised tables go like this: A database with many FK to a to a central table in the business logic that hasn’t been properly designed works fine most of the time.

Django or any other MVC (MTV) frameworks that manage the database schema and include some central tables (like User) suffer from this issue

As the system grows, many models FK directly to the existing models rather than redesigning the database to reduce the number of FKs to a single model. These models become accidental Aggregates.

2. Long database transactions are a common pattern in MVC frameworks: As the system grows, more and more database operations need to happen with transactional guarantees. These frameworks offer the option to use the database transaction to coordinate all of these operations.

In web applications where many workers access the same database, this is a good trade off as it can be represented in code without any external tools. Generally this coordination of what needs to be simultaneous is one of the hard problems to figure out in a system.

3. Schema migrations of tables

These are also common in MVC frameworks. In python both django and alembic offer to manage the database schema by reading the model definitions from python files. But partly because of their ease to change and apply, in some configurations these happen at any time.

Everything works fine until it doesn’t

Imagine the following set of N transactions:


T1 -> T2 -> T3 -> T4 -> T5 -> ... -> TN

None of these factors cause issues on their own.

Accidental aggregates add extra cost to any FK changes, but not enough to be noticeable.
Long transactions could cause a problem, but most of the time the will lock individual rows and not interfere with other operations.
Schema changes could block the database briefly because of the expensive table lock, but avoiding changing already populated big tables is generally enough

Combining all of these though can cause the database to become unresponsive. Something as simple as creating a new table with a FK to a central table has the power to block any operation that reads or writes the central table.

So for the same set of transactions:

T2 is a long transaction
T3 is a schema change that requires Access Exclusive


T1 -> T2(Long) -> T3(Lock) -> T4 -> T5 -> ... -> TN

If T2 was short, T3 would be able to start immediately. If T3 didn’t lock the whole table, it would allow the subsequent transactions to continue.

In this scenario:

T4 -> ... -> TN are all blocked by T3 because they all refer to the centralised model and postgres honours the order of operations.
T3 is blocked until T2 finishes because its lock requires no other operation running on the table

So no read operation happens to any table affected until T2 and T3 are done. Which is enough to bring a web app down.

Bad visibility

These type of issues are hard to see from regular observability tools like datadog and grafana.

https://grafana.com/docs/grafana-cloud/monitor-infrastructure/integrations/integration-reference/integration-postgres/

My personal experience is with datadog. It is possible to see the long queries and the long transactions but it is impossible to attribute causality to it. So if your system features long transactions and frequent schema changes, it will take some time to infer which transaction caused it and what to do

Solutions

There are many ways to tackle mitigate the risk of this scenario:

Plan for downtime For some systems this might be a viable option: Establish a time of the day where the schema change happen, avoid long transactions during that time.

Use write constraints https://medium.com/doctolib/adding-a-not-null-constraint-on-pg-faster-with-minimal-locking-38b2c00c4d1c It is possible to avoid using Foreign keys entirely and instead use a constraint. It will mitigate the risk of locking but add a penalty on every single write

Disallow long transactions Postgres allows to cancel long transactions https://www.postgresql.org/docs/17/runtime-config-client.html#GUC-TRANSACTION-TIMEOUT It is feasible to refactor the application to rely on external mechanisms instead of database locks. Even an advisory lock will be enough as it doesn’t use any particular table.

Refactor the models If having a central model causes this problem, separating the models within or even outside the database will fix it. Moving the coordination away from a single database has its own trade offs. Added complexity and out of order operations are common in microservice architectures.

*#postgres

https://www.postgresql.org/docs/current/explicit-locking.html#LOCKING-TABLES

Tiny Experiments review

Tue, 30 Jun 2026 00:00:00 +0100

tiny experiments by Anne-Laure Le Cunff is a self help book. I found it in my local library in my quest for deciding my next steps, although it was in the science section instead of self help.

I wrote recently on self help books:

[[2026-06-23-self-help-books-review]]

I think this book is one of the best I have read so far. It isn’t a single methodology book like “bullet journal” or “Pomodoro technique”. This book contains a few methodologies combined, all under the umbrella of having a experimental mindset:

It suggests having personal pacts, and review them like “Triggers” or “The one thing”
It touches on purpose in a concise and direct way, similar to “the purpose code” [[2025-03-20-the-happiness-lab-how-to-find-your-purpose]]
The chapters on time and legacy contain similar in ideas to “Four thousand weeks”
The social flow and learning in public quotes “Flow”, also it reminded me of some ideas on general openness

It is science friendly, like tiny habits, but it feels less like “hacking yourself”. It is as broad on goals as “Building a second brain”, but I think it is more cogent. It is told as a personal story at times, but without leaning too much on it (like “learning systems thinking” [[2025-11-17-systems-thinking-review]]).

I would recommend it to anyone that feels like they have reached a dead end and are looking for a self help book.

*#book #review

Remembering failure

Mon, 29 Jun 2026 00:00:00 +0100

Today some memories of a videogame came back. It was “little king story” a wii game similar to pikmin with a king instead of Olimar and workers instead of pikmins. The classical music was memorable too, it matched the style of the game.

https://en.wikipedia.org/wiki/Little_King_Story

I haven’t played this game since 2011. I enjoyed think I finished it but I don’t remember that much. It worked well as a distraction, as entertainment while I was trying to start a web product.

It wasn’t a good time to start a company, neither inside nor outside. The job market wasn’t great back then. The career change I made towards NLP didn’t work that well for me. I moved too far from Madrid so I lost most of the people I had.

Despite all of that, my family offered some support to start to write some web services I had in mind for some time. The idea was to use what is currently pydsl and attach a repository of languages, DSLs and a way to connect them. A hybrid between a language workbench and yahoo pipes.

I had some months to see if it worked. I developed a few services, a website for it in php and that’s about it. I didn’t do basic market research nor found the people that would be able to help pushing it forward. I didn’t fully implement the pipes idea (I called them boards, but same thing) as it would have required a long time and frontend skills I did not have.

So after a few months, I gave up and released the source code and decided to go to London and try my luck.

Lessons

What did I get out of it? Well, some parser knowledge, improved my python and php a tiny bit more. Also to learn a few bits of the bureaucracy of starting a company in Spain.

More importantly, the failure of it was the big learning, although I didn’t notice until much later. With no market iteration and no people to support the actual work nor to collaborate, I had no chance. I didn’t have any measurement in place other than code and visits. It was just an idea and the hope that implementing the idea would be enough.

It was closer to a dissertation than a business idea. A mental experiment and little more.

I don’t know I have fully internalised that lesson yet. I still do hope that coming with an idea is the best part and I apply that to the way I write programs. I believe that getting the structure right is the most important part of the work, which is not a common belief. But some of it stayed with me and there is some structure to the way I think on projects that wasn’t there before.

In hindsight, this is a very expensive way to learn which I was privileged enough to experience.

The cycle repeats

I don’t think it is coincidence that Little King Story appeared in my mind. This moment resembles that one in many ways. A change of career path. An environment going downwards. The temptation of focusing on an idea as a form of getting out of it, without that much feedback.

Or simply getting distracted with a good game not to think about any of this.

tags:#job #failure

More on 2026 job search

Fri, 26 Jun 2026 00:00:00 +0100

I recently finished the first stint of job seeking efforts. I started gathering momentum in early may and applied for many jobs for a few weeks, but unfortunately nothing has worked so far and I decided not to continue while waiting for the results.

Context: London jobs, remote or hybrid, around my python/fintech experience, nothing bigger than 500 people.

My strategy has been using my current experience (mostly python, platform and infrastructure work) to get a job similar to previous roles, aiming for a slightly lower salary than what I had.

I have been quite selective and I have looked for things that were both similar enough and I found interesting for one reason or another.

Some stats

I applied to 14 roles.

2 Initiated by recruiter, 12 by me.
I used the platform cord for around 8 roles, using regular web search or job board search for the rest

10 (71%) were rejected or ghosted straight away.

7 (50%) rejected
3 (21%) ghosted

The 4 (29%) left I got an answer. All of them I successfully got through the screening (I am human enough after all).

1 (7%) I did not pass the assignment (rejected)
1 (7%) I did not pass the technical interview. (ghosted).
2 (14%) I got through the technical interview but I did not pass the “fit” interview. (rejected)

The different failures

ghosted or rejected before starting Pretty much any role non python, in health or energys sectors. If the experience is not relevant companies are not in the mood of figuring it out.

assignment: The main issue was that the interview prompted for 5 hours of work at most. This company relied heavily on LLMs for all parts of the process and from the feedback they expected more features. I don’t think I am that slow but maybe, although I suspect that they expect LLM usage. They didn’t value the manual process or testing that much, so I think I need to either skip these LLM implicit roles or clarify if the time limit is with AI.

technical interview It was a tough 2 hour interview sharing screen. The first hour, a writing code test, I got well enough to get it implemented. But the second part, a giant code review of a codebase that I didn’t have access to until the interview was very difficult. I do not know what they were seeking for, as I find unrealistic to read both the existing code and the changes at the same time. Perhaps better communication or stress management? It could have been that they expect LLM input too.

company fit 1 I wasn’t particularly well prepared, nor I was feeling well during the interview, but I believe that they were looking for good leadership skills. My experience has been mostly of support and steering with loads of coding and I don’t think that’s where they were looking for.

company fit 2 I was better prepared for this one, but even with that I failed on a similar set of questions, although this time hypothetical rather than personal experience. They were looking for someone with strong coding+product and leadership role, which again is not my direct experience (My leadership has been more subtle in general)

What did I learn and where do I go

I can confidently say that with the market so tight and that the purely software developer role is gone. Right now companies are looking for someone with very relevant experience for their stack, combined with some other skill. I saw these subroles as requirements on top of software skills:

devops/infrastructure
product/leadership
LLM workflows

Anything outside being an almost perfect match in stack is a non starter, and failing in the subrole requirements means not passing the other stages of the interview.

I also noticed that some of these companies keep the roles open and well advertised, which means that they have a strict process that they are happy to go through many candidates to find the “perfect fit”.

On the next steps: to widen the requirements and see what’s outside the “similar to before” jobs. If that doesn’t work I might try again with a more focused strategy.

tags:#job #search
Part 1 [[2026-05-15-notes-on-2026-job-market]]

Exploring LLMs from a DSL perspective

Thu, 25 Jun 2026 00:00:00 +0100

Long time ago I wrote pydsl thinking that DSLs would, if easy enough to make and use, be equivalent to programming languages.

This turned out to be mostly wrong, as programming languages remain the most common way to tackle business problems. Ian Cooper wrote about different generations of programming languages and accidental complexity of DSLs

https://ian-cooper.writeas.com/is-ai-a-silver-bullet

I wanted to explore the transformer architecture and compare it to DSLs.

The DSL/parser architecture

The way I thought about it is that tiny reusable languages could be composed easily:

Where each language would accept certain types of inputs, have their own tokens and produce outputs. The difficulty is to write these parsers easily and all the error handling that would come with it.

The Transformer architecture

This is my current understanding of the transformer architecture for a decoder only: (read from https://dev.to/pranaybathini/the-transformer-architecture-a-deep-dive-into-how-llms-actually-work-4c46)

step1

tokenisation: Where token is some stem, prefixes and suffixes combination
embedding: rather than individual symbols, tokens are vectors of hundreds or thousands of dimensions vector width Number of parameters: a lot of it comes from embedding layer

step2

Position information: A new vector vector width per position that combines with the embedding. It says sine and cosine wave so maybe some FFT,,

step3

Attention: the model has somewhere between 8 and 128 attention heads. They mean grammar or meaning or similar relationships , each one do the following:

For each (word? token?) it has a

query: what am I looking for
key: what do I contain (? the vector)
value: information it carries

all vector width. query and key multiply to produce a score for a pair of (word? token?)

This becomes the new value (?). All of these attention heads combine and feed into step4

step4

Thinking: Feed forward network FFN. With all the outputs of step3 it creates an enriched vector with all the attention heads. There is a bigger size vector going on and it gets compressed again.

Comparing both approaches

Compared with the LLM transformer architecture, DSLs:

have a small number of tokens
tokens are an entity as in individual unique thing, not a vector
The attention step is similar to context dependent grammars, but the context is built in the grammar rules
The FFN is the production part of the parsing as in creation of the intermediate representation or whatever output of the compiling is. The compiler production rules are equivalent to the expansion that happens in this step
There is no loop nor layers. In transformers 3 and 4 repeat many times
there is the possibility of “generative” aspects in the DSL, like enumerating all the possible accepted inputs https://codeberg.org/nesaro/pydsl/src/branch/master/pydsl/grammar/definition.py#L29

Overall, the ability to automate both the tokenisation and the parsing to produce a plausible outcome opens the door to a lot of automation, with the trade off of lack of certainty and massive resource consumptions.

I imagine designing a calculator as an efficiency achievement, whereas this is more like a plausible generator of output. An automation achievement

tags:#llm #dsl #pydsl