With the decision to push the boundaries of what was possible with a local first predictive scheduling, I realised that it will be necessary to abstract the various background tasks for keeping the AI Prediction in the Browser using Vector Embeddings up to date.

The first consideration is that these embeddings would need to be ready when the user started to plan their schedule. They don't take a lot of time to build at the moment but I expect this to slow as the model gets more expansive.

They will become stale after any change to lessons on the calendar, however because they are based on the last 6 months worth of data it would be a waste of resources if we recalculated them after every change. It would be better to send them to a background job to perform on a separate worker.

Changes often come in batches so it would also be good to debounce or throttle the worker jobs in a way which is ergonomic to the developer. I want it to be structured in a way that the higher level code can be more idiomatic and not think too much about the finer details.

I have done some research to find examples of job queues and background workers for the Browser context. They should come with persistent Storage allow for running with a delay and handle failures. All of the examples found are no longer maintained and do not provide the functionality needed for the task.

Let's consider what would make a robust browser-based background worker.

This is running in a browser so it will need multi-tab support. This means a robust locking mechanism to prevent multiple tabs from processing the same job. The design of this queue doesn't need to cope with thousands of concurrent jobs. A leader election pattern where only one tab acts as the job processor will be fit for purpose.

> [!note] IndexedDB READ_WRITE transactions ensure that the data being read is consistent across tabs.

The Web Locks API can be used to implement this leader election. It "allows scripts running in one tab or worker to asynchronously acquire a lock, hold it while work is performed, then release it". Jacob Wright has also built a really simple tab-election package which will handle this as well as provide the interface for passing messages and events between the tabs.

Jobs should survive browser crashes and tab closures. It will need to track job states (pending, processing, completed, failed) and implement recovery logic to restart interrupted jobs when the application loads.

Failed jobs should be retried up to a number defined in configuration and a list of failed jobs can be accessed easily from an index on status. An index on status can be used to easily accessed failed jobs for troubleshooting and debugging or even to redrive.

There should be support for delayed/scheduled jobs (run after X minutes, run at specific time) and recurring jobs. This requires storing execution timestamps and a scheduler that checks for ready jobs.

Job scheduling has various considerations when run within a browser / local context. We cannot have much certainty at all, that the job worker will be available to execute the job at the scheduled time. This means that a scheduled job must communicate the length of time it is acceptable to remain alive. If the system encounters a job due to be scheduled outside of this "time-to-live" then it should either be marked as failed or cancelled with an error code giving the reason.

Different job types may need different priority levels, allowing critical jobs to jump ahead in the queue. The design of the system is to cope with 10s of messages in the queue so prioritisation algorithm can be very simple. A job can be given a priority, this will allow it to be prioritised within a queue. Queues themselves will be used to control order, currently it makes sense that jobs within the same queue are not run in parallel. Multiple queues can be used to separate jobs which are designed to have no direct relationship with each other.

It would be possible to use XState with Persistence to achieve a similar outcome. In order to do this, a background task would be considered as a piece of a larger state-machine and be run within a Worker. I expect that we will still encounter Concurrency Control issues when running the application in multiple tabs. This doesn't need to be an either-or situation although, as with any feature, maintenance should not be underestimated, in this situation it is hard to gauge whether integrating and updating an external framework like XState would simplify maintenance, or cause more complexity.

There are a number of complexities that go into building a stateful background job system. I would say that even though the Browser context brings it's own challenges, because we are not designing a system to handle large volumes of jobs, the implementation can be drastically simplified.

https://developer.mozilla.org/en-US/docs/Web/API/Web_Locks_API

> The **Web Locks API** allows scripts running in one tab or worker to asynchronously acquire a lock, hold it while work is performed, then release it. While held, no other script executing in the same origin can acquire the same lock, which allows a web app running in multiple tabs or workers to coordinate work and the use of resources.

https://stackoverflow.com/questions/5518692/locking-model-for-indexeddb#10511285

> The IndexedDB specifications determine that "If multiple READ_WRITE transactions are attempting to access the same object store (i.e. if they have overlapping scope), the transaction that was created first must be the transaction which gets access to the object store first. Due to the requirements in the previous paragraph, this also means that it is the only transaction which has access to the object store until the transaction is finished."

That means that when a transaction is in a READ_WRITE mode the objectStore will be locked for other READ_WRITE transactions up until the transaction will finish.

You can read more about the IndexedDB transaction modes from here - http://www.w3.org/TR/IndexedDB/#dfn-mode

https://www.w3.org/TR/IndexedDB/#transaction-scheduling

> Any transaction created after a read/write transaction sees the changes written by the read/write transaction. For example, if a read/write transaction A, is created, and later another transaction B, is created, and the two transactions have overlapping scopes, then transaction B sees any changes made to any object stores that are part of that overlapping scope. This also means that transaction B does not have access to any object stores in that overlapping scope until transaction A is finished.

https://github.com/dabblewriter/tab-election

> Provides leadership election and communication in the browser across tabs _and_ workers using the Locks API and BroadcastChannel. It works in modern browsers.

> The [Locks API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Locks_API) allows us to have a very reliable leadership election, with virtually no delay in database or server connections and app startup time. When the existing leader is closed, the next tab will become the new leader immediately. The Tab interface allows calls and messages to be queued before a leader is elected and sent afterwards. The Tab interface supports everything you need to have all tabs communicate with one leader for loading, saving, and syncing data between tabs, including calling API methods the leader provides, broadcasting messages to other tabs, and state syncing.