Devlog Entry #3 - State of the Art

State Management is a Bish.

The other day I saw a hackernews thread asking Why is GUI programming so hard? and among several of the comments the subject of State Management came up 19 times in a thread with 49 comments (at the time of writing this entry) - that's a whopping 39%.

On top of that, while OP didn't explicitly mention web development several responders specifically referred to the complexity of building UI in browsers and the complex relationship between HTML, CSS, and Javascript as the medium.

I've asked Bard (meh, I cancelled my ChatGPT Plus subscription several months ago due to reduced [almost non-existent] usage, just prior to the introduction of GPTs) to summarize the thread and its response is:

As a web developer for nearly two decades now, I'm building Djot (and in the near future Djitsu) to address at least some of these difficulties and make developing web applications easier and simpler.

And I'm using these very technologies that are extremly difficult to wrangle, in a project that entirely revolves around taming the exact tools and techniques that makes creating modern GUI applications both easier and paradoxically infinitely more complex.

I've started this Devlog almost a month ago, and I've managed to barely create a basic application layout with very little actual business logic related to the application at hand (reminder: I'm building a OpenAI API Assistant application).

Primary reason being the fact that I'm both building the Assistant project inside or rather using Djot, and building/developing/fixing the Djot application itself at the same time.

I'm a strong believer in Dogfooding when it comes to building products, and building the Djot Assistant app for me is exactly that - I'm testing and verifying my assumption that Djot is a useful tool, but also making sure that it actually functions and performs at the quality standard that I myself have become accustomed to in the vast landscape of the modern software and hardware - and that's much much harder to achieve, especially being a one-man-show Djot is at the moment.

State Management is a bish… it really is.

For the last three weeks or so since the last Devlog entry I’ve been refactoring the state management of the djots at the higher level (e.g. the app that manages the different djots running in it at the same time) and the inner djot management which is the individual djots being worked on.

Neither is easy. Luckily in this iteration of building Djitsu/Djot I’ve spent a considerable amount of time (nearly two years!) just trying out and building the infrastructure for it, so the way it is structured is in such a way that the inner Djot Editor is a detached and encapsulated component/app on its own from the upper level application that manages the different djot tabs, which makes wrangling the two state trees somewhat more manageable.

Once I've added support for both directories and different file types (such as images) the neive state management Djot previously had couldn't cope with even a somewhat lightweight project with ~50 files and a few images (1-5mb in size each) - the UI simply grinds to a halt as each keystroke is both causing transpilation of the entire codebase AND being sent to the upper level application that syncs the djot contents with the local filesystem, which is less-then-efficient to say the least.

Addressing those issues required to rewrite several key areas of the Djot application,

• Compiler (or Transpiler if you wish) now performs better by taking into account the actual files that have changed (based on their CRC32 hashes), and guestimating which of the entries require re-compile

• File management and upsteram updates occur on a file-by-file changes rather than sending the entire codebase on each keystroke

• Opening Djots management has been revamped and several issues have been fixed - primarily properly recalling which djots are being edited/updated, tabs ordering when opening/creating new djot, and a few other issues addressed

These are just a few major areas that were heavily modified in the recent weeks - each of them alone and all of them combined affect a lot of the Djot codebase as can be seen in the screenshot from the pull request above - and this was the third time I had refactored these specific state trees since I've started coding this project in Sep this year.

Several times thorughout the last couple of weeks I just wanted to throw the laptop at the wall cursing at the spiraling complexity of the project - at the surface it seems simple enough, but even at this early stage in the project the number of moving parts is becoming increasingly larger and larger and reconciling them all and continuing adding features becomes harder and harder.

As mentioned above I somewhat manage to handle all that thanks to the decoupled and isolated fashion I've chosen to structure it all - which is a result of many years of (trying) writing such applications - both from scratch using my own design or as part of an existing design which I inherited/joined to.

➜ tree -d -L 4 --gitignore .
.
├── apps
│   ├── electron-app
│   │   └── packages
│   │       ├── main
│   │       ├── preload
│   │       └── renderer
│   ├── express-app
│   ├── nextjs-13-app
│   ├── nextjs-app
│   ├── vite-app
│   ├── vite-plugin-ssr-app
│   └── vite-ssr-app
├── components
│   ├── app-ui
│   ├── app-ui-themed
│   ├── emtwodeapp-template
│   ├── emtwodeminiapp-template
│   ├── emtwota-template
│   └── emtwozero-template
├── configs
│   ├── eslint
│   ├── jest-presets
│   ├── prettier
│   └── tsconfig
├── core
│   ├── djitsu
│   │   ├── apps
│   │   │   ├── djitsu-web
│   │   │   └── express-app
│   │   ├── desktop
│   │   │   ├── desktop-api
│   │   │   ├── desktop-app-module
│   │   │   ├── desktop-module
│   │   │   └── desktop-signin-client
│   │   ├── djot
│   │   │   ├── djot-app-module
│   │   │   └── djot-web-app-module
│   │   ├── docs
│   │   │   ├── storybook
│   │   │   └── vitepress
│   │   ├── editor
│   │   │   └── editor
│   │   ├── shared
│   │   │   ├── app-module
│   │   │   ├── services-module
│   │   │   └── types-module
│   │   ├── site
│   │   │   └── site-module
│   │   └── web
│   │       └── web-app-module
│   ├── djot
│   │   ├── apps
│   │   │   ├── djot-docs
│   │   │   └── djot-web
│   │   ├── desktop
│   │   │   ├── desktop-app-module
│   │   │   └── desktop-module
│   │   ├── shared
│   │   │   ├── app-module
│   │   │   └── djot-manager
│   │   └── web
│   │       └── web-app-module
│   └── shared
│       ├── apps
│       │   └── docs
│       ├── components
│       │   ├── djot-editor
│       │   └── ui-components
│       ├── desktop
│       │   └── desktop-signin-client
│       └── shared
│           ├── babel-runtime-standalone
│           ├── firebase-module
│           ├── icons
│           ├── rollup-bundler
│           ├── services-module
│           └── types-module
├── core-djitsu-apps
│   └── djitsu-desktop
│       └── packages
│           ├── main
│           ├── preload
│           ├── renderer
│           └── shared
├── core-djot-apps
│   └── djot-desktop
│       └── packages
│           ├── main
│           ├── preload
│           ├── renderer
│           └── shared
├── docs
│   ├── codedoc-app
│   ├── ladle-app
│   ├── m2-docs
│   ├── storybook-app
│   └── vitepress-app
├── packages
│   ├── deputils
│   └── logger
├── services
│   ├── firebase-service
│   │   └── functions
│   ├── pocketbase-service
│   └── trpc-service
└── src
    └── docs

The above is a redacted output of the tree command showing directories up to four levels deep inside the main Djitsu repo, and it just to show (off?) how (needlessly?) complex such a "small" application like Djot can be, or rather is.

While the project lacks many (many!) other features that will someday allow me to more quickly and easily modify and refactor the various aspects of it, such as Unit and Integration tests, the fact that I've completely decoupled the executing applications (e.g. the Electron app, the vite web app, docs, storybook, etc') from the components it is composed of (e.g. djot-desktop-app-module, djot-editor) makes working on each isolated component that much saner forcing me to think and code in a way that makes polluting the various levels of states managed by the overarching application less likely and at times physically impossible.

This leads to sometimes overly complex data traversal which would be easily sidestepped had I written everything in a single package which shares the state with its various components - but over the years I've found that while it's initially cheaper and produces higher volume of code it becomes very difficult to maintain very quickly even in applications that are far simpler that have way less moving parts than what Djot and Djitsu are at this stage.

And don't get me wrong - even at this sprawling codebase and its packages there is still much to be desired just in the isolation/encapsulation of individual components in the app - not to mention other areas of the project that require better and more robust handling (versioning, deployment, documentation, and testing just to name a few).

I hope that the third (or is it the forth?) time is the charm for refactoring the two major state trees of Djot as it stands now - it's aint fun to say the least, and as more features and capabilities are added in the harder it is to take everything in consideration when rewriting such a low-level part of the software, especially without proper tests in place (or peer code review).

But considering the atrocities I've had to make peace with in the name of overall progress I've encountered in past projects (commercial and otherwise) I've been involved in, and the compromises I had to take in the last few weeks this is probably going to be a false hope, a wish that will require a very powerful genie to fulfill.

This is one of the main reasons I absolutely enjoy working on this project for so many years now - during my tenure as a developer-for-hire and as a team or tech lead in the past couple of decades I've encountered many such situations where the codebase and the written moving parts were just plain wrong - but in a commercial setting you rarely have the oppertunity and privilege to refactor something as low-level as the entire infrastructure, the main state trees, and even more rare is success of such an endeavor - as evident in this short experience that I've had to refactor these parts for the third time in three-months time span - a feat that would at the very least raise some eyebrows at the office and few devs and managers would even dare to suggest, much less to attempt.

Djot being a personal passion project still unfunded (I am looking for proper funding and/or sponserships for Djot if you are interested!) I allow myself these indulgences of pursuing the perfect implementation - I know that in a few weeks/months time the current iteration will prove itself short in various aspects despite the enormous investment poured into it, but that doesn't detract from the fact that I'm utterly enjoying writing these parts The Right Way™️ after so many years succumbing to just make do with whatever mistakes have been introduced because that's just how things are when finances are involved and businesses are expected to deliver even if the actual result is far below par (at the best of cases, in most cases) and only its veneer of sexy UI/UX is what makes the software shipped as passable.

But alas, even I and Djot need to keep on shipping, and regardless of the amount of prose I'll endow upon these pages, no software is perfect and probably never will be, and that's just how things are. I wonder if Pygmalion too had to accept some imperfections in order to eventually achieve his perfect goal - or perhaps it's the imperfections themselves that make something more perfect and beuatiful?

This entry is part of a series of posts I'm writing to share my progress and lack-of while working on Djot - a part of coding/development tool/platform I've started way back in early 2020 named Djitsu.