Yet Another (hopefully better) idea on UI, layouts and text

General
1

I’ve been studying user interfaces and text rendering for so long I think I’ve lost count how many libraries I’ve tried from multiple different languages. I’ve tried Flutter (Dart) - lasted about 5 minutes because I didn’t like it - Windows Forms (my most proficient use case), Java’s Swing (for a brief time), Web tech (of course), Dear ImGui, RayGui, wxWidgets, libagar (briefly), and many more.

I can certainly understand why people could get frustrated with any one of these libraries and most people end up settling for a layout library and building their own widgets on top. This seems to be the way to go, right?

Well, I’ve decided to make a repo over on GitHub discussing what layout libraries and widget building would likely look like in the future.

I’ve pondered over this for long enough and it’s time to get my attention on this as it’s an area I’m most passionate about. Ironically, it’s not something I most enjoy fiddling about with, because UI should be trivial, but I have wondered why people always make the decision to build their own. Is it truly their way, or is it simply because the market just hasn’t filled the gaps that people crave?

Out-of-the-box pre-built widgets not reliant on a framework comes to mind. Dear ImGui comes closest to this solution, hence why it’s so popular, but here you have to contend with the fact that Dear ImGui is limited in one aspect: it’s aesthetic. Yes, you can tweak the colours, spacing and supply your own font, but that’s about it. If you want anything else, you’re better building your own custom UI engine - likely with a hand-rolled SDF shader and a layout engine like Clay.

Also, if you want IME because your audience is, let’s say, business customers and they’re worldwide, you end up back to retained mode UIs since they’re the most battle tested for IME text input.

Marketing, sales and purchasing departments like to be sold on functionality and features, but the biggest feature is the user experience, and this is primarily a UI concern. If you’re building apps sold to businesses at a large scale, IME is going to be top priority.

In my opinion, aesthetically pleasing UI will never beat a borked IME system. Most people choose to let the OS do the IME for them, which is a fair trade off for games, but not for general UI. If your audience is businesses, there are few choices on the market for UI at present:

  • The Web - as unironic as this sounds, the web (unfortunately) offers the most standardised, complete and cross-platform solution for both UI and text input with full IME support. That means support for mid-flight character input, bidirectional text, and more.
  • Gtk, Qt or wxWidgets - All these play on desktop native UI and abstract as necessary, with Qt probably the biggest of the three. It is because they use the native text input controls supplied by the OS they get IME basically for free.

That’s about as far as I know and I’m not aware of other complete solutions besides the native APIs supplied by each OS, but I’m not going to bother going there.

One argument can be said with all of this and it’s worth raising as it was mentioned in an interview involving Nic Barker, the creator of Clay, who said, and I quote:

Some people go as far as to create a native text input widget and draw it off-screen, only to then feed the results back into the frame loop.

A hacky solution, but works for IME support all the same. The question this raises: do people want the hacky solution or a seamless, high performant experience?

My final question I would then raise is probably the text input caching method. Piece Table/Tree? Gap buffer (unlikely)? Line-by-line split? Or whatever else that exists.

I think given the number of times I’ve seen people try and work out their own UI library, I think it is reasonable to say that these people fall into one of two camps:

  • They’re doing it for fun and to learn, which is fair enough; or,
  • They have tried dozens of solutions but none of them fit the bill (people like me)

I can’t fathom the idea of trying to build my own UI library with widgets and built-in IME text input, but considering there is no cross-platform solution outside the Web and not wanting to touch HTML/CSS/JavaScript and all the desktop Virtual JS Runtimes that come with it, it seems it’s worth discussing these points before attempting such an ambitious project. I did try it with my AI generated InterfaceAPI project, and in many ways is the solution, but I’ve come to accept two things from this experience:

  • Most people do not like AI generated projects and don’t want anything to do with it, and I can understand why.
  • As an R&D project testing the ability of AI, it was fun to learn and experience everything that I’ve discussed with AI and what works and what doesn’t.

In many ways, I think the next project will be to accommodate everything I’ve learned, sit down and take a serious deep dive into the problems that most people face with UI development:

  • Text Rendering (lack of text input support in most cases, and where there is a lack of IME)
  • Building a layout-less widget system relying only on rects (potentially) so that it can be dropped into any layout system of your choice
  • Building stateful primitives that don’t rely on sync or reactivity to make data and UI work.

On that last point, most people I think probably like the ImGui paradigm for this because it is simply the case of passing in a pointer to a state you own. This makes the most sense and I would probably stick with this approach, it would be the case of working out if my other project would work well with this. Possible union:

ValueRef :: union($T : typeid) {
    T,
    Reference(T),
}

// for widgets that need a retained state, like text editors or graphs.
Reference :: struct($T : typeid) {
    value : ^T,
    change_counter : u64,
}

Sorry for the long post, just thought it would be worth discussing. I’d certainly be interested what people think and if it’s worth the pursuit on the points I’ve made above.

2 Likes
2

Take a look at the following GUI framework written in Odin using SDL. I have already talked about this in another discussion:

3

Another thing that should be noted is that retained UI’s aren’t the only things that can use IME support. SDL offers it’s text editing events that allow you to also tie into the IME events as well and it works quite well (and is cross-platform). It does require the application (or UI framework) to retain some state but it ends up being quite trivial to do and isn’t more state than you would have already had.

4

Maybe. I haven’t tried it, but does it support mid-flight character rendering? That’s generally a render concern, though, not a key input concern. For true IME, you’d have to cache the key sequence somewhere and render the glyph it resolves to.

Does SDL have this mechanism, because if so, that would certainly solve the IME story.

And by the way, my comment on retained mode UI being the most battle tested with IME was more to do with rendering, not necessarily the event loop. Most native text fields - like Win32 TextField will support Microsoft IME, for example, which gives it direct access to the pre- and mid-flight character sequences as you type and render that glyph along with an indicator that it’s an in-progress character.

I’m not familiar with the SDL3 character event system, so I will take a look, but my understanding is that most libraries that provide character events only know the last character that was input by the user and doesn’t cache character sequences by itself.

5

SDL does properly handle the composition apis (which is what i imagine the mid-flight rendering comment was about). as for character rendering, that’s just text rendering which isn’t particularly new/unique. you can render the contents of the text box as you normally would and respond to the text events and things will be handled for you. in fact, SDL goes into some use of the api here: https://wiki.libsdl.org/SDL3/BestKeyboardPractices#the-chat-box

6

Thanks for this. Just a note as that article says exactly what I thought was going to happen:

If you’re writing a fullscreen game, you might want to render IME UI yourself, so you’d set SDL_HINT_IME_IMPLEMENTED_UI appropriately and handle the SDL_EVENT_TEXT_EDITING and SDL_EVENT_TEXT_EDITING_CANDIDATES events.

Okay, yes, it does also say that the UI can also be handled by the OS. Great for users of SDL. That said, I’m not familiar with SDL that well.

I’m most familiar with Raylib, and my exposure with SDL has mostly been building SDL statically into Raylib for it’s Wayland support on Linux, remaining on the Raylib ecosystem.

I would certainly be curious if GetCharPressed does the text input mentioned - I assume so if this is the only way to get character presses through SDL, but candidate windows would likely be a manual wiring without direct SDL access I assume.

7

As I continue researching this “project” if you will, I have noticed there is a proposal for the potential to expand somewhat into the declarative realms with the with keyword or something similar in the Odin language as found here.

An interesting discussion, but it did get me thinking more about how I’m going to write the next article for this repo, as it’s important for the UI system.

My original thesis for the next article was going to be about “Incorporating Clay”, which uses @(deferred_*) procedure attributes on it’s UI proc to match its equivalent macros in C.

I think the possible issues with this is that these are features “bolted onto the language” that is designed to fix a specific problem, judging by the discussion previously linked.

UI wants to be declarative, users want to declare UI elements (like HTML), but incorporate it with the benefits of an imperative language.

I, myself, have tackled this with my UI library. I’ve tried XML which involved using Lua for event call backs. I didn’t like it, so it got ditched. I tried a custom DSL with markup designed to generate imperative Odin code. I didn’t like that due to complexity creep but not the worst idea. I tried a macro CLI processor by scanning for tags in Odin and doing code expansions at their relevant call sites. Didn’t like that either, so it got ditched.

I’m now trying an imperative View system that’s more Odin code. It’s far less declarative than the other three options, but so far I am not having many “feelings of dread” over it just yet.

Declarative UI is “nice”, but does “nice” and “convenient shortcuts” equate to the “joy of programming”? If HTML is anything to go by, I would hazard a guess the answer is “no”. Invariably, when people tackle the declarative approach, everyone wants it different. That’s not necessarily a bad thing, as it opens the doors for innovative solutions, but used by fewer and fewer people. Eventually, users only use their UI library because they think their different way is better than anyone else’s, and none of them are complete, lack even the most basic features, generally sluggish, and are not designed for general use. They are designed for that one developer’s goals, not the goals of the general developer.

I’m going to continue writing articles for this repo, but I think my next one will be discussing how a “layout-less” system might look like. A generic command based system might be the way to go, but basically reinventing PostScript, and we already have that (cough SVG cough).

Layout should remain an imperative concern, in my opinion, and decouple widgets from the layout. A few reasons why I think this:

  • Layout is concerned with where things go, not how things behave. Widgets are behaviour + data.
  • Widgets whose event systems are tightly defined coupled with layout makes race conditions inevitable. What I mean by this is that a popup widget whose menu option click event is meant to close the popup, can re-open the popup next frame because the click was not cleared by the time the event calling the popup open was re-evaluated, so you get in effect, two clicks in the same frame, one for a button in the popup, one for the button that opened the popup. When widgets handle themselves in isolation, this isn’t a problem.
  • Even in the case where event handling is handled carefully and managed once the first layout has passed, the cases where event handling becomes tricky is for text input widgets and drag & drop mechanisms. A pre-built layout before the update pass is easier to manage.

The way my current UI library works, and that’s if I decide to continue working on it and replacing some things (should I choose to):

  • Build retained widget tree (init)
  • In the frame loop, it goes: Layout → Update → User Handled Events → Render
  • User Handled events trigger behaviour, update or layout changes, and they just so happen to be decoupled enough that it’s simple to view.

My only gripe with this is the retained widget tree. But in any case, my point is that the layout and widget declaration are separate concerns.

I couldn’t say if this is the “correct” approach. As said before, most people like to do things their way. But I would like to add one more point to this discussion, which is about reusable widgets not reliant on layout.

You probably still have shapes that define what the widgets look like and how they behave, but how this is implemented in practice is likely to be a strong point of contention, mostly for subjective reasons.

A thought occurs that one could declare UI and bind events to the shapes to synchronize user data:

nud_comp := define_component(&components, "NumericUpDown")
set_param(&nud_comp, "value", f64, 0.0)

lay := begin_layout(&nud_comp, min_size= { 75, 30 })
cmd_fill := push_rect(lay, make_id("fill"), { 0, 0, 1.0, 1.0 }, .Percent)
cmd_up := push_rect(lay, make_id("up"), { cmd_fill.w - 20, 0, 20, cmd_fill.h / 2 }, .Pixels)
cmd_down := push_rect(lay, make_id("down"), { cmd_fill.w - 20, cmd_fill.h / 2, 20, cmd_fill.h / 2 }, .Pixels)
cmd_input := push_rect(lay, make_id("input"), { 0, 0, cmd_fill.w - 20, cmd_fill.h }, .Pixels)
cmd_input.accepts_text_input = true
end_layout(lay)

comp_event_subscriber(&nud_comp, proc(e : ^EventHandler) {
  nud_value : f64
  if value, ok := e.params["value"]; ok {
    nud_value = value.? or_else 0.0
  }
  if e.lay["up"].is_clicked {
    nud_value += 1.0
  }
  if e.lay["down"].is_clicked {
    nud_value -= 1.0
  }
  if state, changed := e.lay["input"].text_input; changed {
    temp := nud_value
    if !strconv.parsef64(state, &nud_value) do nud_value = temp
  }

  if event_diff(e, "value", nud_value) == .Changed {
    e.triggers += { NUD_VALUE_CHANGED }
  }
})

This is still imperative code, as you can quite clearly see. This is not intended to be working code, but the idea is that usage looks like this:

nud := numeric_up_down(make_auto_id(), &value, &rect)
if NUD_VALUE_CHANGED in nud.triggers {
  // do something on a custom trigger
}

Where it’s definition looks like this:

numeric_up_down :: proc(id : Id, value : ^f64, rect : ^Rect) -> ^ComponentInstance {
    gui := get_global_ui()
    if instance, exists := gui_get_instance(id); exists {
        instance_perform_update(&instance)
        return &instance
    }

    instance := create_component(gui, "NumericUpDown", rect^)
    set_instance_param(&instance, "value", f64, value^)
    
    return &instance
}

What I’m getting at here is the gap between Clay and Dear ImGui.

You have pure layout libraries but you have to build the widgets yourself and that could be difficult to do as you still need to a lot of heavy lifting.

Full frameworks like Dear ImGui but opinionated and stuck with its limitations.

I am strongly considering filling in the gap that enables Clay and other layout users to be able to pick up a “widget definition library” or something like that enabling them to build their own widgets without all the complexity like text input, drag & drop, etc.

So, here comes the ultimate question: do people want this?

I apologise that this long rant came to one simple question, but I thought the context was necessary.