• 0 Posts
  • 12 Comments
Joined 2 years ago
cake
Cake day: July 10th, 2024

help-circle


  • Decades worth of windoofs hate. Finally did the full switch a few months ago. I have been using windoofs for gaming only anyway and professionally I already had plenty of Linux experience.

    Linux isn’t great either though and can be a real pain in the ass. Recently had to fix few lines of code in the kernel, because I had no audio. But the fact that one can do it in principle is already a huge win over windoofs. And although some games stubbornly won’t run (stable), a lot do without having to tinker much, thanks to Wine, Proton, and a couple of startup parameters.

    Professionally I mainly use Ubuntu, for gaming I did so at first as well but switched to CachyOS recently.










  • If we’re speaking of transformer models like ChatGPT, BERT or whatever: They don’t have memory at all.

    The closest thing that resembles memory is the accepted length of the input sequence combined with the attention mechanism. (If left unmodified though, this will lead to a quadratic increase in computation time the longer that sequence becomes.) And since the attention weights are a learned property, it is in practise probable that earlier tokens of the input sequence get basically ignored the further they lie “in the past”, as they usually do not contribute much to the current context.

    “In the past”: Transformers technically “see” the whole input sequence at once. But they are equipped with positional encoding which incorporates spatial and/or temporal ordering into the input sequence (e.g., position of words in a sentence). That way they can model sequential relationships as those found in natural language (sentences), videos, movement trajectories and other kinds of contextually coherent sequences.