• 0 Posts
  • 16 Comments
Joined 1 year ago
cake
Cake day: January 17th, 2024

help-circle





  • The 1.5B version that can be run basically on anything. My friend runs it in his shitty laptop with 512MB iGPU and 8GB of RAM (inference takes 30 seconds)

    You don’t even need a GPU with good VRAM, as you can offload it to RAM (slower inference, though)

    I’ve run the 14B version on my AMD 6700XT GPU and it only takes ~9GB of VRAM (inference over 1k tokens takes 20 seconds). The 8B version takes around 5-6GB of VRAM (inference over 1k tokens takes 5 seconds)

    The numbers in your second link are waaaaaay off.


  • Thanks for telling me lol. I remember sharing your enthusiasm when I started.

    If you don’t mind me sharing, here are some tools I use the most in the console:

    • htop: resource monitoring and process killing. Mint has a GUI alternative
    • btop: better resource monitoring, but worse process killing than htop.
    • lazygit: amazing interface for git. Seems hard to get started, but IMO, not at all. There are GUI alternatived.
    • tmux: multiple consoles and console manager. A bit hard to get started.
    • nano: text editor. Reeaaaallly simple to use, prefer it over emacs and vi/vim.
    • grep: you already know this one.
    • cronjobs/crontab: allows you to run periodical commands. Say, a cleanup script all days at 7:08 AM.

    Also, some GUI programs I love:

    • KDE Connect: device pairing with your cellphone and PC. Includes remote mouse input, multimedia control and file sharing.
    • Steam: Almost all the games I play on Steam run flawlessly on Linux.
    • Stellarium: astronomy/planetary app.

    Pick your poison lol. If you don’t mind, we can start talking via ptivate message.