So I had a micro PC that was running one of my core services and it only supports NVMe drives. Unfortunately, this little guy cooked itself and I’m not in a position to replace the drive. The system is still good and is fairly powerful, so I want to be able to reuse it.

I’m thinking I want to set up some kind of netboot appliance on another server to be able to allow me to boot the system without ever having a local disk. One thing I want to is run some docker images (specifically Frigate) but i wont be able to write anything to persistent storage locally. NFS shares are common in my setup.

Is it even possible to make a ‘gold image’ of a docker host and have it netboot? I expect that memory limitations (16GB) will be my main issue, but I’m just trying to think of how to bring this system back into use. I have two NAS appliances that I can use for backend long term storage (where I keep my docker files and non-database files anyway), so it shouldn’t be too difficult to have some kind of easily editable storage solution. I don’t want to use USB drives as persistent storage due to lifespan concerns from using them in production environments.

  • Possibly linux@lemmy.zip
    link
    fedilink
    English
    arrow-up
    3
    ·
    5 months ago

    I wouldn’t do this. If you are spending the time to do netboot you might as well get a proper boot drive.

  • Scrubbles@poptalk.scrubbles.tech
    link
    fedilink
    English
    arrow-up
    2
    ·
    5 months ago

    Kind of, but probably not. I started writing this and was like “totally it could be stateless”. Docker runs stateless, and I believe when it starts it is still stateless (or at least could be mounted on a ramdrive) - but then I started thinking, and what about the images? Have to be downloaded and ran somewhere, and that’s going to eat ram quickly. So I amend to you don’t need it to be stateful, you could have an image like you talked about that is loaded every time (that’s essentially what kubernetes does), but you will still need space somewhere as scratch drive. A place docker will places images and temporary file systems while it’s running.

    For state, check out docker’s volume backings here: https://docs.docker.com/engine/storage/volumes/. You could use nfs to another server as an example for your volumes. Your volumes would never need to be on your “app server”, but instead could be loaded via nfs from your storage server.

    This is all nearing into kubernetes territory though. If you’re thinking about netboot and automatically starting containers, and handling stateless volumes and storing volumes in a way that are synced with a storage server… it might be time for kubernetes.

      • ddh@lemmy.sdf.org
        cake
        link
        fedilink
        English
        arrow-up
        0
        ·
        5 months ago

        Correct, I run docker on a compute host that has no local storage. The host’s disks are on iSCSI LUNs.

        • Passerby6497@lemmy.worldOP
          link
          fedilink
          English
          arrow-up
          0
          ·
          5 months ago

          That’s really good to know. Do you ever have issues writing database files on those disks? Database files on nfs mounts have been the bane of my existence.

          • ddh@lemmy.sdf.org
            cake
            link
            fedilink
            English
            arrow-up
            1
            ·
            5 months ago

            FWIW I run only very small databases e.g., sqlite ones shipped with applications, but haven’t had any problems in about a year now, and nothing that wasn’t recoverable from backup.

    • Passerby6497@lemmy.worldOP
      link
      fedilink
      English
      arrow-up
      0
      ·
      5 months ago

      So I amend to you don’t need it to be stateful, you could have an image like you talked about that is loaded every time (that’s essentially what kubernetes does), but you will still need space somewhere as scratch drive. A place docker will places images and temporary file systems while it’s running.

      Putting the image somewhere is easy. I’ve got TBs of space available on my NAS drives, especially right now with not acquiring any additional linux ISOs.

      For state, check out docker’s volume backings here: https://docs.docker.com/engine/storage/volumes/. You could use nfs to another server as an example for your volumes. Your volumes would never need to be on your “app server”, but instead could be loaded via nfs from your storage server.

      I’ll check that out. If that allows me to actually write databases to disk on the nfs backing volume, that would be amazing. That’s the biggest issue I run into (regularly).

      This is all nearing into kubernetes territory though. If you’re thinking about netboot and automatically starting containers, and handling stateless volumes and storing volumes in a way that are synced with a storage server… it might be time for kubernetes.

      I don’t think I’ve ever looked into kubernetes. I’ll have to look into that at some point… Any good beginner resources?

      • Scrubbles@poptalk.scrubbles.tech
        link
        fedilink
        English
        arrow-up
        1
        ·
        5 months ago

        Personally I suggest k3s, setting up a test cluster and playing with it. For volume management I use longhorn. It’s a HUGE learning curve, but it’s officially something companies will shell out big money for too if you’re willing to learn it. Soup to nuts from setting up test cluster and playing with it all the way to all of my services running was about 4 months of tinkering for me- but I’ll never go back

  • mosiacmango@lemm.ee
    link
    fedilink
    English
    arrow-up
    2
    ·
    edit-2
    5 months ago

    Id be pretty wary of using any system that “cooked” an nvme. That not the sign of an actual healthy system.

    Was the failure just heat damage?

    • Passerby6497@lemmy.worldOP
      link
      fedilink
      English
      arrow-up
      1
      ·
      5 months ago

      I’m actually not 100% what killed the drive. It could have been an issue with the drive wearing out, but my services didn’t write much locally and it wasn’t super old so I assume its a heat issue with a fanless micro system. I try to write everything important to my NASs so I don’t have to worry about random hardware failures, but this one didn’t have backups configured before it failed. Other than the drive issue its been solid for 1.5-2 years of near constant uptime.

      • Jimmycakes@lemmy.world
        link
        fedilink
        English
        arrow-up
        1
        arrow-down
        1
        ·
        5 months ago

        Unless you are writing petabytes the nvme did not just burn “wear” out. Probably shouldn’t do anything until you figured out what caused this failure

        • MangoPenguin@lemmy.blahaj.zone
          link
          fedilink
          English
          arrow-up
          1
          ·
          5 months ago

          Consumer SSDs generally only have a 200-600TBW rating, not petabytes. Its pretty easy to wear one out in a few years installed in a server.

  • mlaga97@lemmy.mlaga97.space
    link
    fedilink
    English
    arrow-up
    1
    ·
    5 months ago

    I’ve done something extremely similar with a custom NixOS iso for my docker VMs to make versioning and backups easier (golden image live disk with SSH+Docker+Dockge shared between all VMs + local persistent storage specific to each VM).

    You can configure frigate via OCI container with custom config, as well as NFS mounts, SSH server, etc and then have a read-only live disk that boots up, mounts NFS share, and then starts up frigate.

  • sugar_in_your_tea@sh.itjust.works
    link
    fedilink
    English
    arrow-up
    1
    ·
    5 months ago

    It’s possible, but I recommend more RAM because you won’t have any swap.

    But adding a network boot is a hassle, I recommend buying another drive, it’ll be cheaper than RAM esp since you don’t need much storage.

  • fruitycoder@sh.itjust.works
    link
    fedilink
    English
    arrow-up
    1
    ·
    5 months ago

    I do it with k3s right now on fedora. I like it personally.

    Nice thing if you use k8s settings up persistent net storage with something like longhorn is an option too.

  • thumdinger@lemmy.world
    link
    fedilink
    English
    arrow-up
    1
    ·
    edit-2
    5 months ago

    You mention frigate specifically. Were you running this on the system when the drive failed, or is this a future endeavour?

    I bring this up because I also use frigate, and for some time I was running with a misconfigured docker compose that drove my SSD wearout to 40% in a matter of months.

    Make sure that the tmpfs is configured per the frigate documentation and example config. If misconfigured like mine was, all of that IO is on disk. I believe the ramdisk is used for temp storage of camera streams, until an event occurs and the corresponding clip is committed to disk.

    Good luck!

    • Passerby6497@lemmy.worldOP
      link
      fedilink
      English
      arrow-up
      0
      ·
      5 months ago

      I don’t want to use a USB for storage, because those aren’t going to have a great lifespan in my experience. I’ve used them as the install media for something like ESX, but I’d rather not run a ‘real’ OS from a disk because I wasn’t impressed with overall lifespan on some of the systems we managed at work.