Took me 9 days to get a tiny AI model running on my old laptop

I got curious about running a small language model on my own machine instead of paying for cloud access every month. Thought it would take an afternoon. It did not. The install failed 6 times because my Python version was too new for the driver, and I kept skipping the error messages like an idiot. Day 4 I finally read the logs and saw it wanted an older build. After that the model loaded but ran so slow I could make coffee between every sentence. Last Friday I got it working with a 3 billion parameter model and it spits out text in about 4 seconds now. Anyone else mess with local models on older hardware, and what size actually runs smooth for you?

1 comments

Log in to join the discussion

Log In

1 Comment

umab866h ago

Boy, nine days to save what, ten bucks a month? At that rate the electricity your old laptop burned trying to run that thing probably cost more than just paying for the cloud. And four seconds between replies still sounds painful, that is not a chat, that is a slow letter in the mail. I get wanting to own your setup, but sometimes the cheap and easy path is cheap and easy for a reason. Runs smooth? On old hardware, nothing runs smooth, you just pick which kind of slow you can live with.

7