Local models for agent work

Local models are AI models that you download and run on your own computer. Speech-to-text already runs this way on my Mac (see MLX Whisper). This week I tested local language models for agent work too.

I tried two small ones on my Mac, which has 16 GB of memory. Both were no cheaper and no faster than cloud models when I sent one job at a time. The larger one filled the memory. The Mac then used its slower disk as extra memory.

Locals, a new open-source program that runs local models on a Mac, claims up to three times faster answers when many agents run at once. I have not tested it on my Mac.

For a one-person business this is not practical yet. By my own estimate, someone with $10,000 to $50,000 for hardware may still want it. The data stays on their own machine, and answers can come faster. I will test it again when the hardware or the models change.