Hey everyone, I’m looking for a way to use an open source local large language model (LLM) on Linux, particularly on low-spec hardware like Raspberry Pi, to generate lengthy, coherent stories of 10k+ words from a single prompt. I recall reading about methods described in scientific papers such as “Re3: Generating Longer Stories With Recursive Reprompting and Revision”, announced in this Twitter thread from October 2022 and “DOC: Improving Long Story Coherence With Detailed Outline Control”, announced in this Twitter thread from December 2022. These papers used GPT-3, and since it’s been a while since then, I was hoping there might be something similar made using only open source tools. Does anyone have experience with this or know of any resources that could help me achieve long, coherent story generation with an open source LLM? Any advice or pointers would be greatly appreciated. Thank you!
Well check out gpt4all.
There are a number of different models to download and run.
I think your real challenge is to find the right model.
Do you know any programming languages?
Gpt4all is easy to use
Raspberry Pi
This is a famously high-compute problem and you want to chuck it on a pi? Most llm models require a good GPU.
Just want to piggyback this. You will probably need more than 6gb vram to run good enough models with a acceptable speed and coherent output, but the more the better.
Using a CPU is possible thanks to llama,cpp, but you’d still want more RAM than the Pi offers.
Using a raspberry pi seems very underpowered, best case you will be limited to something like 4-7B models on a 8GB RPi4. You may need to configure it with very long timeouts and expect it to output something like a token every few minutes.
I ran a 6B model on a i7 without a GPU and it didn’t give good results before I got CUDA up and running. Probably because of timeout.
I think you’ll struggle with the coherent part
Most LLMs can do a few paragraphs and stay on topic but after thet they need better guidance, usually by changing the prompt to stay relevant. 10k+ words can be hard for normal authors to stay coherent on a single prompt, let alone a GPT-3 model
Reading the articles you attached OP, this is exactly the technology they are still struggling with. I dont think any open source consumer level models will have quite what you’re looking for… yet!
How much time do you have? Because even small models will take alot of time on that kind of hardware to spit out a long text…
And the small models arent that great. I think the current best and economic model would be a mistral, mixtral or dolphin.
If you got the power, nous-capybara is very good and “only” 34B parameters (loading alone needs like 40GB of memory).-
you need a gpu or gpu resources to run a GPT level model
-
stop thinking of it as “make me a story” and start thinking of it as “let’s make a story together”
First talk back and forth over the basic idea.
Then get a short outline.
Then work on each chapter of the outline.
Then repeat and refine.Try to keep overall queries under a certain character limit. Google what some good ranges are.
Make sure to save each conversation, and use the outline and previous chapter as the start of your conversation for making the next chapter.
The AI won’t write it for you, but it can be an amazing accelerator if you’re willing to put in the work.
-
If you want something local and open source, I think your main problem will be the number of parameters (the
bthing). ChatGPT-3 is (was?) noticeably big and open source models are usually smaller. There is, of course, an exchange about how much the size of the model matters and how the quality of the training data affects the results. But when I did a non-scientific comparison ~half a year ago, there was a noticeable difference between smaller models and bigger ones.Having said all of that, check out https://huggingface.co/ it aims to be like GitHub for AIs. Most of the models are more or less open source, you will only need to figure out how to run one and if you have some bottlenecks on PI
I use machine learning/ai pretty much daily and I run stuff at home locally when I do it. What you’re asking is possible, but might require some experimentation on your side, and you might have to really consider what’s important in your project because there will be some serious trade-offs.
If you’re adamant about running locally on a Rasberry Pi, then you’ll want a RPi 4 or 5, preferably an RPi 5. You’ll also want as much RAM as you can get (I think 8gb is the current max). You’re not going to have much VRAM since RPi’s don’t have a dedicated graphics card, so you’ll have to use it’s CPU and normal RAM to do the work. This will be a slow process, but if you don’t mind waiting a couple minutes per paragraph of text, then it may work for your use case. Because of the limited memory of Pis in general you’ll want to limit what size LLM models you use. Something specialized like a 7b story telling LLM, or a really good general purpose model like Mistral Open Orca 7b is a good place to start. You aren’t going to be able to run much larger models than that, however, and that could be a bit creatively limiting. As good as I think Mistral Open Orca 7b is, it lacks a lot of content that would make it interesting as a story teller.
Alternatively, you could run your LLM on a desktop and then use an RPi to connect to it over a local network. If you’ve got a decent graphics card with like 24gb of VRAM you could run a 30b model locally, and get decent results fairly fast.
As for the 10k words prompt, that’s going to be tricky. Most LLMs have a certain number of tokens they can spit out before they have to start up again. I think some of the 30b models I use have a context length of 4096 tokens… so no matter what you do you’ll have to tell your LLM to do multiple jobs.
Personally, I’d use LM Studio (not open source) to see if the results you get from running locally are acceptable. If you decide that its not performing as well as you had hoped, LM studio also generates python code so you could send commands to an LLM on a local network.
I use koboldAI
Is this something that a TPU module could accelerate?
I wondered this out loud previously and what I got back is that LLMs need more RAM than they do compute, so not really.
Makes sense, thank you.
Perhaps if you broke your story up into chapters, each of those would be a more digestible chunk for the LLM?
Just FYI you are not the only one thinking about using LLMs on a RPI:
Install ollama and create an alias for
ollama run <model>for ease of access is what I did
