- cross-posted to:
- [email protected]
- cross-posted to:
- [email protected]
cross-posted from: https://lemmy.intai.tech/post/5502
cross-posted from: https://lemmy.intai.tech/post/5501
Can someone give me a simple explanation of the differences between the main branches on this graphic?
All the colored branches are based on the Transformer neural network architecture. If we allow for a lot of handwaving, a transformer allows each word to look at other words in the sequence in order to make decisions/produce output.
In a decoder transformer architecture, each word can only look at the words that come before it in the sequence. Such models are naturally suited for the task of next word prediction: you can ask each word to predict what word comes next, and they can look at all the words before them, but cannot “cheat” by looking ahead. These models are used for text generation: start with an empty sequence, and repeatedly predict what word should come next.
On the other hand, in an encoder architecture, each word can look at every other word, in front of and behind it. You can’t use these for next word prediction, since they can cheat, but they are commonly used for masked language modeling, a task where we delete some words from the input and then try to predict which words were deleted. It isn’t straightforward to generate text with these models, but they can learn a lot of deep statistical properties of text, which can then be used for other tasks/models.
Some model architectures might use both encoders and decoders, but I am not too familiar with how they are used for language modeling. The classic example of such a model is a translation system, which would use an encoder to “encoder” the source language text, and the decoder to generate target language text from that encoding.
Thanks for the explanation, I think I can wrap my head around that.
I was going to ask which are transformer based but it’s actually nearly all of them (non grey) 😳. I thought BERT was still pre transformers. Goes to show how quickly things have evolved in recent years.
yeah, the idea of making it transformers all the way down was key, each node is a complete Turing machine.
Can someone explain Decoder only vs Encoder/Decoder?
Also of note is RWKV, the only purely RNN structured LLM. I’m keeping an eye on that one because of the theoretically infinite context length
I don’t know what transformer vs non transformer means. But the tree is interesting!
Where’s Eliza?
Whooooa. Neat.
Doesn’t Bard run on Lamda?
Wikipedia says it was initially based on LaMDA then PaLM.
So in that case, shouldn’t the branch reflect that? It looks as though they’re different projects.
And currently PaLM 2
Companies that open source their code, you love to see it.
deleted by creator