Harald Sack · @lysander07
741 followers · 454 posts · Server sigmoid.social
Last leg in our brief #timeline of (Large) #languagemodels (so far) is 2023, which saw the advent of many new and updated #LLMs:
- BARD #chatbot is introduced by Google
- LLaMA is introduced by Meta
- GPT-4 is introduced by OpenAI.
- LLaMA2.0 is introduced by Meta
- and many others...
#ISE2023 #lecture slides: https://drive.google.com/file/d/1atNvMYNkeKDwXP3olHXzloa09S5pzjXb/view?usp=drive_link
GPT-4 tech report: https://arxiv.org/pdf/2303.08774
@fizise @KIT_Karlsruhe #ai #artificialintelligence #llm #llms #gpt #openai #llama #lamda #bard
#timeline #languagemodels #LLMs #ChatBot #ise2023 #lecture #ai #artificialintelligence #llm #gpt #openai #llama #lamda #bard
Harald Sack · @lysander07
741 followers · 454 posts · Server sigmoid.social
Next stop on our brief #timeline of (Large) #LanguageModels is 2022:
InstructGPT is introduced by OpenAI, a GPT-3 model complemented and fine-tuned with reinforcement learning from human feedback.
ChatGPT is introduced by OpenAI as a combination of GPT-3, Codex, and InstructGPT including lots of additional engineering.
#ise2023 lecture slides: https://drive.google.com/file/d/1atNvMYNkeKDwXP3olHXzloa09S5pzjXb/view?usp=drive_link
#RLHF explained: https://huggingface.co/blog/rlhf
#ai #creativeai #rlhf #gpt3 #gpt #openai #chatgpt #lecture #artificialintelligence #llm
#timeline #languagemodels #ise2023 #ai #creativeai #rlhf #gpt3 #gpt #openai #chatgpt #lecture #artificialintelligence #llm
Harald Sack · @lysander07
719 followers · 434 posts · Server sigmoid.social
Next stop in our brief #timeline of (large) #languagemodels is 2021:
DALL-E is released by OpenAI and raises test2img to a new level.
Codex is released by OpenAI able to translate natural language into programming code.
WebGPT is released by OpenAI for answering open-ended questions.
LaMDA is introduced by Google.
Slides from #ise2023 #lecture: https://drive.google.com/file/d/1atNvMYNkeKDwXP3olHXzloa09S5pzjXb/view?usp=drive_link
Codex paper: https://arxiv.org/abs/2107.03374
DALL-E paper: https://arxiv.org/abs/2102.12092
@fizise #ai #generativeAI #GPT #dalle #openai
#lamda
#timeline #languagemodels #ise2023 #lecture #ai #generativeAI #gpt #dalle #openai #lamda
Harald Sack · @lysander07
681 followers · 419 posts · Server sigmoid.social
Next leg in our brief history of (Large) #LanguageModel is 2020, when #GPT-3 was released by OpenAI, based on 45TB data crawled from the web. A “data quality” predictor was trained to boil down the training data to 550GB “high quality” data. Learning from the prompt was introduced (few-shot learning)
Lecture slides: https://drive.google.com/file/d/1atNvMYNkeKDwXP3olHXzloa09S5pzjXb/view?usp=drive_link
paper: https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
@fizise #ai #artificialintelligence #creativeai #llm #ise2023 #lecture
#languagemodel #gpt #ai #artificialintelligence #creativeai #llm #ise2023 #lecture
Harald Sack · @lysander07
679 followers · 417 posts · Server sigmoid.social
Next stop in our Brief History of (Large) #languagemodels is 2019: GPT-2 was released by OpenAI as a direct scale-up of GPT, comprising 1.5B parameters and trained on 8M web pages.
Slides (from #ise2023 lecture): https://drive.google.com/file/d/1atNvMYNkeKDwXP3olHXzloa09S5pzjXb/view?usp=drive_link
Paper: https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf
#llm #llms #ai #artificialintelligence #generativeai #gpt #lecture #historyofAI
#languagemodels #ise2023 #llm #LLMs #ai #artificialintelligence #generativeAI #gpt #lecture #historyofai
Harald Sack · @lysander07
679 followers · 417 posts · Server sigmoid.social
@bsletten The #ise2023 summer lecture was not recorded. The intention was to bring students back to university lecture halls. There is a 80% overlap with the #ise2021 lecture which is already on #youtube. However, if you are interested in our latest lecture on #knowledgegraphs, don't miss the (free) #KG2023 online course "Knowledge Graphs - Foundations and Applications" on #OpenHPI which starts in Oct 2023.
https://open.hpi.de/courses/knowledgegraphs2023
@fizise @tabea @sashabruns @Hasso_Plattner_Institute
#ise2023 #ise2021 #youtube #KnowledgeGraphs #kg2023 #openhpi
Harald Sack · @lysander07
679 followers · 417 posts · Server sigmoid.social
Next leg in our brief history of (Large) #language models is then advent of the first (real) pretrained #LLMs: ElMO (Allan Institute for AI, 2017), GPT (OpenAI, 2018) and BERT (Google, 2018).
Slides: https://drive.google.com/file/d/1atNvMYNkeKDwXP3olHXzloa09S5pzjXb/view?usp=drive_link
ELMO: https://aclanthology.org/N18-1202.pdf
GPT: https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf
BERT: https://aclanthology.org/N19-1423
#AI @fizise #languagemodel #nlp #artificialintelligence #generativeai #lecture #ise2023
#language #LLMs #ai #languagemodel #nlp #artificialintelligence #generativeAI #lecture #ise2023
Harald Sack · @lysander07
659 followers · 384 posts · Server sigmoid.social
Next stop in our brief Timeline of (large) #languagemodels from the #ise2023 lecture is the advent of the Graphical Processing Units #gpu. In 1999 Nvidias GeForce 256 was one of the very first, which enabled highly parallel computations for #neuralnetworks
Slides: https://drive.google.com/file/d/1atNvMYNkeKDwXP3olHXzloa09S5pzjXb/view?usp=drive_link
@fizise #artificialintelligence #lecture #ai #machinelearning #llm
#languagemodels #ise2023 #gpu #neuralnetworks #artificialintelligence #lecture #ai #machinelearning #llm
Harald Sack · @lysander07
654 followers · 383 posts · Server sigmoid.social
1997 with the advent of Long Short-Term Memory recurrent #neuralnetworks marks the subsequent step in our brief history of )large) #languagemodels from last week's #ise2023 lecture. Introduced by Sepp Hochreiter and Jürgen Schmidhuber #LSTM #RNNs enabled efficient processing of sequences of data.
Slides: https://drive.google.com/file/d/1atNvMYNkeKDwXP3olHXzloa09S5pzjXb/view?usp=drive_link
#nlp #llm #llms #ai #artificialintelligence #lecture @fizise
#neuralnetworks #languagemodels #ise2023 #LSTM #rnns #nlp #llm #LLMs #ai #artificialintelligence #lecture
Harald Sack · @lysander07
652 followers · 380 posts · Server sigmoid.social
Next step in our brief timeline of (large) #languagemodels from our #ise2023 lecture was statistical language modeling with n-grams based on large text corpora as introduced and popularized by Frederick Jelinek and Stanley F. Chen using statistical tricks like Bayes Theorem, Markov Assumption, and Maximum Likelihood Estimation, etc.
Slides: https://drive.google.com/file/d/1atNvMYNkeKDwXP3olHXzloa09S5pzjXb/view?usp=drive_link
@fizise #nlp #llm #llms #artificialintelligence #ai #lecture #creativeAI
#languagemodels #ise2023 #nlp #llm #LLMs #artificialintelligence #ai #lecture #creativeai
Harald Sack · @lysander07
652 followers · 380 posts · Server sigmoid.social
Went to the university office to collect the #ise2023 final exams for later reviewing together with the @fizise ta team. But for now, a #perfectEspresso at home… because it was too rainy today for walking to Espresso Stazione ☕️ #lecture #coffeechallenge #karlsruhe @KIT_Karlsruhe
#ise2023 #perfectespresso #lecture #coffeechallenge #karlsruhe
Harald Sack · @lysander07
646 followers · 376 posts · Server sigmoid.social
Slide 2 of our Brief Timeline for (Large) #LanguageModels from the last #ise2023 lecture introduced us to #ELIZA, Joseph Weizenbaum's simple #Chatbot from 1966 that simulates a conversation with a psychoanalyst. Weizenbaum was shocked that some persons including his secretary attributed human-like feelings to the computer program...
Slides: https://drive.google.com/file/d/1atNvMYNkeKDwXP3olHXzloa09S5pzjXb/view?usp=drive_link
#nlp #ai #llm #artificialintelligence
#languagemodels #ise2023 #eliza #ChatBot #nlp #ai #llm #artificialintelligence
Harald Sack · @lysander07
636 followers · 372 posts · Server sigmoid.social
One of the final sections of the #ise2023 lecture was an excursion with a #timeline of (Large) #LanguageModels. We started our tour in 1948 with Claude Shannon's seminal work "A Mathematical Theory of Communication""
Slides: https://drive.google.com/file/d/1atNvMYNkeKDwXP3olHXzloa09S5pzjXb/view?usp=drive_link
@fizise #llm #ai #nlp #artificialintelligence #informationtheory #lecture
#ise2023 #timeline #languagemodels #llm #ai #nlp #artificialintelligence #informationtheory #lecture
Harald Sack · @lysander07
628 followers · 362 posts · Server sigmoid.social
As a 2nd topic of this last #ise2023 lecture, we were discussing #KnowledgeGraph Completion. Most simple approach for unsupervised #linkprediction based on (here translation-based) knowledge graph embeddings was explained on the example of Isaac Asimov.
Slides: https://drive.google.com/file/d/1atNvMYNkeKDwXP3olHXzloa09S5pzjXb/view?usp=drive_link
@fizise @enorouzi #scifi #knowledgegraphs #ai #deeplearning #embeddings
#ise2023 #knowledgegraph #linkprediction #scifi #KnowledgeGraphs #ai #deeplearning #embeddings
Harald Sack · @lysander07
625 followers · 359 posts · Server sigmoid.socialOk, I tried out Runway Gen-2. I did some tests with prompt only but also with uploaded images (mostly generated by another generative AI). Lessons learned: 1) don't expect too much...
2) you have to try very often...
3) don't expect too much ;-)
Below you can see the video generated based on my stablediffusion "Singularity" picture from the #ise2023 lecture. #generativeAI #runway #stablediffusion #stablediffusionart #aiart #singularity
#ise2023 #generativeAI #runway #StableDiffusion #stablediffusionart #aiart #singularity
Harald Sack · @lysander07
617 followers · 356 posts · Server sigmoid.social
How can we find out the importance of a node in a #knowledgeGraph? In the last #ise2023 lecture, we were discussing graph centrality measures and how they can be applied in the context of knowledge graphs.
Slides: https://drive.google.com/file/d/1atNvMYNkeKDwXP3olHXzloa09S5pzjXb/view?usp=drive_link
SPARQL query (cf image below, the 100 most "important" #SciFi authors according to #wikidata) https://w.wiki/78Un
@fizise @enorouzi #semanticweb #lecture #ai #datascience #analytics
#knowledgegraph #ise2023 #scifi #wikidata #semanticweb #lecture #ai #DataScience #analytics
Harald Sack · @lysander07
613 followers · 352 posts · Server sigmoid.social
Topics of the last #ise2023 lecture; The Graph in #KnowledgeGraphs, Knowledge Graph Completion, A Brief History of Large Language Models, and Knowledge Graphs and Large Language Models. I will highlight some topics with the upcoming toots...
Slides: https://drive.google.com/file/d/1atNvMYNkeKDwXP3olHXzloa09S5pzjXb/view?usp=drive_link
#llms #languagemodels #deeplearning #linkprediction #kgc #lecture #machinelearning #transformers #gpt @fizise @enorouzi
#ise2023 #KnowledgeGraphs #LLMs #languagemodels #deeplearning #linkprediction #KGC #lecture #machinelearning #Transformers #gpt
Harald Sack · @lysander07
613 followers · 352 posts · Server sigmoid.social
Last #ise2023 lecture of this semester is about to start. 8:00AM is always tough for the students as well as for the professor 🥳 @fizise @KIT_Karlsruhe @enorouzi #ai #machinelearning #KnowledgeGraphs
#ise2023 #ai #machinelearning #KnowledgeGraphs
Harald Sack · @lysander07
610 followers · 349 posts · Server sigmoid.social
Last thing we've discussed in the "Limits of #AI" chapter of the #ISE2023 lecture was the threat of the so-called #Singularity. What is the singularity? Under which circumstances can it possibly happen? How real is this threat and should we better already aim for potential regulations?
Slides: https://drive.google.com/file/d/1LUOA-NiE4nJ4sn5elbXUN4c7YDtK9uXB/view?usp=drive_link
@fizise @enorouzi @KIT_Karlsruhe #machinelearning #deeplearning #philosophy #aiart #stablediffusionart #creativeai
#ai #ise2023 #singularity #machinelearning #deeplearning #philosophy #aiart #stablediffusionart #creativeai
Harald Sack · @lysander07
610 followers · 349 posts · Server sigmoid.social
When discussing the limits of #AI in last week's #ise2023 lecture, we also talked about the Chinese Room Problem introduced by John Searle in 1980.
Slides: https://drive.google.com/file/d/1LUOA-NiE4nJ4sn5elbXUN4c7YDtK9uXB/view?usp=drive_link
#machinelearning #artificialintelligence #deeplearning #lecture #philosophy @fizise @enorouzi
#ai #ise2023 #machinelearning #artificialintelligence #deeplearning #lecture #philosophy