Benjamin Han · @BenjaminHan
286 followers · 477 posts · Server sigmoid.social
3/
- On average it achieves 64.33% accuracy on reasoning hence is not a reliable reasoner. It performs worse on induction compared to deduction/abduction. But it's good at commonsense, causal and analogical reasoning (screenshot 3).
CAVEAT that I didn't see addressed in the paper (or maybe impossible to address): the possibility that some of these test sets are included in the training set.
#NLG #nlp #nlu #NLGPU #nlproc #deeplearning #ai #paper
Benjamin Han · @BenjaminHan
286 followers · 476 posts · Server sigmoid.social
2/
- Compared to many #SOTA #LLMs, ChatGPT outperforms almost all on zero-shot tests (except for Open-domain KGD), and a few (3) finetuned ones (screenshot 1).
- Focusing on #Reasoning, evals are broken into logical (deduction, induction and abduction), temporal, spatial, mathematical, #commonsense, etc (screenshot 2).
#sota #LLMs #reasoning #commonsense #NLG #nlp #nlu #NLGPU #nlproc #deeplearning #ai #paper
Benjamin Han · @BenjaminHan
286 followers · 475 posts · Server sigmoid.social1/ A comprehensive -- #multitask, #multilingual, #multimodal -- #evaluation of #ChatGPT:
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity. arXiv [cs.CL]. https://arxiv.org/abs/2302.04023
#multitask #multilingual #Multimodal #evaluation #chatgpt #NLG #nlp #nlu #NLGPU #nlproc #deeplearning #ai #paper
Benjamin Han · @BenjaminHan
150 followers · 170 posts · Server sigmoid.social(2/2)
We are interested not just in the construction (#KGC) and serving aspects of KG, but also in how to apply #KG to any downstream AI tasks to make them better: all stages of #NLGPU, #inference and #reasoning, and many user-facing features too!
Love to learn more from the community about the latest development in all related areas!
#KGC #kg #NLGPU #inference #reasoning #PaperThread #industry
Benjamin Han · @BenjaminHan
150 followers · 170 posts · Server sigmoid.social#Apple 2023 #AIResidency has opened, deadline 12/7: https://machinelearning.apple.com/updates/aiml-residency-program-application-2023
Come to work with us at #AppleKnowledgePlatform (direct link): https://jobs.apple.com/en-us/details/200438172/ai-ml-resident-apple-knowledge-platform?team=MLAI
#Hiring #KnowledgeGraphs #NLGPU #MachineLearning #DeepLearning #KnowledgeRepresentation #Reasoning #AI #Apple
Please boost!
#apple #AIResidency #AppleKnowledgePlatform #hiring #KnowledgeGraphs #NLGPU #machinelearning #deeplearning #knowledgerepresentation #reasoning #ai
Benjamin Han · @BenjaminHan
150 followers · 170 posts · Server sigmoid.socialMy #introduction: currently working on #knowledge #knowledgeGraphs and #NLGPU (#NLG + #NLP + #NLU) at #Apple Knowledge Platform. Nice to see you all!
#Introduction #knowledge #KnowledgeGraphs #NLGPU #NLG #nlp #nlu #apple