Welcome! This is the sixth and final lesson on how to build fullstack apps with Bolt and Supabase If you’re just joining, you’re in luck, ‘cause we already have tons of content for you to enjoy while turning your app from just an idea to deployed web application in just an afternoon. Before you dive into this lesson, here’s where you can find Part 1 , Part 2 , Part 3 , Part 4 , and Part 5 if you want to get up to speed (which you probably do, otherwise, what exactly are you going to deploy? 🤔)

In the last article we covered statistical metrics like Perplexity, BLEU, ROUGE and more, as well as some of the statistical concepts that underpin them, their strengths (accuracy, reliability) and weaknesses (no subjective focus, use of reference texts. Between human evaluation (manual testing) and statistical measures we get a mix of high-value qualitative assessment on a small part of the test surface, and a rigorous but limited view on a wider area. That still leaves a lot of middle ground uncovered! That’s why there’s been a push the last few years to get coverage for the space between - something that has a level of subjectivity and nuance but that also scales up. This is where LLM-as-a-Judge comes in. In our manual testing for LLMs article I compared this to a kind of ouroboros where AI validates AI - and rightly so, that isn’t necessarily a bad thing. LLMs are able to do some things better than humans and LLM-as-a-Judge plays to those strengths - but it does not replace the need for human oversight and statistical assessment. There’s also metrics that combine LLM-as-a-Judge with statistical metrics - but we’ll talk more about that later.

In one of our earlier articles , we touched on statistical metrics and how they can be used in evaluation - we also briefly discussed precision, recall, and F1-score in our article on benchmarking . Today, we’ll go into more detail on how to apply these metrics more directly, and more complex metrics derived from these that can be used to assess LLM performance. This is a standard measure in statistics, and has long been used to measure the performance of ML systems. In simple terms, this is a measure of how many samples are correctly categorised (true positives) or predicted by a model out of the total set of samples predicted to be positive (true positives + false positives). If we take a simple examples of an ML tool that takes a photo as an input and tells you if there is a dog in the picture, this would be:

In our previous article , we talked about the highest level of testing and evaluation for LLM models, and went into detail about some of the most commonly used benchmarks for validating LLM performance at a high level. Today, we’re going to look a at some more fine-grained evaluation metrics that you can use while building an LLM-based tool. Here we make the distinction between statistical metrics - that is those computed using a statistical model - and more generalised metrics that attempt to measure the more ‘subjective’ elements of LLM performance (such as those used in manual testing) and that use AI to evaluate how useful a model is in its given context. In this article we’ll give an overview of the different classes of metrics used and cover human evaluation and its importance before moving on to common statistical metrics and LLM-as-Judge evaluations in the following articles.

Welcome! This is part 5 of our course on how to build fullstack apps with Bolt and Supabase If you’re just joining, I highly recommend you take the course in the correct order before diving into this one. Here you can find Part 1 , Part 2 , Part 3 , and Part 4 .

Everyone knows chatgpt, but how do modern large language models fully work? The fundamentals start at the transformer. This workshop is a workshop to dymstify the transformer and be able to run through concept to code on how the transformer work. This workshops combines concept at an intutive level, to code, to math all with the intent at providing an end to end understanding at the fundamentals of large language models.

<!-- Your template is like this (markdown): --> Learn the strategies senior engineers use to configure their projects to run securely, anywhere with environment variables. <!-- ( the summary is short - one or two lines ) -->

Large Language Models are reshaping industries, yet integrating them into real-time streaming UIs presents unique challenges. In this course we will learn how to seamlessly integrate LLM APIs into applications and build AI-powered streaming text and chat UIs with TypeScript, React, and Python. Step-by-step, we will build a full-stack AI application with quality code and very flexible implementation. The LLM application in this course includes: - Completion Use-Case (english to emojis) - Chat - Retrieval Augmented Generation use-case - AI Agent Use-Cases (code execution, data-Analyste agent) This app can be used as a starting point in most projects, saving a huge amount of time, and its flexibilty allows new tools to be added as needed. At the end of this course, you will have mastered end-to-end implementation of a flexible and high-quality LLM application. This course will also equip you with the knowledge and skills necessary to create sophisticated LLM solutions of your own.

This course will teach you how to fetch in a performant way for React without relying on a third-party library or the useEffect hook. We will build a custom data fetching library from scratch that will prefetch, cache, and address all the common issues with fetching data.

Learn Pro Patterns for Hooks, Testing, Redux, SSR, and GraphQL

Practical Security for Busy People

Learn Data Structures and Algorithms in JavaScript

A Field Guide to Your New Career

The Complete Guide to Developing Data Visualizations with D3