AI Talks by Zeo (Episode 2)

Using AI assistants for advanced data analysis, automated reporting, and dynamic marketing dashboards.

Speakers: Yiğit Konur, Ozan Ketenci

03 Sept 2024

Video Transcript

⏱ 29 min read• 5,132 words

Read the full speech or switch languages

— Uh, is this the third or fourth one, Ozan?

— I’m not counting. Counting just makes it harder. If I track it, I start feeling like, "Ugh, that’s enough."

— But somehow, we’re staying consistent. Knock on wood. It’s a busy week, unlike the last one. I’ve got a lot to cover. We’ll dive into a spontaneous intro again. The biggest headline this week is... Probably Sonnet 3.7. I think you’d agree. For the beginners: it’s the latest generation from Anthropic. We had 3.5 version 2 before. Now, we have Sonnet 3.7.

— This is the first one, right? The first hybrid model. It does reasoning and short answers.

— Yeah, exactly. Even if it’s not on paper. I read somewhere it’s based on 3 Mini or 4o Mini. Since every reasoning model has a foundation model, it’s basically the same logic.

— It doesn't work like that. When you prompt O3, it forces the thinking process. It might be brief, but it still does it.

— Right. You mean it can decide for itself, right?

— Exactly. Yeah, exactly.

— True. I’ve been using ThinkBody for a while. I actually canceled my Sonnet sub. But this brings me back into the game. Let’s run a few prompts for those who haven't tried it. So, what are we doing?

— People say the OneShot code is great. Most examples I see are for OneShot code. Some people are even building games with it.

— Yeah, totally. OneShot is producing some wild results. People are mostly talking about the artifact features.

— Yeah, exactly.

— Look, someone made a 3D city with one prompt. We should test that. Let's try: "Write a prompt to generate a 3D city using JS libraries." We need details to model New York. Let's expand our prompt first. These Anthropic model capabilities are pretty fun.

— Great start. I love it. Diving right in and getting our hands dirty.

— This is getting exciting. Nice. Whoa, it gave us Brooklyn and Manhattan separately. That’s a bit of a flex.

— A bit much.

— Was all that really necessary? It’s just one line of info. Let’s try two versions. One over-prompted and one human-style.

— Yeah, keep it simple.

— Let’s look at the simplified version. It really leaned into the deep research'ü. It’s super detailed. We’ll use it anyway. Since the prompt is long, it came as an attachment. That’s fine. Let’s use artifact to trigger that section. I didn’t enable thinking, right? Just standard. Let’s see what happens. This tests the model's skills. Let’s also run the detailed prompt with thinking enabled. Let’s go with extended. I’ll turn off the style setting. It was left on. Let’s see the output. We’ll also run the simplified prompt twice. Once with thinking and once standard.

Let's use artifacts to build. We set up our scenario. Okay. We'll also use extended thinking. Let's benchmark it from our perspective. What do we have? A massive prompt written by Anthropic with extended thinking. We want it built with artifacts. Let's see what happens. I don't expect anything mind-blowing. But the libraries it imports might be impressive. The functions are starting to roll in. Manhattan island, Brooklyn, grid, crazy.

— Can it actually do this with artifacts?

— Honestly, it feels like a show. Let's see. We sent four prompts. One should work. That's what I think. Based on what this guy did.

— Are we sure this is an artifact? This isn't an artifact. Trust me. He just had it write and run the code.

— Oh, I wonder why. Yeah, they built this in Cursor. Makes sense.

— Exactly. They wrote it straight in Cursor.

— Maybe it's a JS build. Mine is more ready-made. It uses Vanilla JS. We'll see soon. Both models keep writing. Let's check the model benchmarks. I don't see a separate thinking benchmark here. Unlike other models, it's missing. That's interesting.

— I didn't get that either. It felt way behind. It's a new model. It shouldn't trail 2.0 Flash.

— Yeah, Grok 3 leads the foundational models. I used Grok a lot this week. I really leaned into the deep research and deep search features. Maybe we'll pivot there after the Anthropic stuff. I needed a benchmark analysis report. For different AIs. To use in ThinkBuddy. I had ChatGPT, Grok, and Perplexity run these analyses. I wanted to test their deep research capabilities. Then I merged them with Gemini 2 Thinking. To create unique benchmarks. I use this for the ThinkBuddy prompt router. When you ask a question, it routes it. Should it go to Sonnet or 4o for algebra?

It shouldn't just look at math broadly. It needs to evaluate algebra and geometry separately. I tried to find the top-performing models for every specific benchmark sub-category. This is just one model. I think there are 48 benchmarks here. A human couldn't research this manually. Deep research helped me make huge progress.

— Did you do that in ChatGPT? I used it there, as far as I know. Did you use deep research for both Grok and ChatGPT?

— Exactly. I also used Gemini's deep research. It was terrible. I gave up on it.

— Gemini is honestly terrible.

— Exactly. It’s really bad.

— They’d be better off without that research module.

— Give me 30 minutes and I’ll crush it. It’s so slow, too. It hangs constantly. You have to keep refreshing. It’s wild that Google has all this data but built such a weak agent.

— They’ll get there. Their time will come. I actually trust Google now. I really do. Opening up AI Studio and giving people access was a brilliant move. Yeah, I think we’ve saved enough time. Let’s look at these prompts.

— Let's see if it works.

— Let's check. The preview didn't even load.

— It didn't show up.

— It hit a limit due to message size. Naturally. Same issue again. Ah, finally. Okay, this was our soft prompt. Not bad at all. Look, there's Central Park in the middle.

— It's even zooming in.

— Yeah, yeah. Very successful.

— Let's hit the streets. Grab a coffee. Do something.

— Let me take a look.

— Check it out.

— Looking at it now, it’s actually nice. It’s perfectly flush. It looks super clean. I don't know how they did it. The way these skyscrapers cluster in zones is cool. Look, they even added signature tops to those two towers.

— Yeah, exactly.

— That’s a pretty interesting touch.

— Yeah, it did its thing. It pulled it off.

— Exactly. The bridge details are super original. Pulling this off on the first try is impressive.

— It mentions Street View or something similar.

— Right, that’s probably in the perspective preset.

— It’s pulling data and explaining things.

— It’s interesting. Getting this on the first try is impressive. People really liked the detail. They found the model's output high quality. Our thinking model caused an issue again. Let's get that solved too.

— Was the previous one the 3.7 standard?

— We were previewing the 3.7 standard. It had thinking enabled. The other model was super detailed from start to finish. It got a bit heavy. Let's simplify this. Let's try to fit it under 3K or 4K tokens. That token limit might not be the best prompt for the AI. But let's roll with it. Let's try to see the details here anyway. Honestly, the user feedback is super impressive. I haven't built a code project with Sonnet 3.7 yet. Just a few things.

— They built a web-based code assistant, right? Something called Cloudy code?

— Yeah, yeah, yeah. They built something new for the CLI. I don't think it's live yet. Or maybe I haven't poked around enough. Let's check out Claude's latest blog post. That should give us more info. Okay, it's a research preview for Claude Code. It looks like a competitor to Replit and Cursor. But it's CLI-based. Let's check the launch video. That will be clearer. This is interesting. You go to your project folder. You type "Claude." A CLI tool pops up. Then you just type your prompts right in there.

I'm honestly surprised. I wasn't expecting this at all.

— So you can work on your codebase with this. Or just handle all your terminal tasks.

— Yeah, it's an agent. Exactly. You can offload all your terminal work. You're right. My bad.

— It works great with SQL and stuff.

— I use it for my Todo app in Cursor. I gave it a full profile. I shared my strengths, weaknesses, and current life status. Before starting a task, I explain it to the LLM. It checks my list and keeps me on track. It tells me what I should actually focus on. It’s basically my personal coach.

— So, like a task shepherd?

— Exactly. It’s a human shepherd. I don't always listen, but it works great. I keep everything in a task.md file. I have a life.md for my status. I also have an observations.md for its notes on me. It saves its judgments there. I interact with this setup via the Cursor chat. Maybe try this with Cloud Code. That could be pretty impressive. It’s an interesting approach. I was surprised they built a CLI product.

— Yeah, they took a different angle. It’s something else.

— It is surprising. We don't see that often. Everyone usually just copies each other in AI.

— Google jumped in too. They released a code assistant with a high free tier.

— Are you talking about Google's IDX? Is that the one?

— No, this one is brand new.

— I know that one. It started last year but didn't really take off. If Google jumps in, they'll do it right.

— Search for Google Code Assistant. Gemini Code Assistant, specifically.

— I haven't seen that one.

— It's brand new. It dropped maybe two or three days ago. Friday at the latest.

— Hmm.

— It came out last week. It's great. You get 180,000 completions for free. It's a model fine-tuned on Gemini 2.0.

— Nice. So, right now.

— It works in Cursor and stuff.

— Google's foundation models used to be bad. I would've said "meh" before. But they've improved. So.

— I still think they're bad at code. Gemini 2 was weak. Maybe the fine-tuned version is better. I don't know. I haven't seen any benchmarks yet.

— I actually tried Gemini 2 Flash in Cursor. It's super fast. I could iterate really quickly. That's why.

— Get an error, write, get an error, write.

— Exactly. Since we can't use Cursor's agent mode, we're stuck with O3, 4o, and Sonnet. That's been a bit of a pain. People keep highlighting the model's perceived quality. They're raving about the code quality and the output. I wouldn't trust the benchmarks too much. This doesn't seem real. It feels like a total exaggeration. I can't believe this. It's not very realistic.

— It wrote it in 17 minutes.

— Honestly, this feels fake to me.

— Yeah, you're right.

— What is this?

— It’s an 8K auto token. JavaScript doesn’t have a game engine like this. Not like Unity. Maybe they wrote a great prompt. They probably pulled the full codebase.

— Still, it’s 8,000 tokens. Considering the output limit, that’s a lot.

— I don’t think it’s that convincing, honestly.

— It won't amount to anything. Without the background details and scenery, I’d buy it. But this feels fake to me. Still, these models evolving is useful. It pushes products like Polymath forward by leaps and bounds. If you have a product built on these agents, it upgrades automatically. That’s a huge win. I saw similar progress with Bolt. The new Bolt running on 3.7 should produce some unique stuff. Bolt has been crushing it lately. This Dynamic Reasoning update is a first for Bolt. I'm curious to see what they do next.

— Someone mentioned they trained Claude 3.7 really well on UX cases.

— Yeah, Anthropic is becoming the gold standard for coding LLM'leri.

— They have an Apple-like approach to this.

— True. I'm not a huge fan of the Sonnet 3.7 feel. But I'm really into Grok 3 lately. We've done some great work with it. The dialogue and communication feel very natural. It handles Turkish really smoothly. The translations are top-tier. The style feels incredibly human. It's a huge success. I hope the API drops soon. It will be a game changer once it hits code editors. I hope Elon Musk isn't too stingy. I suspect Cursor and Windsurf have special deals with Anthropic. Anthropic is likely subsidizing these LLM'leri to gain market share. They’re probably giving these companies a sweetheart deal.

Otherwise, it’s not sustainable. Even at 20 bucks, it’s barely sustainable.

— It’s the most expensive one right now.

— Yeah.

— Except for the top-tier models like O1 Pro. Even 3.7 is way pricier than the rest.

— The output token cost for Claude is a total ripoff. I checked out the competition, like T3 Chat, and their pricing too.

— It’s 15 bucks, man.

— They were crying about it too. Their daily cloud costs are around 1,000 bucks. It’s not easy. But people are getting their money's worth for code projects. Code is the most valuable text output from an LLM. Financially speaking, it’s still worth the price. For the cloud, that’s my take on 3.7. I want to talk about benchmarks now. Ozan, should I pass it to you, or keep going?

— Let’s dive straight into the benchmarks.

— Awesome. I’ve been analyzing these deep research tools by provider. ChatGPT, Perplexity, Grok, and Gemini all have products. My gut says ChatGPT’s O3 mini-based deep research is still number one. I got way more detailed outputs. I wanted an analysis of every benchmark's subcategories. Take MMLU. It's one of the most popular benchmarks out there. My goal was to see model performance by category. For example, Gemini 2 Flash might match Sonnet in biology. But it clearly outperforms in business or lags in chemistry.

These minor differences really matter in practice. It needed to read papers and understand benchmark details. It had to perform deep research. Good info on this is hard to find. You can't find this without digging through massive PDFs. It's tough to find solid info online. You have to check model cards. That's how you see the inner workings of LLM'lerin. They show more detailed approaches to benchmarks. Did I need to analyze based on those for my use case? I researched every benchmark in the world. I'd easily use OpenAI for this.

The only issue is the 100-message monthly limit on Plus. I burned through that immediately. That was a bummer. OpenAI doesn't even show your remaining limit. Once it's gone, it stops doing deep research. That feature felt a bit weird to me. I’d easily put Grok in second place. Grok’s deep research was pretty solid. OpenAI started rolling out deep research to Pro accounts too. I’m sure the Grok effect played a part there. They offer a totally free product. Most people don't know this about ChatGPT. They’ve integrated deep research with operators. It can open a page directly in a browser. It pulls info right from there.

That’s a huge plus. It navigates pages and takes action. Say you want baby carrots from an e-commerce site. It can go to the vegetable category. It can even apply filters for you. You can use it for deep analysis. Maybe you want to compare prices. It’s not just a classic Bing search. It doesn't just scrape the top results. That’s a major advantage. I built something cool with this over the weekend. It’ll be super useful for me. I want to create content based on benchmarks soon. I plan to use it for that.

— You’ll be writing some stuff there too.

— Yeah. I've seen benchmarks around for a while. I hadn't dove in deep yet. I found some downtime to look into them. I grouped them into categories. There's coding and math. Some benchmarks focus on non-English content. That's huge for non-English speaking countries. Style benchmarks really caught my eye. They usually focus on specific traits. LLM outputs can be too consistent. People criticize ChatGPT for this. It feels robotic. It's too structured and stiff. That's where these style benchmarks come in. Take Clan Bench, for example. It tests how conversational an LLM is. It measures how human the dialogue feels. I also found this interesting EQ bench recently.

If you're choosing LLM'leri for specific use cases, check it out. EQ bench ranks them by emotional intelligence. Sorry about that. There are also benchmarks for creative writing. Can it hit the target length? It scores vocab and other criteria. I didn't know these niche benchmarks existed. I really love that level of detail. I can group them this way. There are standards. There are benchmarks for hallucinations and accuracy. Many of these are created by companies. OpenAI, for example, has many benchmark techniques. Simple QA is one of them. It shows how accurate they are. I really liked that. I thought we could do this at Zeo too. Industry-specific benchmarks. A company called Vals AI finds the best LLM'leri for corporate finance.

They find the best LLM'leri for corporate finance. They weighted specific topics. They combined the outputs based on accuracy, latency, and cost. They shared the results. I worked on these a bit. I really want to create these types of benchmarks. It was very useful for me. I'll share it with you, Ozan. Maybe we can look into creating sector-specific benchmarks.

— When you digest all of this. How do you see the current status of the models?

— Good question. I wish there were more of these. There are cases where you can say this LLM is perfect for this problem set. It happens. Especially in long context benchmarks. For example, there is a benchmark type called Needle in a Haystack. It's a benchmark for finding details in long texts. It's basically the needle in a haystack test.

— Exactly. Exactly. Exactly.

— There's also the bug in the code stack test. They're pretty similar. They hide a bug in a massive codebase. It tests if LLM's can spot the bug. It measures their attention span as the code grows. Some LLM's really stand out here. Needle in a Haystack has a few categories. In my experience, Google's models are incredibly successful. For deep research, O3 and O1 are the go-to benchmarks. DeepSeek is also leading in many benchmarks. Surprisingly, DeepSeek performs great in creative writing too.

That actually caught me off guard. Otherwise, Sonnet is the winner in almost every field. That's what stood out in specific vertical benchmarks. I couldn't find much else. Companies usually just cherry-pick benchmarks when they release models. They share scores like GPQA. But nobody benchmarks these companies. I’m actually surprised by that. They just test themselves and publish the results. No independent body is running these tests again. That surprised me. There’s a clear need for that. One benchmark caught my eye. It’s called GPQA. It uses tough test questions. You can’t just Google the answers.

You’ve probably never heard of a model called IF Pro. It’s a company building AI for students. They focus strictly on homework and test prep. I’m not sure if their intentions are pure.

— A friend to academia. Nice.

— Exactly. They probably did some crazy fine-tuning for multiple-choice questions. It actually outperformed O1. I honestly love that. It proves problem-specific fine-tuning still has real value. It shows what’s possible. Assuming these results aren't faked, we need to check their methods. Still, it’s great to see a niche model succeed like this. That’s my take on benchmarks for now. I’ll digest this more next week. Maybe I’ll have more concrete insights then.

— I did a similar study last week. Well, maybe 1% of this scale. I was looking for the best LLM for SQL. I wanted to see if any models really dominated that space. I couldn't find a clear winner for SQL. There are a few open-source options. But the SOTA models outperform them all. It really just came down to Claude and ChatGPT. OpenAI, exactly.

— One thing might surprise you, Ozan. Models like DeepSeek Coder or Code Llama. The inference is tricky, though. Where would you even run those?

— I’ll use an API for the integration. I’m building an agent for a complex database. I checked out those coding models. But honestly, o3 mini just crushes them. It doesn't make much sense to go back there. Updating the other side is easy. You never know when the code will get an update.

— Exactly. If I were dealing with your problem. I’d write four or five tough SQL queries myself. I’d use few-shot prompting. Just feed it user-assistant pairs. Every model opens up after that. The only issue is this. You might have felt it too. Reasoning models sometimes ignore few-shot examples from chat history. When you use O3 mini or O1. The AI can lose track of previous user-assistant messages. The only AI that handles this well is Gemini 2 Thinking. I have a prompt for translating books. I can show you that part. I feed the AI a book.

I want it translated into Turkish for an audiobook format. I need it to sound natural. I want that specific, natural tone. I saw this book recently. I have to recommend this. It’s the best app I’ve used in years. It’s 11 Reader. You get all 11 Labs voices for free. You can’t download it on iPhone yet.

— Does it have any limits? Or are they capping it?

— It's totally free. I’ve easily listened to 7-8 hours of audiobooks. I create them with AI and listen here. I use AI to translate books I like. I copy-paste into 11 Reader. I listen while I'm at the gym.

— Want to sell your books?

— Let me tell you something. I love that idea. I thought about it this morning.

— I sent you the link. 11 Labs just announced it.

— Haha. I saw that on 11 Labs. Can we really sell it?

— Exactly. They just launched their own marketplace.

— That’s huge.

— I think you can even publish to Spotify.

— That’s incredible. I’m already translating other people's books. I could easily adapt those. It shouldn't be too hard. It’s just a matter of prompting. Ozan, this is genius. Let me tell you something. This is a dream job for me. I have so much fun doing this.

— Yeah. That’s cool. It could definitely be a side project.

— You could scale this really well. I worked on it today. I nailed a beast of a prompt. It works perfectly in Turkish. You could build something huge for the Turkish market. We should look into it. If copyright isn't an issue, I could translate every popular business book into readable Turkish. That would be fun. That’s a great idea.

— You're right about the SQL side. It’s complex. Few-shot prompting won't solve it. The data is massive and multidimensional. Some tables have like 77 columns. Explaining that is tough. You have to decide what to include. The system relies on complex joins. Sometimes you need multiple joins or virtual tables to get the right answer. I’m thinking of using reasoning models for the queries. Maybe generate multiple queries first. Then I’ll handle the execution control. I’ll just return the API error or the final result.

I'll build a proper table.

— That makes total sense. It sees the error. It’s better if it’s constructive.

— Something like that could work. Let me mention these multimodel tools. No need to share my screen. Luma AI just dropped a video-to-audio feature. Now, when you generate a video, it creates the matching audio too. If someone plays piano, it adds the piano track. If you have waves hitting the shore, it adds the sound. It’s pretty cool. VO2 came out a few weeks ago.

— True. You’re more into video AI than I am. I ignore most other AI tools, unfortunately.

— It’s such a fun field. A friend pushed me to get into it. It’s really fun. Generating stuff is a blast. It tries to create something new every single time. Doesn't that get boring? Honestly, no. You just have to get how it works. I always thought it was purely creative. Creative work isn't my strong suit. I usually get bored with it. I hate the creative struggle. This removes that entirely. It brings analytical thinking into the mix. When you write a prompt, you start to get it. You realize, "Hey, this won't work in the CLI."

It won't grasp the physics of it. To make it understand, we used a specific method. We were trying to animate something. We needed an animation for another project. Say, a piece of furniture transforming. You just write it out. The weird and cool part is the variety. You get so many different results from one prompt. The world of possibilities is huge.

— Do you get different results by asking the same prompt repeatedly? Or is it different models?

— You can get totally different results. Even with the same prompt, same platform, and same model.

— It’s not ideal. But it boosts creativity and opens up possibilities. If you’re too restricted, you can’t replicate good results. In the worst case, you’re stuck with bad output. You have to learn how to set the right limits. If the same prompt breaks, you need to nudge it toward the previous version. Have you checked out the seed stuff? You provide a seed integer. It’s common in image generation like Midjourney. Midjourney is leading the pack. They produce incredible visuals. You can even train your own Midjourney. A month or so ago, after 1,000 or 10,000 generations, it adapted to your style.

You see the like/dislike buttons. Now you can create boards with specific styles. It’s a whole different world. I’ve been out of the loop, but I found Ideogram. I thought, "Ideogram is great, I’ll stick with this." I still think Ideogram is solid. But for pros, I expected a video model from Midjourney. Small studios are shaking things up. A friend of mine follows them closely and joins their online sessions. The guy running it is a bit arrogant. But they’re only ten people. They earned the right to be cocky. They lead the entire industry with such a small team.

— They have a novel approach to images. It’s legit. They won't enter a space unless they crush it. But they might pivot eventually. Honestly, if they don't do it, who will?

— Exactly. I haven't tracked them in a while. I should check. My focus is LLM and audio generation. That field is fire. Everyone, please install 11 Reader. It's a perfect app. You'll probably be able to buy my e-books there soon.

— Just put them there. I’d rather listen to your book.

— Guess what I did today? I have books I’ve been meaning to finish. I’m reading Venture Deals right now. I grab the English version. I use Gemini 2 Thinking to summarize it chapter by chapter. I get the Turkish translation. Then I feed it to Grok 3. I soften the style a bit. Then I edit my prompt in Think Buddy. It’s like Gemini 2 Thinking wrote it that way from the start. Then I jump to the next chapter. The translations start to click after a while. I stack them up and drop them into 11 Reader. I put my headphones on. I haven't been able to read books lately.

I don't know how much you read. It’s become a huge challenge for all of us. Even though I don't use Instagram or TikTok, my attention span has really dropped.

— It’s partly work-related. It’s about where you choose to spend your time.

— Exactly. I put my headphones on today while riding my scooter. I rode for two hours. It’s super engaging. I don't get bored because I'm moving. I realized this is how I’ll read books from now on. I had started making my own ebooks when you sent me that link. Soon, I’ll be the top Turkish book provider for 11 Labs.

— The first creator partner from Turkey.

— Exactly. Keep the source books on the down-low. We’ll tweak the content before posting. That’s great. It was a fun week. Everything moved so fast. We’ll keep tracking those Sonnet updates until next week. We’ll bring some cool use cases. I think we should do this every week. Sharing inspiring examples motivates people to build. It shows everyone what’s actually possible. I think we’ll come back with more of those. Let’s check the questions and wrap things up.

— Let’s take a look.

— Right. Ufun mentioned that 3D program. He probably pasted code from an AI. That’s what it looked like to me. But would it even fit in 8K tokens?

— No. He’s probably talking about that other thing. He didn't expect us to provide that 3D model. We were more focused on that earlier.

— Remember that guy claiming he did it in GTA?

— Not him. I thought he was relevant. I'll check him out. Send me that Twitter link on Discord. The market is wild right now. Everyone claims they can do everything. People don't even question if you actually did it anymore. It might just be hype. I only tested Sonnet 3.7 with my personal assistant. It's the product I call Yiğit's Shepherd in Cursor. It prioritizes my tasks. Honestly, I didn't feel much of a difference there. I'll definitely do a coding project this week. I'll test 3.7 with SQL. Let's see. If you use five-shot prompting and enable thinking in Ozi, it should be insane. Especially with that agent logic and feedback loops.

Some cool new stuff dropped for that. We'll sync up. I'll tell you about the assistant later. Don't want to hog the show. I think that's all the questions. Sorry if my audio was peaking, Murat. It sounded fine to me. I can't even pronounce my S's well anyway. Thanks for the kind words. That's a wrap for this stream. Take care, everyone. See you in the next one. Stay tuned. Take care. Bye for now. See you later.

This transcript was generated using AI speech recognition and verified against subtitle style rules.

Yiğit Konur

Content author

Yiğit Konur

both

Contact us

How we handle your data

Open as a full page