How well do LLMs build Shiny apps? How do we test that? | Karan Gathani | Data Science Lab @PositPBC
How well do LLMs build Shiny apps? How do we test that? | Karan Gathani | Data Science Lab  @PositPBC
Uploaded October 2026 | Updated November 2026, 11 hours ago
Karan Gathani and the Shiny team at Posit benchmark various LLMs across various measures and identify key strategies to optimize their performance. They take building excellent Shiny apps very seriously, whether you're building them line-by-line or with LLMs. LLM hype is everywhere, but what interests Karan most is moving beyond marketing claims to demonstrate how these models can be scientifically improved for specific use cases. That way, we can assess with confidence whether newly released models are genuinely better or worse.

Karan is a Senior Quality Assurance Engineer at Posit, where he works on Shiny. He has worked with tech startups across the San Francisco Bay Area, helping them ship products efficiently without compromising on quality. In his spare time, he takes part in BioBlitzes around the Bay Area and contributes to this natural history blog.

The Data Science Lab is a live weekly call. Register at pos.it/dslab! Discord invites go out each week on lives calls. We'd love to have you! It's is an open, messy space for learning and asking questions. Think of it like pair coding with a friend or two. Learn something new, and share what you know to help others grow.

Hosting crew from Posit: Libby Heeren, Isabella Velasquez

Karan Gathani's GitHub: github.com/karangattu

Resources:
Download Positron (IDE that Karan uses): positron.posit.co/download.html
Vision language models are blind: openaccess.thecvf.com/content/ACCV2024/papers/Rahmanzadehgervi_Vision_language_models_are_blind_ACCV_2024_paper.pdf
Open Code: opencode.ai
The vitals package! Writing evals for your LLM product: vitals.tidyverse.org/articles/writing-evals.html
Inspect (for Python): inspect.aisi.org.uk
Karan's blog post on Inspect: posit.co/blog/using-evals-in-python
Built-in Skill for Shiny in Posit Assistant (using bslib!): github.com/posit-dev/skills/blob/main/shiny/shiny-bslib/SKILL.md
Agent Skills overview: agentskills.io/home
Shinygen: karangattu.github.io/shinygen
How well do LLMs build Shiny apps? How do we test that? | Karan Gathani | Data Science LabKelly Bodwin: Quarto hacks, AI in the classroom, and why R should stay weirdHow can you add AI voice agents to a mobile Shiny app? | Lucas Da Silva | DS LabThe Answer Was Never Us — with Leilani BattleMind-Blowing Quarto Slide Extensions 💬 🖍️ 🤯 | Emil Hvitfeldt | Data Science LabUsing R package structure for data science projects | Kylie Ainslie | Data Science LabThe skydiver to data scientist pipeline | Kevin Dalton | Data Science HangoutClean up your Shiny UI with popovers and tooltipsOperationalizing Agentic AI: Governance, Observability, and Human Control at ScaleTesting R code outside of WorkbenchYour VP Is Doing a Rogue Analysis in Cursor Right Now — with Nell ThomasThe dessert-first approach to teaching data science | Mine Cetinkaya-Rundel | Data Science Hangout
Posit PBC |

How well do LLMs build Shiny apps? How do we test that? | Karan Gathani | Data Science Lab

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER