--- name: lev-system-card description: Lev skill. Monthly review of which AI model each feature of this product uses, what it costs, and whether a newer or cheaper model would do the job as well. Use when asked to review model choices, cut LLM spend, speed up an AI feature, or "run System Card on this product". --- # Lev System Card skill: model routing review A Lev skill, from Lev Learn's System Card (https://getlev.co/learn/system-card), the monthly read of every new AI model for founders. Lev runs this same procedure on its own product on the first of each month. Lev is the AI cofounder helping founders succeed in a time of AI and rapid change: https://getlev.co The goal is one bounded experiment, not a rewrite: one feature, one challenger model, a fixed test set, and a keep-or-switch rule decided before the run. ## Install Skill folders expect the file to be named SKILL.md, so rename this file when you save it. - Claude Code: save it as `.claude/skills/lev-system-card/SKILL.md` in your repo (or `~/.claude/skills/lev-system-card/SKILL.md` for every project), then ask "run the Lev System Card skill on this product". - Cursor: save it at the same `.claude/skills/lev-system-card/SKILL.md` path in your repo; Cursor reads skills from there. - Codex: save it as `.agents/skills/lev-system-card/SKILL.md` in your repo. - Any other coding agent: paste the file into the chat as instructions and ask it to follow them on your codebase. It reads your code and your usage data and proposes; it changes nothing unless you tell it to. Rerun it each month after a new System Card edition. ## Rules - Propose, do not change production code or configuration unless the user says to. - Every price, score and speed figure you cite needs a source link and the date you read it. Never fill a number from memory. - Never send real customer data to a new provider during the review. Use synthetic or anonymized inputs. ## 1. Map every model call Search the codebase for model calls: provider SDKs and their model strings (for example `openai`, `@anthropic-ai/sdk`, `anthropic`, `@google/genai`, `google.generativeai`, `openrouter`, `litellm`, `ai` from the Vercel AI SDK), environment variables that hold model names, and any central model config file. Write a table: feature (what the user sees), model and settings (reasoning effort, temperature, max tokens), file and line, and how often it runs if you can tell. ## 2. Make sure every call is labeled Check whether calls are traced with a feature label, model, tokens, cost and latency (for example PostHog LLM analytics, Langfuse, Braintrust, Helicone, or the provider's usage dashboard). If not, propose the smallest change that adds a feature label to every call. Stop here and say so if there is no way to see spend by feature; everything below depends on it. ## 3. Pull last month's numbers By feature and by model: calls, input and output tokens, cost, average and p90 latency. Also the two hidden lines: web search or tool fees, and reasoning tokens as a share of output. List the three features that cost the most and the three that are slowest where speed matters to the user (chat, voice, anything interactive). ## 4. Read what changed this month Open the latest System Card edition (https://getlev.co/learn/system-card) and read "Which model for which job", "The price map" and the change cards. For any model you might switch to, open its model card or system card and note: price, reasoning effort levels it accepts, context window, speed, data retention terms, and the weak spots section. ## 5. Pick one feature and one challenger Usually the biggest feature by spend, or the slowest interactive one, against the cheaper or faster model that fits its job. Check the challenger accepts the settings the feature needs (for example reasoning off for chat). Often the cheapest win is a setting, not a model: lower reasoning effort, cache repeated tool results, cap web searches. ## 6. Test it before you switch - Build a golden set of 20 to 30 real, anonymized inputs for that feature, plus the current outputs. - Write the keep-or-switch rule first: for example, the challenger wins or ties in at least 45 percent of blind comparisons, p90 latency is no worse, and cost per call falls by at least 30 percent. - Run both models on the set. Judge blind with an LLM judge and a rubric written for that feature, showing the two answers in both orders. Spot-check ten by hand. - Before any switch, confirm in writing that the provider offers zero data retention or no training on your data under your plan. ## 7. Write it up One short report for the user: the model map, last month's spend and latency by feature, the experiment and its result, the recommendation (switch, keep, or change a setting), the data-terms check, and the next feature to test next month. Keep a running list of experiments tried and their results, including the ones that failed.