The event kicked off with Ariel Juodziukynas’ talk “Warnings en Ruby: NO LOS IGNORES!!!”. Ariel focused on an easy part of everyday Ruby work to ignore: warnings. His talk was a reminder that the yellow text in our terminal often points to real problems. He showed practical examples of common warnings and how to fix them.

After Ariel’s talk, we took a break to catch up, meet new people, and keep the conversations going over drinks and empanadas.

The lightning talks took the night in a looser direction:

• Simon shared how he fixed a tricky performance issue with a fulltext search, walking us through his debugging process and the solution he found.
• Viktor presented aj, a tool he built to improve his AI workflow, showing how Ruby developers can build utilities to streamline their daily work.
• Santiago taught us “how to do juggling”. Yes, actual juggling with balls, not code.
• Ariel came back for a second round with “how to tie your shoes”. It was even better watching everyone try Ariel’s method on the spot.

Thanks to the organizers, the sponsors (SINAPTIA, Rootstrap, Ombulabs, and Eryx), and everyone who showed up. We’re already looking forward to April’s online meetup.

LOL, no. We are not the next tier of Luddites! We started using AI-assisted code generation a little over a year ago, and from the looks of it, we are going to use it even more in the near future. AI is here to stay, and programming will never be the same, what a time to be alive and so on, and so on…

But also, in a couple of years, we’ll all lose our jobs. What a time to be alive, indeed.

We continually ask ourselves: what’s the AI capable of? Can it really replace us? Can it change the way we work? Will it make our problems simpler? Or there will be another kind of problems, more problems? Are we attending the death of software engineering?

There’s only one way to find out.

Initially, we were somewhat skeptical about coding agents. We tried them, but the autocomplete feature was usually annoying, and the chat mode wasn’t smart enough to understand the context of the code. But then, at some point, Claude Code and Cursor became smart enough to be of massive help while building features: they could explain things easily in the context they were in, and they could edit text nicely.

Since then LLMs and coding agents became a key part of our daily routine and we started building more and more features using AI, and growing tools for the RubyLLM ecosystem (such as RubyLLM::Instrumentation, RubyLLM::Monitoring, and RubyLLM::Evals) that we use in our projects and we open-sourced for the larger community in hope to make Ruby one of the top languages for building with AI.

The problem with AI-generated code

I bet at some point you stumbled upon a piece of code that achieved something, but it was hard to understand, poorly thought out, or just ugly (eg, for us, Rubyists, seeing Ruby code that doesn’t feel Ruby). That happened a lot in the StackOverflow era.

With AI-generated code, it’s the same as with any code you didn’t write yourself. You might find AI-generated code hard to understand, poorly thought out, or even solving problems that no one asked to solve.

We’ve seen coding agents working without oversight and proper feedback going down paths that ended up in code that no one would be able to manage. Neither the coding agents nor the humans. Discard and regenerate is a possibility, yes, but tokens are not actually free, and budgeting and financials are something coding agents cannot fix either.

Models and agents will become better over time, and the barrier might be farther away each time, but the problem will always be there.

The hard parts

After more than a year of using coding agents for our daily work, developing AI-powered features, and running several experiments, we feel that AI is truly a multiplier force, but it isn’t changing the most fundamental bits of programming and software production: the hard parts of software production are still hard.

The power of solving a complex problem with a simple and elegant solution/architecture is what makes a good engineer a great one. And AI can help one do so, but is not able to come up with such architectural decisions on its own (at least not yet).

But in the areas where the AI excels, we should try to leverage it. I, honestly, cannot care less if you wrote the code we just deployed to production by punching holes in a card, taping keys in a keyboard letter by letter, copying-pasting snippets from StackOverflow, or you used a high-rate probabilistic word predictor that can produce hundreds of words per second. Tools are not at trial here. But, regardless of how that code came to life, I do care you don’t fall prey to the laziness of not curating the code you produce, of validating it and understanding it (There are many ways to do this, and they are changing, but isn’t this what engineering is about?)

Striving for quality and good architectural decisions is still central. Simplicity is still the only way software remains workable in the long run. For humans, yes, but for AI agents too.

Simple was hard in the pre-AI era and is still hard today (perhaps even harder), but what’s simpler for humans is also simpler for agents. And it is still worth all the effort.

So we say: There is no such thing as AI code. We just have code: good, bad, simple, or complex. And we have tools and processes to deal with it. Like we always did.

So, long live power tools. Long live software engineering.

At SINAPTIA, we specialize in helping businesses implement AI solutions that deliver real value. If you’re facing challenges with prompt engineering or AI integration, we’d love to help.

# db/migrate/XXXXXXXXXXXXXX_create_reports.rb
class CreateReports < ActiveRecord::Migration[7.1]
  create_table :reports do |t|
    t.string :title
    t.integer :reason, default: 0, null: false
  end
end

# app/models/report.rb
class Report < ApplicationRecord
  enum :reason, {
    spam: 0,
    harassment: 1,
    inappropriate_content: 2,
    copyright: 3,
    misinformation: 4
  }, prefix: true
end

Everything worked fine until the client requested the ability to select multiple reasons for a report. A user could report content for being spam and containing misinformation. The set of possible reasons is fixed and defined by developers.

We tried four approaches.

Four ways to solve it

• Bitwise Operations: store multiple values in a single integer using bit-level flags.
• PostgreSQL Array: use native PostgreSQL array columns with enum-like syntax.
• JSONB: store reasons as a JSON array inside a JSONB column.
• HABTM: the classic many-to-many approach with a join table.

1. Bitwise Operations

They use bit-level operations to store multiple values in a single integer. Each reason occupies a specific bit. The active_flag gem provides a clean DSL for this:

# db/migrate/XXXXXXXXXXXXXX_add_reasons_to_reports.rb
class AddReasonsToReports < ActiveRecord::Migration[7.1]
  def change
    add_column :reports, :reasons, :bigint, default: 0, null: false

    # migrate data to new format

    remove_column :reports, :reason
  end
end

# app/models/report.rb
class Report < ApplicationRecord
  flag :reasons, [:spam, :harassment, :inappropriate_content, :copyright, :misinformation]
end

# Usage
report = Report.new(reasons: [:spam, :misinformation])
report.reasons # => [:spam, :misinformation]
report.reasons.spam?   # => true
report.reasons = [:spam]
report.save!

# Read: check if spam is included
report.reasons.spam? # => true

# Validation: invalid values raise ArgumentError
report.reasons = [:invalid] # => raises ArgumentError

# Query: find all reports with spam AND misinformation
Report.where_reasons(:spam, :misinformation)

The benefits here are clear: extremely compact storage, very fast boolean checks, and no GIN index required, just a standard integer index. But the trade-offs become apparent quickly. Database values are inexpressive (what does value 4 mean?). You’re limited to 64 values with bigint. And query operations are less intuitive than standard ActiveRecord.

Note: Most Ruby gems (including active_flag) are based on integers with a 64-bit limit. While PostgreSQL supports the bit string data type, which can store many more flags without this limitation, the Ruby ecosystem doesn’t have widely adopted gems for this approach.

2. PostgreSQL Array Field (Multivalued Column)

PostgreSQL has native array support. We can store reason strings directly in a string array column without any gem:

# db/migrate/XXXXXXXXXXXXXX_add_reasons_to_reports.rb
class AddReasonsToReports < ActiveRecord::Migration[7.1]
  def change
    add_column :reports, :reasons, :string, array: true, default: []
    add_index :reports, :reasons, using: 'gin'

    # migrate data to new format

    remove_column :reports, :reason
  end
end

# app/models/report.rb
class Report < ApplicationRecord
  VALID_REASONS = %w[spam harassment inappropriate_content copyright misinformation].freeze

  validate :reasons_must_be_valid

  private

  def reasons_must_be_valid
    return if reasons.blank? || reasons.all? { |r| VALID_REASONS.include?(r) }

    errors.add(:reasons, "contain invalid values")
  end
end

# Usage
report = Report.create(title: "Report 1", reasons: ["spam", "misinformation"])

# Read: check if spam is included
report.reasons.include?("spam") # => true

# Query: find all reports with spam AND misinformation
Report.where("reasons @> ARRAY[?]::varchar[]", ["spam", "misinformation"])

The upside: no gem dependency, human-readable values in the database, and easy to expand. The downside: it’s PostgreSQL-specific, validations require manual implementation, and it’s less familiar to developers who don’t use PostgreSQL regularly.

3. JSONB (Multivalued Column)

We store reasons as a JSON array inside a JSONB column.

# db/migrate/XXXXXXXXXXXXXX_add_reasons_to_reports.rb
class AddReasonsToReports < ActiveRecord::Migration[7.1]
  def change
    add_column :reports, :reasons, :jsonb, default: [], null: false
    add_index :reports, :reasons, using: 'gin'

    # migrate data to new format

    remove_column :reports, :reason
  end
end

# app/models/report.rb
class Report < ApplicationRecord
  VALID_REASONS = %w[spam harassment inappropriate_content copyright misinformation].freeze

  validate :reasons_must_be_valid

  private

  def reasons_must_be_valid
    return if reasons.blank? || reasons.all? { |r| VALID_REASONS.include?(r) }

    errors.add(:reasons, "contain invalid values")
  end
end

# Usage
report = Report.create(title: "Report 1", reasons: ["spam", "misinformation"])

# Read: check if spam is included
report.reasons.include?("spam") # => true

# Query: find all reports with spam AND misinformation
Report.where("reasons @> ?", '["spam", "misinformation"]')

The benefit is GIN indexing for fast searches. The cost: the flexible structure requires stronger validations to enforce the expected format, it’s slightly more verbose than alternatives, and there’s JSON parsing overhead in some cases.

4. HABTM Table

The classic many-to-many approach with a join table.

# db/migrate/XXXXXXXXXXXXXX_create_reasons_and_join_table.rb
class CreateReasonsAndJoinTable < ActiveRecord::Migration[7.1]
  def change
    create_table :reasons do |t|
      t.string :name, null: false
      t.timestamps
    end
    add_index :reasons, :name, unique: true

    create_join_table :reports, :reasons do |t|
      t.index [:report_id, :reason_id]
      t.index [:reason_id, :report_id]
    end

    # migrate data to new format

    remove_column :reports, :reason
  end
end

# app/models/report.rb
class Report < ApplicationRecord
  has_and_belongs_to_many :reasons
end

# app/models/reason.rb
class Reason < ApplicationRecord
  has_and_belongs_to_many :reports

  validates :name, inclusion: { in: %w[spam harassment inappropriate_content copyright  misinformation] }
end

# Usage
report = Report.create(title: "Report 1", reasons: Reason.where(name: ["spam", "misinformation"]))

# Read: check if spam is included
report.reasons.exists?(name: "spam") # => true

# Query: find all reports with spam AND misinformation
Report.joins(:reasons)
      .where(reasons: { name: ["spam", "misinformation"] })
      .group(:id)
      .having("COUNT(DISTINCT reasons.id) = 2")

Benefits include total flexibility to add additional metadata, extreme familiarity for Rails developers, and unlimited scalability. The drawback: slower writes (create/update) and the need for more queries or joins to read data.

Comparison

Here’s a summary of how each approach compares across the attributes that matter most:

Property vs Entity

In data modeling terms, reasons are a property of a report, not an entity with its own identity and lifecycle. Nobody queries “show me all attributes of the spam reason”. That makes the multivalued column approaches (array, JSONB, bitwise) more true to the conceptual model, even though they break First Normal Form. The HABTM is the more purely relational approach, but it treats a property more as if it were an entity.

Performance

We ran benchmarks with 1000 operations for create, find, and update scenarios. You can find the full benchmark code and results in the multivalued_attributes repository.

The HABTM is ~4x slower on creates and ~3-5x slower on updates compared to the other methods. However, it’s worth noting that most performance issues related to HABTM and JOINs aren’t actually caused by the JOIN itself. Databases like PostgreSQL are highly optimized for these operations. The real culprits are usually N+1 queries, missing indexes, or poorly designed queries. With proper indexing and eager loading, the read performance gap narrows significantly. The first three approaches (bitwise, array, and jsonb) have very similar performance. On reads, the difference is marginal (only 2-3 extra seconds over 1000 queries).

Query Simplicity

One often overlooked aspect is query complexity at the call site. If your codebase filters by these values extensively, in scopes, serializers, admin interfaces, and column-based approaches, keep queries simple and on a single table. With array_enum or bitwise, you can do:

Report.with_reason(:spam)
Report.where.not(reasons: [])

With a HABTM, every call site would need a .joins(:reasons), adding complexity and increasing the risk of N+1 queries if eager loading is forgotten. While you can mitigate this with default scopes or associations, it adds friction that column-based approaches simply don’t have.

Default Values

With array or JSONB columns, you can set default values directly in the migration:

t.integer :reasons, array: true, default: ["spam", "harassment"]

New records automatically get these values. No callbacks required. With a HABTM, you’d need a callback to seed join table rows, adding an extra step that can be forgotten or misconfigured.

Database-Level Validation

With a HABTM, foreign keys enforce valid references automatically:

INSERT INTO reasons_reports (report_id, reason_id)
VALUES ('some-uuid', 999);
-- ERROR: Key (reason_id)=(999) is not present in table "reasons"

You also get cascading behavior. ON DELETE CASCADE automatically cleans up join table rows when a reason is removed.

With array or JSONB, you can use PostgreSQL CHECK constraints to enforce valid values at the database level:

ALTER TABLE reports
ADD CONSTRAINT valid_reasons
CHECK (reasons <@ ARRAY['spam', 'harassment', 'inappropriate_content', 'copyright', 'misinformation']);

UPDATE reports
SET reasons = ARRAY['spam', 'harassment', 'oops_typo'];
-- ERROR: new row for relation "reports" violates check constraint "valid_reasons"

Both approaches can enforce valid values at the database level. The difference is ergonomics: foreign keys handle this naturally, while CHECK constraints require explicit maintenance when adding new values.

Extensibility

• Bitwise: Limited to 64 values. To add more, you need to change to bigint or use bit strings.
• Array: Easy to expand, but storing many values (dozens+) becomes unwieldy. Row size grows, and GIN indexes start having performance issues.
• JSONB: Flexible, but the same problem as arrays when storing many values.
• HABTM: The most extensible. You can add additional columns (e.g., reason_details, reported_at in the join table). Also ideal for dynamic or user-defined values.

Familiarity

• Bitwise: Strange for most. Requires explaining bit-level operations.
• Array: Intuitive if you know Ruby/PostgreSQL.
• JSONB: Familiar to those who use modern REST APIs.
• HABTM: The most idiomatic. Any Rails developer immediately understands has_and_belongs_to_many.

Ecosystem

• Bitwise/Array/JSONB: Require custom form inputs, serializers, and manual validations. Some support from gems.
• HABTM: Works out-of-the-box with ActiveAdmin, RailsAdmin, nested forms, and bulk operations.

Database Compatibility

PostgreSQL arrays are PostgreSQL-specific. Bitwise and HABTM work across any database. JSON works across MySQL, SQLite, and PostgreSQL (though with different syntax and index types. MySQL uses GIN indexes similarly to PostgreSQL, while SQLite has the JSON1 extension).

Conclusion

If your value set is static and small, PostgreSQL array is the better choice over bitwise. They offer nearly identical performance without introducing unfamiliar bit-level operations that confuse most Rails developers.

But don’t dismiss HABTM. For most applications, the write overhead is irrelevant compared to the cost of maintaining “clever” code that the next developer doesn’t understand.

In our specific case, a reporting system where users can select multiple reasons, we chose HABTM. Not because it was the fastest, it wasn’t. But because it didn’t need justification. Any new developer on the team immediately understands it, and that clarity is worth more than the performance gains we’d rarely notice in practice.

The best technical choice is the one that doesn’t need a justification.

Every change on a prompt opens up a tree of choices. Which provider should we use? Which model? How detailed should the instructions be? Would more samples in the prompt work better? How much context per message? Should we use a reasoning model? Or augment the data available to the model with multi-modal input? There’s also the cost vs. accuracy tradeoff: is 10x the price worth a 5% improvement for this specific feature?

The combinatorial explosion gets overwhelming fast, and the result of the process has this feeling of uncertainty… is there a branch I missed that works better? Or that costs less?

The pragmatic choice: spreadsheets

We needed a methodology to track changes across iterations so the team can follow along. Naturally, we took a pragmatic stance: we started using spreadsheets for each feature tracking results across prompt/provider/model configurations, all run against the same data. It worked quite well, and over several features, we started seeing a workflow emerge, but…

Spreadsheets don’t scale

We knew the limits going in, but they became harder to ignore over time:

• They fragment. People make copies. When you’re sharing with non-technical collaborators, you end up with multiple sources of truth.
• No enforced structure. Each feature ended up with its own format. You have to re-learn how to read each one, and not all of them track the same metrics the same way.
• Hard to compare. Eyeballing results across configurations isn’t intuitive, and people get confused.
• No regression baseline. Once you settle on a configuration, how do you catch regressions later?
• Prompts drift. Someone edits the spreadsheet and forgets to update the code. Nobody notices until something breaks.
• Disconnected from code. Prompts and evaluations should live where the application lives.

In one project with many AI features, this all came apart. Links got lost, copies multiplied across different people’s drives with small divergences. Building eval datasets meant downloading images and re-uploading them to sheets. Running prompts required manual dev work because the data lived in Google Drive, but prompts had to go through the LLM provider. We built some internal tooling to help, but since every sheet and feature had a different format, nothing was reusable.

But they were useful to uncover what we needed: a place where you can couple a prompt configuration with a curated dataset extracted from real data, that helps you find the right balance between accuracy and costs for the feature at hand. Ideally, without leaving the Rails app.

So we built RubyLLM::Evals, a Rails engine for testing, comparing, and improving LLM prompts directly inside your application.

RubyLLM::Evals

Since we’re using RubyLLM, it made sense to build on top of it.

The core abstractions are prompts and samples. A prompt captures a full configuration: provider, model, system instructions, message template (with Liquid variables), tools, and output schemas. If you already have tools or schemas in your app, you can reuse them. Samples are your test cases: each one defines an evaluation type (exact match, contains, regex, LLM judge, or human judge) and an expected output.

The interesting design choice was making the LLM-as-judge a first-class eval type. For features like summarization or classification with fuzzy boundaries, exact matching doesn’t cut it. You need another model to assess whether the response is good enough. It’s not perfect, the judge has its own biases and failure modes, but for iterative prompt development, it’s a pragmatic tradeoff: fast feedback now, human review on the edge cases.

Each run saves a snapshot of the prompt settings and records accuracy, cost, and duration. A comparison tool lays all runs of a prompt side by side, so you can spot what changed and why.

Real application data

One thing we really wanted was the ability to populate samples from the application’s data. For example, in our image categorization feature, we can:

prompt = RubyLLM::Evals::Prompt.find_by(slug: "image-categorization")

Image.uncategorized.limit(50).each do |image|
  sample = prompt.samples.create(eval_type: :human)
  sample.files.attach(image.attachment.blob)
end

Now you’re iterating on your prompt with actual production data, not synthetic examples.

The temptation is to throw hundreds of samples at a prompt and see what sticks. In practice, a smaller curated set that covers your edge cases tells you more than a large random one. We typically start with 20-30 samples: a mix of straightforward cases, known hard cases from production, and a few adversarial examples. If accuracy looks promising, we expand. If not, the small set is faster to iterate on.

In production

Once you’re happy with a prompt, you can use it directly in your application:

response = RubyLLM::Evals::Prompt.execute(
  "image-categorization",
  files: [image.attachment.blob]
)
response.content  # => "deck"

The configuration lives in the database, versioned through your evaluation runs, always in sync with what you tested. Rolling back to a previous version or A/B testing a new iteration becomes straightforward.

Where this leaves us

Production data has a way of surprising you: new usage patterns, edge cases you never curated a sample for, a provider silently updating a model or its pricing… your prompt’s accuracy can degrade, or your cost can skyrocket without a single line of code changing. This is a challenge that has no single solution, but monitoring a prompt’s performance in production is key. Each feature will require something different and use different metrics, but you need feedback, so when your metrics surface a drift, lower quality results, or higher error cases, higher costs, you can pull new samples into RubyLLM::Evals and adjust the prompt to the new reality.

The pattern we keep seeing across projects is that prompts are never done. Models get updated, data distributions shift, and what worked last month might silently degrade and fail over time. Continuous testing and monitoring are critical.

RubyLLM::Evals and RubyLLM::Monitoring are how we go from concept to production. Both are open source and built for Rails.

At SINAPTIA, we specialize in helping businesses implement AI solutions that deliver real value. If you’re facing challenges with prompt engineering or AI integration, we’d love to help.

“The first version of the sublayer gem was actually a coding agent, but it was coming together so quickly… I was like, wait… if this is so easy for me, it’s going to be easy for everybody, and everybody is going to be making these…”

Last week, we open-sourced a minimal but feature-packed coding agent. We were after the simplest, straightforward, stupidly effective agent possible, so we named it Detritus, after Lance Constable Detritus of the Ankh-Morpork City Watch from Discworld (thanks for so much and so many, Sir Terry).

Detritus is built in just 250 lines of code, yet it packs a CLI with history, custom slash commands and skills (sort of), save/resume chats, subagents, and a two-level configuration (project and global). A full-featured coding agent.

While building this basic agent, we confirmed, firsthand, what Scott had said. And I kept wondering:

Why? What makes it so easy?

Is it the LLMs? Is it Ruby? Is it that it’s fun, so you don’t really feel the pain? Or is it something else?

After giving it some thought and talking about this with teammates, we converged on two key factors:

The first key: general availability of LLMs

General availability of LLMs changed the nature of the problem of building something like Detritus. Before, building a coding AI was unthinkable, but current LLMs made impossible things almost trivial:

Code some utility functions for the LLM to call (one for editing files, one for bash commands), hook up an LLM via API, put it all in a loop, and that’s it. You have a coding agent.

What used to be a research problem is now an integration problem. The problem migrated from the lab to the workshop.

The second key: Ruby’s power

Ruby is well known for its historical focus on developer happiness: “A programmer’s best friend”. I think this is a fundamental characteristic of the language, but sometimes I feel it’s a little superficial, and it doesn’t tell you why.

I think Ruby brings something else that is a much more fundamental property that emerges out of its design and philosophy: Power.

Originally, the idea of “powerful programming languages” came to me via Amir Rajan, creator of DragonRuby, when he shared this article from Paul Graham, “Beating the averages”. We talked about how and why Lisp was the most powerful language, with Ruby being a close second. Graham’s key insight — what he calls the “Blub paradox” — is that power in programming languages sits on a continuum, and you can only recognize a more powerful language from above, never from below.

Any general-purpose programming language is nowadays more or less equivalent, equally capable. You can build Detritus with the exact same features in Python, Go, JavaScript, or even in C. And yet, the experience of building this in Ruby feels fluid and frictionless. Like cutting a wooden block with a hand saw or a circular saw. Both will cut the wood just fine (they are equally capable), and you can probably enjoy both (personal taste is not the matter here), but one will make you feel more powerfully invested than the other.

I think power in programming languages is not just capability, but the relation between using the capability and the effort the developer has to invest in wielding it.

In this sense, Ruby has the ability to maximize the capability/effort ratio. The amount of power condensed in a few lines of code feels extraordinary.

If you take a look at Detritus’ source code, this is how you set up the agent:

def create_chat(instructions: state.instructions, tools: [EditFile, Bash, WebSearch, SubAgent], persist: true)
  chat = RubyLLM::Chat.new(model: state.model, provider: $state.provider)
  chat.with_instructions(instructions) if instructions
  chat.on_end_message { |msg| save_chat } if persist
  chat.with_tools(*tools)
end

Five lines of RubyLLM set the model, system prompt, and tools. That’s all you need to set the agentic loop ready to go.

And the rest of the code is the same: chat persistence is Marshal.dump. The CLI router is a case statement. The subagent is a tool that calls create_chat. None of this code is clever or magical; it’s just plain Ruby. That’s exactly the point. When the language is powerful enough, building an AI agent doesn’t require anything special, just the mundane. And Ruby makes the mundane exquisite, short.

Detritus’ history started when Thorsten Ball published The Emperor Has No Clothes, a guide to building a super basic coding agent in Go. My immediate thought after the head explosion was: if we did this in Ruby, it would take a fraction of the code and give us twice the features. So, as Thorsten suggested, “I went and tried how far I could get”. I got this far.

Raised to the power

LLMs’ general availability turned AI from a “research problem” into an “integration problem”. The nature of the work changed to match Ruby’s strengths: orchestration, expressiveness, and fast iteration.

When you combine Ruby with LLMs, you get compounding power. Power * Power. Power squared.

The key to building an agent is defining what to delegate to the LLM and what to handle in code. For example, Detritus’ skills feature: the code just provides a list of instructions and scripts. The actual skill, knowing when to use each one, how to combine them, that’s all the LLM.

This is where both keys meet. LLMs do the hard part; our job is orchestration. And Ruby makes the orchestration so clean you can see just how little code is actually needed. Compounding power.

The Opportunity

The Ruby AI ecosystem is young, but it’s growing fast. RubyLLM, the gem that powers Detritus, is already spawning its own ecosystem: MCP support, monitoring, agent frameworks, etc. Andrew Kane has quietly built an entire ML infrastructure layer for Ruby: transformers, torch, embeddings, vector search, and ONNX runtime. Officially supported SDKs from OpenAI, Anthropic, and MCP. The foundations are being laid right now, the Ruby way: simple, expressive, and delightful to use.

In the coming years, most of us, Ruby developers, won’t be training models. We will be orchestrating API calls, building agents, capabilities, features, and designing systems. Building products on top of a dynamic, ever-changing landscape. We’ll be doing what Ruby does best: making powerful capabilities accessible through elegant, expressive interfaces. And because of Ruby’s power, we can do those things naturally, frictionlessly, easily.

The same things that made Ruby great for web development 15 years ago are perfectly aligned again, but now with a more mature, faster, and modern Ruby. The potential is huge.

The Ruby community has decades of experience building products and delightful tools. The AI landscape is wide open, the tools are here, and the problem fits like a glove. So… what are we, Rubyists, going to do?

But how did we do it? What mechanism does RubyLLM provide that we can use to capture all LLM requests? Or did we use something else?

RubyLLM event handlers

RubyLLM provides event handlers out of the box. You can use them to capture an event when a message is sent to the LLM and, for example, calculate its cost. This is how you’d use them:

# Provided that you have gemini configured in config/initializers/ruby_llm.rb
chat = RubyLLM.chat provider: "gemini", model: "gemini-2.5-flash"

chat.on_end_message do |message|
  Event.create(
    provider: chat.model.provider,
    model: chat.model.id,
    input_tokens: message&.input_tokens || 0,
    output_tokens: message&.output_tokens || 0
  )
end

response = chat.ask("Write a short poem about Ruby")

In the code above, an event record is created when a message is completed, and the cost is calculated in an ActiveRecord callback. The solution is pretty simple and works perfectly, but it doesn’t scale very well:

• You need to add this manual tracking everywhere. Every chat instance requires this callback to be set up; otherwise you will lose that data. You can simplify it even more, but you’ll always have to set up the callback.
• Your instrumentation code and your business logic are tightly coupled, which makes both harder to maintain.
• This only works for RubyLLM::Chat instances. What about embeddings, image generation, and other operations? You’d need different mechanisms for each.
• Tracking full request metrics like latency needs more complex and intrusive code.

We needed something more comprehensive and automatic that doesn’t rely on us remembering to hook the instrumentation code everywhere. Luckily, Rails has something neat for us to use baked in.

ActiveSupport::Notifications

ActiveSupport::Notifications is Rails’ instrumentation API. It’s what Rails uses internally to track things like database queries, view rendering, controller executions, and more.

Using it is simple: you make your code emit events by calling ActiveSupport::Notifications#instrument(...), and subscribers can consume those events to do logging, monitoring, or whatever else you need. An interesting example is Prosopite, which hooks into sql.active_record events to detect N+1 queries.

This mechanism is especially important for libraries, as it decouples the business logic of the library and the business logic of the application that uses it. In the case of RubyLLM::Monitoring, the monitoring logic lives separately and subscribes to what it cares about. No coupling between RubyLLM and RubyLLM::Monitoring.

So, this is what we did in RubyLLM::Instrumentation to make RubyLLM emit events after each LLM call. RubyLLM::Monitoring, on the other hand, provides an event subscriber that captures the events and feeds them into its dashboard.

RubyLLM::Instrumentation

Instrumentation should be automatic and invisible. RubyLLM::Instrumentation achieves that: just add it to your Gemfile, run bundle install, and you’re done. RubyLLM will start emitting events for you to subscribe to.

Now, following the example above, the code becomes:

# in config/initializers/ruby_llm.rb
ActiveSupport::Notifications.subscribe(/ruby_llm/) do |event|
  # Do whatever you want with the event, in RubyLLM::Monitoring we store the event data in the database for later use
  Event.create(
    provider: event.payload[:provider],
    model: event.payload[:model],
    input_tokens: event.payload[:input_tokens] || 0,
    output_tokens: event.payload[:output_tokens] || 0
  )
end

# Provided that you have gemini configured in config/initializers/ruby_llm.rb
chat = RubyLLM.chat provider: "gemini", model: "gemini-2.5-flash"

# RubyLLM will emit the event, and it'll be captured by the subscriber above
response = chat.ask("Write a short poem about Ruby")

The code remains practically the same as in the original example, but the instrumentation becomes much simpler and decoupled, and there’s no need to repeat the same hook in multiple places.

In the example above, all ruby_llm events are captured, but you can subscribe to specific events. You can read more about the instrumented events and their payload in the project’s repository.

Wrapping up

RubyLLM::Instrumentation takes off the burden of manually instrumenting the code from the users’ shoulders. Originally written as part of RubyLLM::Monitoring, we extracted it into its own gem because we thought it was a fundamental tool, and as we needed it, other people might need it too to build a different monitoring tool, or an analytics tool, or set up logging differently.

Give it a try, send us feedback, and contribute if you want to!

If you’re building AI-powered applications with Rails and need help with architecture, optimization, or observability, get in touch.

Unlike most APIs, LLM APIs calls have variable costs. They are usage-based, so the price depends on input and output tokens consumed. So, how do you know how your users are using it? Or, how much will it cost you monthly? Is it what you estimated, and are the usage limits you designed ok? Are they needed at all?

Why monitoring LLM interactions matters

Beyond basic visibility, monitoring unlocks practical improvements:

• Cost management: Track which models and features are costing you money, then focus optimization efforts where they matter. When 80% of your costs come from one feature, you can try a cheaper model, add caching, optimize prompts, or, if the provider and feature allow it, batch processing.
• Performance tracking and anomaly detection: Monitor response times to identify slow prompts and set realistic expectations. A sudden spike in latency or requests usually means something changed—a bug causing retries, or model performance issues—and monitoring helps you correlate changes with their impact.
• Capacity planning: Understanding your throughput patterns (requests per minute, hour, day) helps you forecast costs and identify features that might benefit from caching or batching.
• Provider comparison: With multiple LLM providers offering similar capabilities at different price points, monitoring helps you make informed decisions about which model delivers the best results for your use case.
• Reporting: Product managers and stakeholders want to know what AI is costing. With monitoring data in your database, generating reports is a SQL query away.
• Model migration planning: When a provider releases a new model or changes pricing, you can estimate the impact on your costs before making the switch.

Introducing RubyLLM::Monitoring

As you might guess, after deploying our AI-powered features, we had several usage spikes that threatened the viability of the features. We needed to monitor our LLM usage in production. At the beginning, we did it manually, using whatever each inference platform provided. But as we started using different providers and models on several features, manually tracking cost and token usage became complicated and error-prone. So we built RubyLLM::Monitoring: a Rails engine that tracks every LLM request your application makes and provides a dashboard where you can see cost, throughput, response time, and error aggregations. On top of it, you can set up alerts so that when something interesting to you happens, you receive an email or a Slack notification.

As the name suggests, it’s built on top of RubyLLM and integrates seamlessly with your existing setup. No separate infrastructure, no external services, just another engine mounted in your Rails app.

How it works

The engine instruments every LLM request your app makes (stay tuned for a related post) and saves it to your database. Cost is calculated automatically using RubyLLM’s built-in pricing data. Since everything lives in your database, you can run custom queries when the dashboard isn’t enough.

The dashboard

Once installed, you get a dashboard at /monitoring (or wherever you mount it) with:

• Summary cards showing total requests, total cost, average response time, and error rate.
• A breakdown table grouping metrics by provider and model, so you can see at a glance which models are being used and what they’re costing you.
• Metrics:
  • Throughput: Request count over time
  • Cost: Accumulated costs per time window
  • Response time: Average latency trends
  • Error rate: Percentage of failed requests

Demo with slow responses and a high error rate.

Alerts

Beyond the dashboard, you can configure custom alert rules to notify you when specific conditions are met. This is essential for catching cost overruns, error spikes, or unusual patterns before they become problems.

Alert rules are flexible and can trigger based on any condition you can express as a query. Here are some practical examples:

# config/initializers/ruby_llm_monitoring.rb
RubyLLM::Monitoring.channels = {
  email: { to: "team@example.com" },
  slack: { webhook_url: ENV["SLACK_WEBHOOK_URL"] }
}

RubyLLM::Monitoring.alert_rules += [{
  time_range: -> { Time.current.at_beginning_of_month.. },
  rule: ->(events) { events.sum(:cost) >= 500 },
  channels: [:email, :slack],
  message: { text: "More than $500 spent this month" }
}, {
  time_range: -> { 1.day.ago.. },
  rule: ->(events) { events.average(:response_time) > 5000 },
  channels: [:slack],
  message: { text: "Average response time exceeded 5 seconds" }
}]

Alert rules have built-in cooldown periods to prevent notification spam, and you can customize channels for each rule. You can even build custom notification channels beyond the built-in email and Slack options.

Conclusion

Building AI-powered features doesn’t end at deployment. The models you depend on are expensive, their performance varies, and usage patterns shift over time. Models and providers are unstable due to the rapidly evolving AI landscape. Without proper visibility, you have only guesses. So, we built RubyLLM::Monitoring.

Give it a try, send us feedback, and contribute if you want to!

At SINAPTIA, we specialize in helping businesses implement AI solutions that deliver real value. If you’re facing challenges with LLM monitoring or AI integration, we’d love to help.

References

The meetup was made possible thanks to the support of RubyCentral, GitHub, SINAPTIA, and Unagi. Many thanks to our sponsors; this wouldn’t have been possible without them.

Observability en la era de AI

The first talk was presented by Patricio Mac Adden from SINAPTIA, who shared his team’s journey combining observability tools with LLMs to tackle real-world Rails application problems. With an upfront disclaimer that this wasn’t “the definitive solution” but rather hard-earned experience from the trenches, Patricio dove into a fascinating story.

He started by providing context on how SINAPTIA uses LLMs daily—from AI agents that help with programming, code reviews, and debugging, to production features like image classification, image upscaling, similarity search, and MCP integration. The goal? Optimize time and maximize value.

The talk then explored the observability challenges they faced: applications with performance problems, memory leaks, slow actions, and scarce hardware and software resources (think free-tier APMs or no APM at all), all while trying to develop new features. Their DIY APM journey took them from ActiveSupport::Notifications and ActiveSupport::ErrorReporter to OpenTelemetry, eventually leading to SolidTelemetry.

But when they tried combining their APM data with LLMs for automated problem-solving, they hit roadblocks: the process was too manual (exporting traces, exceptions, and performance items was tedious), too repetitive (you had to tell the LLM what to do every time), used too much context (OpenTelemetry exports many spans per trace), and ultimately wasn’t LLM-friendly.

The solution? MiniTelemetry. A simpler, more lightweight approach that replaces traces/spans with events, eliminates metrics, and is built on top of Rails’ native ActiveSupport::Notifications and ActiveSupport::ErrorReporter. Most importantly, it’s LLM-friendly by design. The talk concluded with a live demo showing how this approach works in practice.

Elijo tu propia aventura

The second presentation, delivered by Renzo Quaggia from Unagi, took a creative storytelling approach inspired by the classic “Choose Your Own Adventure” books. Rather than an interactive format, Renzo shared his real-world experience working on a checkout page, a critical part of any e-commerce system where every decision can significantly impact conversion rates.

Renzo walked the audience through the decision points he faced during the project, much like the branching paths in those beloved adventure books. The talk explored how these choices ultimately led him to implement A/B testing as a solution, allowing data rather than assumptions to guide which path to take. It was a practical reminder that in software development, we often face multiple valid approaches, and sometimes the best answer is to test them all.

As with any good Ruby meetup, the event concluded with time for networking, sharing experiences, and connecting with other developers over food and drinks.

Thanks again to RubyCentral, GitHub, SINAPTIA, and Unagi for making this event possible, and to everyone who attended. We hope next year we can see more events like this!

The main talk was given by Fernando E. Silva Jacquier, bringing his point of view about “Expressive coding”, the pros and cons of writing code in a more human way, and the importance of understanding what happens under the hood for expressions that sound the same but behave a bit differently.

After a short break to eat empanadas and drink beers, there were some ⚡ lightning talks ⚡, a space to relax and share whatever you want in 5 minutes or less. Among the highlights were our own Nazareno Moresco sharing his journey understanding the Mayan calendar, which led to a fun side project gem (Mayan), and Gemma Falconi talking about turtles, those lovely and tiny dinosaurs.

Huge thanks to sponsors and organizers (SINAPTIA, Rootstrap, Ombulabs, Eagerworks, Moony, Roxom, Crunchloop, Svitla, and LeWagon) and all the people who made each meeting special. We hope to see you again in 2026.

What users actually experience

When a request hits your Rails app and a response goes back, that total time is just a portion of what users experience. Server response time is crucial, but it’s only one piece of perceived performance:

• Network round-trip matters. Your app might respond in 100ms, but if the user is on a slow connection or geographically far from your server, they might wait 500ms for the round-trip. A fast server doesn’t fix slow networks.
• Download and rendering matter. Once the HTML arrives, the browser needs to download CSS, JavaScript, and images. Then it needs to parse, render, and potentially hydrate a JavaScript framework. A 100ms server response followed by 2 seconds of asset downloads and rendering feels slow to users.

The vision on performance should be integral. Server time, network latency, asset delivery, and browser rendering add up to what users experience. In this post, we will focus exclusively on server response time.

Percentiles: the right way to measure

You’ve got a group of similar actions. Some are fast, some are slow. What metric do you use to decide if it’s “slow”?

You shouldn’t use the average. The average lies. Imagine 99 requests at 50ms and 1 request at 5 seconds. Your average is 99.5ms, which looks great! But 1% of your users just waited 5 seconds. That’s not acceptable. Depending on the size of your user base, that 1% can be considered an outlier, but if your user base is large, it means a lot of people are having a bad experience.

Percentiles show you what real users experience:

• P50 (median): The middle. Half your requests are faster, half are slower.
• P95: 95% of requests are faster than this number.
• P99: 99% of requests are faster than this number.

Here’s what it looks like in practice:

Action: posts#index

• P50: 120ms ← typical case
• P95: 450ms ← 5% of users wait this long or more
• P99: 2.1s ← 1% of users are suffering

That P99 of 2.1 seconds is telling you something. If you have 1000 requests a day, that’s 10 users waiting over 2 seconds every single day.

Which Percentile Should You Use?

P50 (median): Too optimistic

P50 only tells you about the typical case. It completely ignores tail latency, i.e., the slow requests that frustrate users.

If P50 is 120ms but P95 is 2 seconds, you have a serious problem that P50 won’t show you. Half your users get a fast experience, but a significant chunk are having a terrible time.

Don’t use P50 to decide what’s slow. It hides too much.

P95: The sweet spot

It catches problems that affect enough users to matter. If P95 is 2 seconds, that means 5% of your users (1 in 20) are waiting that long. That’s significant.

It’s not so sensitive that every minor blip flags the system. You’re looking at the experience of a meaningful percentage of users, not just the absolute worst cases.

When to use P95:

• Setting performance thresholds for alerts
• Deciding if an action needs optimization
• Comparing performance across different endpoints

P99: More aggressive, catches edge cases

P99 is more aggressive than P95 as it looks at the worst 1% of requests. This catches the outliers, the edge cases, the weird scenarios.

Use P99 when:

• You want to understand your absolute worst-case performance
• You’re debugging specific slow requests
• You have extremely high traffic, and 1% still represents many users
• You’re operating at a scale where tail latency really matters (think Amazon, Google)

But for flagging what’s “slow” in most applications, P99 can be too noisy. That worst 1% might include legitimate edge cases—a user with a massive dataset, a bot, a weird network condition. Flagging everything where P99 exceeds your threshold might give you too many false positives.

The decision rule

Use P95 as your threshold for marking something as slow. Monitor P99 too; it tells you about edge cases worth investigating. But make decisions based on P95. Why? Because P95 catches problems that affect enough users to matter without drowning you in noise from edge cases.

What actually matters: server response time

Rails tells you this for free:

Completed 200 OK in 250ms (Views: 180ms | ActiveRecord: 45ms)

That 250ms is what the server spent processing the request. This is what’s considered in reality:

Fast enough that nobody complains:

• Under 100ms: Feels instant. Users are happy.
• 100-200ms: Still responsive. Most users won’t notice.

Getting into trouble territory:

• 200-500ms: Noticeable. Not great, not terrible.
• 500ms-1s: Users are tapping their fingers.
• 1-3 seconds: You’re losing people.
• Over 3 seconds: They’ve already opened another tab.

Of course, the context matters. A simple action with basic queries should be under 200ms. A complex dashboard with aggregations spending 500ms to a second might be acceptable. But anything consistently over 500ms deserves investigation.

Breaking down the bottlenecks

Your action response time is the sum of its parts. This is what we use as a baseline when we analyze each component of a request. Bear in mind that these values are just guidelines; they can vary from project to project and be influenced by business requirements (eg, SEO penalties) or context (eg, for an admin interface that’s used sparingly for very specific tasks, there’s no problem in relaxing these a little bit).

Database Queries

Your actions are only as fast as your slowest queries.

Fast:

• Under 10ms: Perfect. Nothing to do here, this is probably a properly designed query using the correct indexes.
• 10-50ms: Good for queries with optimized joins.

Acceptable:

• 50-100ms: Fine for moderately complex queries.
• 100-200ms: Okay for heavy aggregations.

Slow:

• 200-500ms: Here we start seeing things that are worth investigating.
• 500ms-1s: Definitely needs work.
• Over 1 second: We can consider these critical and MUST FIX if they are part of a critical path.

Simple queries (single table, indexed columns) should be under 10ms. If User.find(123) is taking 50ms, something’s wrong. Complex queries with joins and aggregations? They should be under 200ms.

Some of the common root causes of these slow queries we usually see when we are doing performance optimization work are missing indexes on foreign keys or WHERE/ORDER BY columns, N+1 queries, full table scans on large tables, and unoptimized LIKE queries with wildcards on both sides.

The power tool to uncover these: EXPLAIN ANALYZE. It will let you see execution plans and identify missing indexes or sequential scans.

View Rendering

View rendering time is usually high because of:

• Rendering too many partials (partials are slow!)
• N+1 queries hidden in view code
• Not using fragment caching where you could

Our suggestion for flagging views as slow is: if they are consistently over 100ms, investigate.

External API Calls

An action is only as fast as its slowest code statement. Hitting an external service in an action will kill your response time. This is not always possible, but we should work hard to avoid hitting 3rd party services via HTTP/network during the process of a request flow. Try to move those calls to background jobs and build a business process that takes into account asynchronicity around them.

In cases where the above is not possible, we try to target under 200ms for API calls. Anything over 500ms should be moved to background jobs or cached aggressively.

If you must make synchronous API calls, remember to set timeouts and have fallback behavior or use circuit breakers.

TL;DR: Thresholds

Here’s what to flag as slow using P95:

• Actions: P95 > 500ms
• Database queries: P95 > 100ms
• API calls: P95 > 200ms

And remember, these thresholds can vary from project to project and be influenced by business requirements (eg, SEO penalties) or context (eg, an admin interface that’s used sparingly for very specific tasks, there’s no problem in relaxing these a little bit), but they work as solid starting points.

Conclusion

Performance is an integral concern: Server time, network latency, asset delivery, and browser rendering, users experience all of it.

Of all these components, server time is where you have the most control. Every millisecond you shave off server response time is a millisecond that doesn’t add to the total user experience.

Look at P95 for your actions. Find the bottlenecks (database queries, view rendering, API calls) and fix what’s making users wait.

Always take the whole picture into account when prioritizing performance-related work, and put your effort where it will give your users the bigger benefits.