CARLoS: Retrieval via Concise Assessment Representation of LoRAs at Scale

Shahar Sarfaty, Adi Haviv, Uri Y. Hacohen, Niva Elkin-Koren, Roi Livni, Amit H. Bermano
Tel Aviv University
arXiv Code

TL;DR: We evaluate LoRAs, their effect and their properties, and provide an efficient retrieval system for a large corpus collected from CivitAI.

CARLoS

Given a large pool of community generated Low Rank Adapters (LoRAs), our method, CARLoS, concisely represents them according to their influence on generation, and retrieves semantically relevant ones given a retrieval query (left). Since our efficient retrieval (right) is generation based, it finds the LoRAs that are visually similar to the query, outperforming retrieval methods that rely on the name and textual descriptions provided by LoRA creators (middle).

Abstract

The rapid proliferation of generative components, such as LoRAs, has created a vast but unstructured ecosystem. Existing discovery methods depend on unreliable user descriptions or biased popularity metrics, hindering usability. We present CARLoS, a large-scale framework for characterizing LoRAs without requiring additional metadata. Analyzing over 650 LoRAs, we employ them in image generation over a variety of prompts and seeds, as a credible way to assess their behavior. Using CLIP embeddings and their difference to a base-model generation, we concisely define a three-part representation: Directions, defining semantic shift; Strength, quantifying the significance of the effect; and Consistency, quantifying how stable the effect is. Using these representations, we develop an efficient retrieval framework that semantically matches textual queries to relevant LoRAs while filtering overly strong or unstable ones, outperforming textual baselines in automated and human evaluations. While retrieval is our primary focus, the same representation also supports analyses linking Strength and Consistency to legal notions of substantiality and volition, key considerations in copyright, positioning CARLoS as a practical system with broader relevance for LoRA analysis.

Example retrieval results by CARLoS

How does it work?

CARLoS Overview

Given a set of curated LoRAs operating over the SDXL backbone, we represent each one as a three parts vector, used for efficient retrieval (left). To create our concise representation (top), we generate for each LoRA and the vanilla backbone images using N = 280 prompts and M = 16 seeds. We measure the semantic difference between the vanilla generation and the LoRAs in CLIP space (CLIP-diff), and store their average as a representative Direction effect, their mean magnitude to represent effect Strength, and their variance as a measure for Consistency. During retrieval (bottom), we measure the average CLIP space difference between a set of N different prompts with and without the retrieval query appended. We then simply retrieve the LoRAs with the most similar Direction vectors, and filter out LoRAs demonstrating above-threshold Strength and under-threshold Consistency.

CARLoS Metrics - Distribution and Visualization

CARLoS Overview

Our LoRA dataset, in Consistency Rank vs. Strength Rank distribution. Too strong and too inconsistent LoRAs (red regions) are filtered out. Example generations for two prompts for different strength and consistency LoRAs are visualized.

Comparisons

Qualitative comparisons of textual description-based retrieval (bottom rows) to CARLoS (top row). While some effects are sufficiently described in text (e.g., Pixel Art) and are therefore retrieved well, more elaborate queries, (such as celestial beings, or futuristic games) are not described well, resorting textual-based retrieval to similar wording as opposed to effects (e.g., clouds, cartoons).

Quantitative Comparison

Retrieval Performance Evaluated by Different VLMs. Scores indicate the quality of retrieved top-3 LoRAs as judged by state-of-the-art Vision-Language Models. CARLoS consistently yields results preferred by all evaluators. The scores are normalized in min-max manner across all queries and retrievers.

User Study

Aggregated results of our subjective user study. Participants compared CARLoS against four strong textual retrieval baselines (QWEN3, E5, BGE, GTE) across three criteria. CARLoS was consistently preferred in all categories.

Legal Application

Legal Application

Legal considerations. LoRAs expose users to liability and rights. Weak or inconsistent LoRAs (top), are unlikely to impose infringement or authorship. Strong consistent LoRAs may infringe depending on protected elements replication, or distinct styles (bottom-right,middle), but not necessarily (bottom-left).