Enhancing Translation Consistency In Weblate With The Similar Strings Tab

In the realm of localization, maintaining consistency across translations is paramount. This is especially true in large projects where multiple translators are involved, or where slight variations in phrasing can lead to inconsistencies in the final product. Currently, Weblate, a powerful web-based translation tool, offers a "Similar keys" tab that aids in this process by highlighting translation keys (msgid) with semantic similarities. However, in projects where keys are either source strings themselves or opaque identifiers, this feature's utility is limited. To address this gap, a proposal has been made to introduce a new "Similar strings" tab, which promises to revolutionize the way translators work by focusing on the textual similarity of source strings.

The Problem: Inconsistent Translations Due to Similar Source Strings

The core challenge lies in the potential for inconsistencies arising from similar, yet not identical, source strings. In many translation projects, keys often mirror the source strings or function as opaque identifiers, making the "Similar keys" tab less effective. Consider these scenarios where textual similarity is crucial:

• Slight variations in a phrase: For example, the difference between "Click the button" and "Please click the button" may seem subtle, but without a tool to highlight this similarity, different translators might render them differently, leading to an inconsistent user experience.
• Terminology inconsistencies: Translators might struggle to recall the specific terminology used for a similar concept elsewhere in the project. This can lead to a fragmented and unprofessional final product.
• Verb conjugations and plural forms: Translating these without the context of their base form can be problematic. A translator might not realize that a particular verb form or plural noun has already been translated in a specific way, leading to inconsistencies.

The current workaround, relying on global search, is far from ideal. It disrupts the translator's workflow, requiring them to leave their current context and manually sift through search results without a ranked list of similarity. This is where the "Similar strings" tab steps in to bridge the gap.

The Solution: Introducing the "Similar Strings" Tab

The proposed solution is to add a new tab in the translation interface, aptly named "Similar strings," positioned alongside existing tabs like "Nearby strings," "Similar keys," and "History." This feature would function much like the "Similar keys" tab, but instead of focusing on key similarity, it would perform a similarity search based on the content of the source string. This shift in focus addresses the core problem of textual variations and their impact on translation consistency.

How the "Similar Strings" Tab Works

Imagine a translator working on the English string "A quick brown fox." The "Similar strings" tab would then display other source strings from the same component, ranked by similarity. This could include:

• "The quick brown fox jumps over the lazy dog." (High similarity)
• "A quick brown dog." (Medium similarity)
• "A slow red fox." (Lower similarity)

This ranked list provides immediate context and helps the translator make informed decisions, ensuring that similar phrases are translated consistently throughout the project. The key is to allow the translation remember in what contexts he has translated same strings to maintain a consistency.

Technical Implementation

Bringing this feature to life involves both backend logic and frontend UI enhancements.

Backend Logic (Search)

Two primary approaches have been considered for implementing the search functionality:

1. Database-backed Full-Text Search (FTS) (Recommended): This scalable approach leverages the power of database indexing for efficient similarity searching. For PostgreSQL, the pg_trgm extension and TrigramSimilarity function offer a robust solution. This implementation would involve:

   • Adding a SearchVectorField to the String model in weblate.trans.models. This field would store the vectorized representation of the source string, enabling fast similarity comparisons.
   • Creating a database migration to add the new field and a mechanism to keep the vector updated whenever a source string is modified. This ensures that the search index is always up-to-date.
   • Developing a function that utilizes TrigramSimilarity or SearchRank to find and rank similar strings within the same component. TrigramSimilarity breaks down strings into trigrams (sequences of three characters) and calculates similarity based on the number of shared trigrams. SearchRank provides a more sophisticated ranking based on term frequency and inverse document frequency.

2. rapidfuzz Library (Simpler): This alternative approach utilizes the existing rapidfuzz library, a Weblate dependency, for fuzzy string matching. A function could iterate over strings in the component and calculate a fuzz.ratio score, which represents the similarity between two strings. While simpler to implement for a proof-of-concept, this approach may be less performant on very large components due to the iterative nature of the search.

API Endpoint

A new API endpoint, such as /api/units/<unit_id>/similar-strings/, would be created to serve the list of similar strings. This endpoint would asynchronously fetch the data when the user clicks the tab, preventing any performance impact on the initial page load. This approach mirrors the implementation of other tabs like "History" and "Machinery," ensuring a consistent user experience.

Frontend UI

The frontend UI would be modified to incorporate the new "Similar strings" tab and a corresponding panel. This involves:

• Modifying the translation view template (weblate/templates/trans/translate.html) to add the new tab.
• Using JavaScript (e.g., in weblate/static/js/trans.js) to handle the AJAX call to the new API endpoint when the tab is activated. This asynchronous call ensures a smooth user experience without blocking the main thread.
• Rendering the returned list of similar strings in the panel, with each item linking to its respective translation unit. This direct link allows the translator to quickly navigate to the related string and ensure consistency.

Alternatives Considered

Before proposing the "Similar strings" tab, alternative solutions were evaluated:

1. Manual Search: As mentioned earlier, manual search is a viable but inefficient workaround. It disrupts the translator's workflow and lacks the ranked similarity list crucial for consistency.
2. Translation Memory (TM): While invaluable for exact or high-percentage matches, TM falls short in identifying conceptually or partially similar strings. For instance, TM might not link "Log in to your account" with "Please sign in to continue," while a textual similarity search would likely highlight this connection. This feature complements the TM by providing a different axis of similarity, focused on the source text itself.

The "Similar Strings" Tab: A Powerful Addition to Weblate

The introduction of the "Similar strings" tab represents a significant enhancement to Weblate's capabilities. It empowers translators to maintain consistency and efficiency, particularly in large projects with numerous similar strings. By leveraging existing technologies within Weblate, such as rapidfuzz and powerful database backends, this feature seamlessly integrates into the existing UI and workflow. It sits perfectly between the "Nearby strings" and "Translation Memory" features, providing a comprehensive suite of tools for translators. The result is a more consistent, professional, and user-friendly translated product.

Screenshots of the Proposed Implementation

The new "Similar strings" tab would be located alongside the existing tabs:

[ Nearby strings ] [ Similar keys ] [ Similar strings ] [ Other languages ] [ History ]

Additional Context

This feature would significantly enhance translator consistency and efficiency, especially in large projects with many similar but not identical strings. It leverages existing technologies in Weblate (rapidfuzz, powerful database backends) and follows established UI patterns within the application. It would be a powerful addition to the translator's toolkit, sitting nicely between the "Nearby strings" and "Translation Memory" features.