This is a topic that I’ve been struggling with since I started using the Realm database.
I’ve raised this question earlier in this thread and I thought that I have found a solution. However, after doing a very deep performance analysis that you can find in this article - Creating memory efficient reactive IObservable<IChangeset<T>> based on Realm database query · reactivemarbles/DynamicData · Discussion #894 (github.com), I’ve understood that the solution I thought I found has major memory and performance issues.
Let me restate the problem for visibility purposes.
We have a pretty complex application that uses MongoDB App Services and Realm.
The realm database has a fantastic feature that allows data to be lazy-loaded when accessed. This has pretty exciting effects. Imagine you have a screen that shows a list of items. Those items can be tens of thousands. In the traditional database or API approach, one would implement some sort of data pagination - either automatic, when the user reaches the end of the list, the next chunk of data is loaded, or manual - the user manually selects the page they should load. With realm, you can fetch all of the items of an entity and directly bind those items to a ListView control in Xamarin. For example:
public class Product : RealmObject
{
public string Name { get; set; }
public decimal Price { get; set; }
}
public class ProductsViewModel
{
private readonly Realm _realm;
public IEnumerable<Product> Products { get; set; }
public ProductsViewModel(Realm realm)
{
_realm = realm;
}
public void LoadProducts()
{
Products = _realm.All<Product>();
}
}
//The xaml page
<ListView
ItemsSource="{Binding Products}"
CachingStrategy="RecycleElement">
<ListView.ItemTemplate>
<DataTemplate>
<TextCell Text="{Binding Name}"/>
</DataTemplate>
</ListView.ItemTemplate>
</ListView>
In the above example, even if there are tens of thousands of products, the ListView will load instantly and will scroll smoothly because of the lazy-loading magic.
The issue arises when we need to transform the native realm object Product into a different class. Imagine we want to create a new view model out of the Product class, called ProductViewModel :
public class ProductViewModel
{
public ProductViewModel(Product product)
{
Name = product.Name;
FormattedPrice = $"{product.Price:C}";
}
public string Name { get; set; }
public string FormattedPrice { get; set; }
}
This is a trivial example of simply formatting the currency into a string. The scenario is much more complicated in our real-world application, but it’s good to show the issue we are facing. One would guess that changing the ProductsViewModel to something like this would do the job:
public class ProductsViewModel
{
private readonly Realms.Realm _realm;
public IEnumerable<ProductViewModel> Products { get; set; }
public ProductsViewModel(Realms.Realm realm)
{
_realm = realm;
}
public void LoadProducts()
{
Products = _realm.All<Product>().Select(p => new ProductViewModel(p));
}
}
However, this approach is naive and will essentially break the list’s lazy-loading aspect. It will iterate over every item in the list, fetching the product’s name and price and creating a new ProductViewModel. Only after all of these operations are completed will the list show any data. If the product collection is extensive, this will take significant time to complete.
How can we approach solving this problem the right way? Essentially, we need some reactive collection transform operator that does the following.
- It preserves the reactive notifications of the original collection. If the original Realm collection changes, the transformed collection should also omit collection change events
- Items of the collection should be evaluated lazily whenever the list shows
The solution that I thought I’ve found was based on using DynamicData and hooking it to the Realm query results`
realm.All<Product>().AsRealmCollection()
.AsObservableChangeSet()
.Filter(item => item.IsValid)
.Transform(p => new ProductViewModel(p));
However, as I mentioned above, I’ve explained in detail in this article why this is not working and essentially materializes the realm collection under the hood.
In that dynamic data discussion thread, we’ve concluded that the only real solution to the problem that will preserve all benefits of the realm query (reactiveness and lazy loading) is the one that would be provided by the Realm SDK itself - a real pagination and transformation support. Something like this
realm.All<Product>().Skip(10).Take(50).Select(p => new ProductViewModel(p));
The above LINQ statement will not work now as the Skip, Take, Select LINQ operators are not supported.
As a temporary solution, we’ve implemented an artificial pagination so that the transformed objects are created only for a subset of items, which won’t take much time. But this is a super undesired approach, as it breaks all of the Realm database’s benefits. This solution is described in the Approach 5 of the article I mentioned a couple of times.
P.S.
If we go further, there’s another issue we do have, which probably is outside of this specific topic and is a more complicated variant. Imagine you have a list that should show data from 2 different Realm collections. How would we manage this? We need a mechanism to merge 2 Realm collections reactively.
I hope we will meet someday so I can return the favor, haha!