AI for Umbraco Media

Empower your editors with ExamineX’s AI capabilities to transform your media library! Seamlessly search using auto-generated Descriptions, Tags, Categories, Locations, People, and more 🙌

You can also automatically have the contents of your Umbraco media files (i.e. PDF, MS Word, etc…) indexed with ExamineX without the need for additional indexes.

Requirements

If you are using Azure to host your Umbraco website it is recommended to use blob storage as your media provider. This provides more flexibility with scaling your solution along with the benefits of CDN support.

With this ExamineX extension, your media document files such as PDFs and Microsoft Office docs will automatically have their content’s indexed. And with AI capabilities enabled, your images will be scanned and and auto-generated metadata can be produced such as OCR extracted text, Descriptions, Tags, Categories, Locations, People, and more 🎉.

Installation

After installing and configuring ExamineX…

public void ConfigureServices(IServiceCollection services)
{
    services.AddUmbraco(_env, _config)
        .AddBackOffice()
        .AddWebsite()
        .AddDeliveryApi()
        .AddComposers()
        .AddExamineXAzureSearch()
        .AddExamineXAzureSearchForMedia()
        .Build();
}

public void ConfigureServices(IServiceCollection services)
{
    services.AddUmbraco(_env, _config)
        .AddBackOffice()
        .AddWebsite()
        .AddDeliveryApi()
        .AddComposers()
        .AddExamineXAzureSearch()
        .AddExamineXForBlobMedia()
        .Build();
}

Once the package is installed any PDF files, MS Office document files, and others file types will automatically be indexed and stored in your corresponding internal/external Umbraco indexes with the field name content.

NOTE: The field name ‘content’ cannot be changed. This is a limitation of Azure Search’s field mapping. For this to work you should not have a Property Type called content.

AI Integration (v6.1+)

Once the above package is installed, you can configure the AI integration for image analysis.

Quick config

The simplest config using defaults to enable this feature in your appsettings.json file will be:

{
  "ExamineX": {
    "AzureSearch": {
      "Media": {
        "EnableImageAnalysis": true
      }
    }
  }
}

Once this is enabled and the Azure Search configured indexer is executed (i.e. a media item is saved), the media items in the index will be populated with generative AI details as per your configuration. ExamineX also automatically configures the Umbraco back office search to include the relevant fields so that your editors can easily find media based on the AI generated information. For example, for this image:

It will generate and index this information:

Now, when your editors search in the backoffice, media will be found based on what is in the image:

Similarly, when searching for media in a media picker, ExamineX has configured the search to work against the index so your editors can find the media they need quickly:

Integrating UI

The ExamineX.AzureSearch.Umbraco.Media package will install a new Property Editor called: ExamineX Media Info which is a readonly property editor to display the generated/indexed information for a media item based on the fields populated by Azure AI Search.

The simplest way to integrate this is to:

• Create a new Data Type based on the ExamineX Media Info.
• Update your Image Media Type, add a Property Type with your newly created Data Type.

ExamineX Media configuration options

Index fields

Several index fields will be created/used based on the image analysis features enabled:

Searching

Searching on this content is exactly the same way you would search any field in Examine. For example, if you wanted to search for a term within the contents of a media file in the ExternalIndex, you could do:

 if(ExamineManager.TryGetIndex("ExternalIndex", out var index))
{
    var searcher = index.GetSearcher();

    // Query on the 'content' field for media
    var results = searcher
        .CreateQuery("media")
        .Field("content", searchTerm)
        .Execute();
}