Use the AddItemsPlugin to add your custom filter to the Studio Library filter section. The component to configure is the SearchFilters class.

<editor:StudioPlugin>
  <editor:rules>
    <editor:SearchFilters>
      <editor:plugins>
        <ui:AddItemsPlugin>
          <ui:items>
            <foo:FooFilterPanel itemId="fooFilter"/>
          </ui:items>
          <ui:after>
            <Component itemId="{SearchFilters.LAST_EDITED_FILTER_ITEM_ID}"/>
          </ui:after>
        </ui:AddItemsPlugin>
      </editor:plugins>
    </editor:SearchFilters>
  </editor:rules>
</editor:StudioPlugin>

You can also open the library in a certain filter state, for example from a button in the favorites toolbar. To that end, the ShowCollectionViewAction provides a property filters that can take SearchFilterState objects. So that the action can configure the correct filter, the filterId attribute must be given, matching the itemId of the configured filter panel. The names and values of the attributes are exactly the property names and values of the state bean used by the filter set.