[mauroservienti]

The licensing queries on EndpointDocuments were dynamic. RavenDB served them with auto indexes in the throughput database, for example Auto/EndpointDocuments/ByEndpointId.ThroughputSource. Same problem as #5953 for retry batches.

Analysis

• An auto index follows the database setting Indexing.Auto.SearchEngineType and has no per-index search engine setting. The migration procedure in the docs changes the engine per index, so it cannot change these indexes. The search engine custom check and the start-up warning keep reporting them after all static indexes use Lucene.
• EndpointIndex existed, but DatabaseSetup deployed every index in the assembly to the error database, which has no endpoint documents. The throughput database had no static index.
• Its fields were named ThroughputSource and Name. The client maps a query on EndpointId.ThroughputSource to the field EndpointId_ThroughputSource, so the index could not serve the queries even in the right database.
• Without this change, the new test finds four auto indexes in the throughput database.

Changes

• IndexDeployment.CreateIndexesAsync takes an optional database name. IndexDeployment.FindIndexes exposes the discovery, which TestHelper.IndexDefinitionText now reuses.
• DatabaseSetup deploys the indexes in the Throughput.Indexes namespace to the throughput database and all others to the error database.
• EndpointIndex maps EndpointId_ThroughputSource, EndpointId_Name, SanitizedName and EndpointIndicators.
• The filtered queries in LicensingDataStore go through EndpointIndex. Queries without a filter, or only on the document id, stay collection queries, which create no index and are never stale.
• Every query through EndpointIndex waits for non-stale results, through one helper. The callers read their own writes: a usage report reads the throughput that was just recorded, the monitoring connection test checks for throughput that just arrived, and UpdateUserIndicatorOnEndpoints updates the documents the query returns. Before, the first dynamic query on a new database created the auto index and RavenDB waited for it. Without the wait, EndpointsTests and When_creating_a_usage_report_on_a_non_broker_transport failed on the Windows build. The index has one entry per endpoint and source, so the wait is short. If it stays stale for longer than the default 15 seconds, the query throws instead of returning stale results.
• New test LicensingDataStoreIndexTests.Licensing_queries_create_no_auto_index. It fails without the change and passes with it.
• DatabaseSetup deletes the throughput indexes from the error database, if they are there, before deploying the indexes. A locked index is unlocked first: RavenDB ignores the deletion of a LockedIgnore index and rejects that of a LockedError index, and an operator might have locked it after migrating it to Lucene. New test ThroughputIndexPlacementTests.Setup_should_delete_the_throughput_index_from_the_error_database, for each lock mode.

Existing databases

• No index is rebuilt. EndpointIndex in the throughput database is new and covers one document per endpoint and source.
• The EndpointIndex in the error database is deleted at start-up. The auto indexes in the throughput database are deleted by RavenDB once idle, or can be deleted in RavenDB Studio.
• IndexDefinitionApprovals changed because of the new EndpointIndex definition. Following docs/testing.md, the upgrade guide of the next release needs an entry, which belongs in docs.particular.net.

Testing

• ServiceControl.Persistence.Tests.RavenDB, filter Throughput|IndexDefinitionApprovals|IndexSetupTests: all pass except tests that fail in setup with a refused connection to the shared embedded server on port 33334, which also happens on master.
• ServiceControl.Audit.Persistence.Tests.RavenDB, filter IndexSetupTests|IndexDefinitionApprovals|DatabaseSettingsTests: 26/26.

Open points

• Add an upgrade guide entry for the changed index definitions in docs.particular.net, as docs/testing.md asks.