多应用+插件架构,代码干净,二开方便,首家独创一键云编译技术,文档视频完善,免费商用码云13.8K 广告
# Geo self-service framework (alpha) > 原文:[https://docs.gitlab.com/ee/development/geo/framework.html](https://docs.gitlab.com/ee/development/geo/framework.html) * [Nomenclature](#nomenclature) * [Geo Domain-Specific Language](#geo-domain-specific-language) * [The replicator](#the-replicator) * [Linking to a model](#linking-to-a-model) * [API](#api) * [Library](#library) * [Existing Replicator Strategies](#existing-replicator-strategies) * [Blob Replicator Strategy](#blob-replicator-strategy) * [Replication](#replication) * [Verification](#verification) * [Metrics](#metrics) * [GraphQL API](#graphql-api) * [Admin UI](#admin-ui) # Geo self-service framework (alpha)[](#geo-self-service-framework-alpha "Permalink") **注意:**本文档可能会随时更改. 这是我们正在研究的建议,一旦实施完成,此文档将得到更新. 跟随[史诗般的](https://gitlab.com/groups/gitlab-org/-/epics/2161)进度.**注意:** Geo 自助服务框架当前处于 Alpha 状态. 如果您需要复制新的数据类型,请与 Geo 小组联系以讨论选项. 您可以在 Slack 的`#g_geo`与他们联系,或在问题或合并请求中提及`@geo-team` . Geo 提供了一个 API,使跨 Geo 节点轻松复制数据类型成为可能. 该 API 以 Ruby 域特定语言(DSL)的形式呈现,旨在使创建数据类型的工程师只需花费很少的精力即可复制数据. ## Nomenclature[](#nomenclature "Permalink") 在深入研究 API 之前,开发人员需要了解一些特定于地理位置的命名约定. Model 模型是活动模型,在整个 Rails 代码库中都是如此. 它通常与数据库表绑定. 从地理角度来看,模型可以具有一个或多个资源. Resource 资源是属于模型的一条数据,由 GitLab 功能生成. 使用存储机制将其持久化. 默认情况下,资源不可复制. Data type Data type is how a resource is stored. Each resource should fit in one of the data types Geo supports: :- Git repository :- Blob :- Database 有关更多详细信息,请参见[数据类型](../../administration/geo/replication/datatypes.html) . Geo Replicable 可复制资源是 Geo 希望在 Geo 节点之间同步的资源. 受支持的可复制数据类型有限. 实现属于已知数据类型之一的资源的复制所需的工作量很小. Geo Replicator 地理复制器是知道如何复制可复制对象的对象. 它负责::-触发事件(生产者):-消费事件(消费者) 它与 Geo Replicable 数据类型相关. 所有复制器都有一个公共接口,可用于处理(即产生和使用)事件. 它负责主节点(产生事件的地方)和次节点(消耗事件的地方)之间的通信. 想要将 Geo 纳入其功能的工程师将使用复制器的 API 来实现这一目标. Geo Domain-Specific Language 语法糖使工程师可以轻松指定应复制哪些资源以及如何复制. ## Geo Domain-Specific Language[](#geo-domain-specific-language "Permalink") ### The replicator[](#the-replicator "Permalink") 首先,您需要编写一个复制器. 复制器位于[`ee/app/replicators/geo`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/app/replicators/geo) . 对于每个需要复制的资源,即使多个资源绑定到同一模型,也应指定一个单独的复制器. 例如,以下复制器复制软件包文件: ``` module Geo class PackageFileReplicator < Gitlab::Geo::Replicator # Include one of the strategies your resource needs include ::Geo::BlobReplicatorStrategy # Specify the CarrierWave uploader needed by the used strategy def carrierwave_uploader model_record.file end # Specify the model this replicator belongs to def self.model ::Packages::PackageFile end end end ``` 类名应该是唯一的. 它还与注册表的表名紧密相关,因此在此示例中,注册表表将为`package_file_registry` . 对于不同的数据类型,Geo 支持包括不同的策略. 选择一个适合您的需求. ### Linking to a model[](#linking-to-a-model "Permalink") 要将此复制器绑定到模型,需要在模型代码中添加以下内容: ``` class Packages::PackageFile < ApplicationRecord include ::Gitlab::Geo::ReplicableModel with_replicator Geo::PackageFileReplicator end ``` ### API[](#api "Permalink") 设置好后,可以通过模型轻松访问复制器: ``` package_file = Packages::PackageFile.find(4) # just a random id as example replicator = package_file.replicator ``` 或者从复制器取回模型: ``` replicator.model_record => <Packages::PackageFile id:4> ``` 复制器可用于生成事件,例如在 ActiveRecord 挂钩中: ``` after_create_commit -> { replicator.publish_created_event } ``` #### Library[](#library "Permalink") 所有这些背后的框架位于[`ee/lib/gitlab/geo/`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/lib/gitlab/geo) . ## Existing Replicator Strategies[](#existing-replicator-strategies "Permalink") 在编写一种新的复制器策略之前,请检查以下内容,以查看现有策略之一是否已经可以处理您的资源. 如果不确定,请咨询地理团队. ### Blob Replicator Strategy[](#blob-replicator-strategy "Permalink") 使用`Geo::BlobReplicatorStrategy`模块,Geo 可以轻松支持使用[CarrierWave 的](https://github.com/carrierwaveuploader/carrierwave) `Uploader::Base`模型. 首先,每个文件应具有其自己的主要 ID 和模型. Geo 强烈建议将*每个文件都*视为头等公民,因为根据我们的经验,这大大简化了跟踪复制和验证状态. 例如,要添加对具有`Widget` `widgets`表的`Widget`模型引用的文件的支持,您将执行以下步骤: #### Replication[](#replication "Permalink") 1. 在`Widget`类中包含`Gitlab::Geo::ReplicableModel` ,并使用`with_replicator Geo::WidgetReplicator`指定 Replicator 类. 此时, `Widget`类应如下所示: ``` # frozen_string_literal: true class Widget < ApplicationRecord include ::Gitlab::Geo::ReplicableModel with_replicator Geo::WidgetReplicator mount_uploader :file, WidgetUploader def self.replicables_for_geo_node # Should be implemented. The idea of the method is to restrict # the set of synced items depending on synchronization settings end ... end ``` 2. 创建`ee/app/replicators/geo/widget_replicator.rb` . 实现`#carrierwave_uploader`方法,该方法应返回`CarrierWave::Uploader` . 并实现类方法`.model`以返回`Widget`类. ``` # frozen_string_literal: true module Geo class WidgetReplicator < Gitlab::Geo::Replicator include ::Geo::BlobReplicatorStrategy def self.model ::Widget end def carrierwave_uploader model_record.file end end end ``` 3. 创建`ee/spec/replicators/geo/widget_replicator_spec.rb`并执行必要的设置,以定义共享示例的`model_record`变量. ``` # frozen_string_literal: true require 'spec_helper' RSpec.describe Geo::WidgetReplicator do let(:model_record) { build(:widget) } it_behaves_like 'a blob replicator' end ``` 4. 创建`widget_registry`表,以便 Geo 次要对象可以跟踪每个 Widget 文件的同步和验证状态: ``` # frozen_string_literal: true class CreateWidgetRegistry < ActiveRecord::Migration[6.0] DOWNTIME = false disable_ddl_transaction! def up unless table_exists?(:widget_registry) ActiveRecord::Base.transaction do create_table :widget_registry, id: :bigserial, force: :cascade do |t| t.integer :widget_id, null: false t.integer :state, default: 0, null: false, limit: 2 t.integer :retry_count, default: 0, limit: 2 t.text :last_sync_failure t.datetime_with_timezone :retry_at t.datetime_with_timezone :last_synced_at t.datetime_with_timezone :created_at, null: false t.index :widget_id t.index :retry_at t.index :state end end end add_text_limit :widget_registry, :last_sync_failure, 255 end def down drop_table :widget_registry end end ``` 5. Create `ee/app/models/geo/widget_registry.rb`: ``` # frozen_string_literal: true class Geo::WidgetRegistry < Geo::BaseRegistry include Geo::ReplicableRegistry MODEL_CLASS = ::Widget MODEL_FOREIGN_KEY = :widget_id belongs_to :widget, class_name: 'Widget' end ``` 方法`has_create_events?` 在大多数情况下应该返回`true` . 但是,如果您添加的实体没有创建事件,则根本不要添加该方法. 6. Update `REGISTRY_CLASSES` in `ee/app/workers/geo/secondary/registry_consistency_worker.rb`. 7. Create `ee/spec/factories/geo/widget_registry.rb`: ``` # frozen_string_literal: true FactoryBot.define do factory :geo_widget_registry, class: 'Geo::WidgetRegistry' do widget state { Geo::WidgetRegistry.state_value(:pending) } trait :synced do state { Geo::WidgetRegistry.state_value(:synced) } last_synced_at { 5.days.ago } end trait :failed do state { Geo::WidgetRegistry.state_value(:failed) } last_synced_at { 1.day.ago } retry_count { 2 } last_sync_failure { 'Random error' } end trait :started do state { Geo::WidgetRegistry.state_value(:started) } last_synced_at { 1.day.ago } retry_count { 0 } end end end ``` 8. Create `ee/spec/models/geo/widget_registry_spec.rb`: ``` # frozen_string_literal: true require 'spec_helper' RSpec.describe Geo::WidgetRegistry, :geo, type: :model do let_it_be(:registry) { create(:geo_widget_registry) } specify 'factory is valid' do expect(registry).to be_valid end include_examples 'a Geo framework registry' describe '.find_registry_differences' do ... # To be implemented end end ``` 小部件现在应该由 Geo 复制! #### Verification[](#verification "Permalink") 1. 将验证状态字段添加到`widgets`表中,以便 Geo 主数据库可以跟踪验证状态: ``` # frozen_string_literal: true class AddVerificationStateToWidgets < ActiveRecord::Migration[6.0] DOWNTIME = false def change add_column :widgets, :verification_retry_at, :datetime_with_timezone add_column :widgets, :verified_at, :datetime_with_timezone add_column :widgets, :verification_checksum, :binary, using: 'verification_checksum::bytea' add_column :widgets, :verification_failure, :string add_column :widgets, :verification_retry_count, :integer end end ``` 2. 在`verification_failure`和`verification_checksum`上添加部分索引,以确保可以高效执行重新验证: ``` # frozen_string_literal: true class AddVerificationFailureIndexToWidgets < ActiveRecord::Migration[6.0] include Gitlab::Database::MigrationHelpers DOWNTIME = false disable_ddl_transaction! def up add_concurrent_index :widgets, :verification_failure, where: "(verification_failure IS NOT NULL)", name: "widgets_verification_failure_partial" add_concurrent_index :widgets, :verification_checksum, where: "(verification_checksum IS NOT NULL)", name: "widgets_verification_checksum_partial" end def down remove_concurrent_index :widgets, :verification_failure remove_concurrent_index :widgets, :verification_checksum end end ``` 要做的事情:在二级服务器上添加验证. 这应作为以下内容的一部分完成[:Geo:自助服务框架-包文件验证的首次实现](https://gitlab.com/groups/gitlab-org/-/epics/1817) 小部件现在应由 Geo 验证! #### Metrics[](#metrics "Permalink") 指标由`Geo::MetricsUpdateWorker`收集,保存在`GeoNodeStatus`以显示在 UI 中,然后发送给 Prometheus. 1. 将字段`widget_count` , `widget_checksummed_count` , `widget_checksum_failed_count` , `widget_synced_count` , `widget_failed_count`和`widget_registry_count`到`ee/app/models/geo_node_status.rb` `GeoNodeStatus#RESOURCE_STATUS_FIELDS`数组中. 2. 将相同的字段添加到`ee/app/models/geo_node_status.rb` `GeoNodeStatus#PROMETHEUS_METRICS`哈希中. 3. 将相同字段添加到`doc/administration/monitoring/prometheus/gitlab_metrics.md` `Sidekiq metrics`表中. 4. 将相同的字段添加到`doc/api/geo_nodes.md` `GET /geo_nodes/status`示例响应中. 5. 将相同的字段添加到`ee/spec/models/geo_node_status_spec.rb`和`ee/spec/factories/geo_node_statuses.rb` . 6. Set `widget_count` in `GeoNodeStatus#load_data_from_current_node`: ``` self.widget_count = Geo::WidgetReplicator.primary_total_count ``` 7. 添加`GeoNodeStatus#load_widgets_data`来设置`widget_synced_count` , `widget_failed_count`和`widget_registry_count` : ``` def load_widget_data self.widget_synced_count = Geo::WidgetReplicator.synced_count self.widget_failed_count = Geo::WidgetReplicator.failed_count self.widget_registry_count = Geo::WidgetReplicator.registry_count end ``` 8. Call `GeoNodeStatus#load_widgets_data` in `GeoNodeStatus#load_secondary_data`. 9. Set `widget_checksummed_count` and `widget_checksum_failed_count` in `GeoNodeStatus#load_verification_data`: ``` self.widget_checksummed_count = Geo::WidgetReplicator.checksummed_count self.widget_checksum_failed_count = Geo::WidgetReplicator.checksum_failed_count ``` 小部件复制和验证指标现在应该可以在 API,管理区域 UI 和 Prometheus 中使用! #### GraphQL API[](#graphql-api "Permalink") 1. 在`ee/app/graphql/types/geo/geo_node_type.rb`向`GeoNodeType`添加一个新字段: ``` field :widget_registries, ::Types::Geo::WidgetRegistryType.connection_type, null: true, resolver: ::Resolvers::Geo::WidgetRegistriesResolver, description: 'Find widget registries on this Geo node', feature_flag: :geo_self_service_framework ``` 2. 新添加`widget_registries`字段名的`expected_fields`在阵列`ee/spec/graphql/types/geo/geo_node_type_spec.rb` . 3. Create `ee/app/graphql/resolvers/geo/widget_registries_resolver.rb`: ``` # frozen_string_literal: true module Resolvers module Geo class WidgetRegistriesResolver < BaseResolver include RegistriesResolver end end end ``` 4. Create `ee/spec/graphql/resolvers/geo/widget_registries_resolver_spec.rb`: ``` # frozen_string_literal: true require 'spec_helper' RSpec.describe Resolvers::Geo::WidgetRegistriesResolver do it_behaves_like 'a Geo registries resolver', :geo_widget_registry end ``` 5. Create `ee/app/finders/geo/widget_registry_finder.rb`: ``` # frozen_string_literal: true module Geo class WidgetRegistryFinder include FrameworkRegistryFinder end end ``` 6. Create `ee/spec/finders/geo/widget_registry_finder_spec.rb`: ``` # frozen_string_literal: true require 'spec_helper' RSpec.describe Geo::WidgetRegistryFinder do it_behaves_like 'a framework registry finder', :geo_widget_registry end ``` 7. Create `ee/app/graphql/types/geo/widget_registry_type.rb`: ``` # frozen_string_literal: true module Types module Geo # rubocop:disable Graphql/AuthorizeTypes because it is included class WidgetRegistryType < BaseObject include ::Types::Geo::RegistryType graphql_name 'WidgetRegistry' description 'Represents the sync and verification state of a widget' field :widget_id, GraphQL::ID_TYPE, null: false, description: 'ID of the Widget' end end end ``` 8. Create `ee/spec/graphql/types/geo/widget_registry_type_spec.rb`: ``` # frozen_string_literal: true require 'spec_helper' RSpec.describe GitlabSchema.types['WidgetRegistry'] do it_behaves_like 'a Geo registry type' it 'has the expected fields (other than those included in RegistryType)' do expected_fields = %i[widget_id] expect(described_class).to have_graphql_fields(*expected_fields).at_least end end ``` 9. Add integration tests for providing Widget registry data to the frontend via the GraphQL API, by duplicating and modifying the following shared examples in `ee/spec/requests/api/graphql/geo/registries_spec.rb`: ``` it_behaves_like 'gets registries for', { field_name: 'widgetRegistries', registry_class_name: 'WidgetRegistry', registry_factory: :geo_widget_registry, registry_foreign_key_field_name: 'widgetId' } ``` 现在应该可以通过 GraphQL API 获得各个小部件同步和验证数据! 1. 注意复制"更新"事件. Geo Framework 目前不支持复制"更新"事件,因为此时添加到框架的所有实体都是不可变的. 如果您要添加的实体属于这种情况,请遵循[https://gitlab.com/gitlab-org/gitlab/-/issues/118743](https://gitlab.com/gitlab-org/gitlab/-/issues/118743)和[https://gitlab.com/gitlab-org/gitlab /// issues / 118745](https://gitlab.com/gitlab-org/gitlab/-/issues/118745)作为添加新事件类型的示例. 添加通知后,请同时删除它. #### Admin UI[](#admin-ui "Permalink") 要做的事情:这应该作为《 [地理手册》的](https://gitlab.com/groups/gitlab-org/-/epics/2525)一部分完成[:实现自助服务框架可复制的前端](https://gitlab.com/groups/gitlab-org/-/epics/2525) 窗口小部件同步和验证数据(总计和个人)现在应该在管理界面中可用!