Docs / Admin control planeSDK downloadsGitHub ↗

Keme Super-Admin Control Plane

Purpose

Keme Control is the restricted operational interface for PushHub platform administrators. It is served at https://admin.pushhub.kemegames.com/admin and is separate from customer workspaces.

Information architecture

The approval-based onboarding expansion adds Trial Requests as a required admin module. It should sit before Organizations because it is the intake queue that provisions new tenant workspaces after Keme review.

Implemented route: /admin/trial-requests. The module supports submitted/approved/rejected/expired filtering, requester/company search, approve with plan/trial settings, reject with admin notes, setup-email delivery state, resend setup email, admin-only setup-link recovery, and audit-backed review actions.

  1. Command Center — tenant, user, app, device, and notification totals; newest organizations; tenant health; core-service summary.
  2. Organizations — searchable tenant directory, status filters, footprint, plan visibility, suspension/reactivation, and plan changes.
  3. Apps & Environments — cross-tenant project inventory, platform/provider readiness, state, and device footprint.
  4. Users & Roles — workspace identities, verification/login state, and Keme super-admin operators.
  5. Plans & Billing — subscription plans, prices, features, and entitlement limits.
  6. Campaigns & Delivery — cross-tenant campaign activity and queued/sent/delivered/opened/failed delivery totals.
  7. API Credentials — safe key metadata, prefixes, scopes, last use, and revocation state; never plaintext secrets.
  8. Integrations — FCM/APNs readiness and the core data-plane connections.
  9. Audit Trail — system and super-admin events, ordered newest first.
  10. Data Governance — implemented safeguards and an explicit enterprise-governance backlog.
  11. Platform Health — API heartbeat and the operational boundary for PostgreSQL, Redis, storage, FCM, and APNs.
  12. Access & Security — named operators, role assignment, secure invitations, activation/deactivation, invitation history, and session posture.
  13. Delivery Engine — dedicated worker freshness, durable job states, leases, retry/dead-letter volume, secret-free provider attempts, and audited recovery.
  14. Provider Fleet — every FCM/APNs/HMS/ADM provider connection across tenants with mode, state, credential version and fingerprint prefix, latest provisioning job, managed company projects, cloud preflight history, the managed-provider emergency stop, and audited retry/suspend/resume. Route: /admin/provider-fleet.

Competitive baseline

The control-plane scope follows recurring capabilities in OneSignal, Pushwoosh, Knock, Braze, CleverTap, and MoEngage administration: organization/app hierarchy, granular access, API-key governance, environment boundaries, billing and usage, message observability, audit logs, provider configuration, and privacy/security controls. Unsupported enterprise functions are identified as planned rather than represented as active.

Authorization

  • Super-admin login uses POST /api/v1/admin/login.
  • Administrators recover access at /admin/forgot-password; POST /api/v1/admin/forgot-password sends a 30-minute, single-use Resend link and POST /api/v1/admin/reset-password consumes it.
  • Recovery responses are account-enumeration safe, stored tokens are SHA-256 hashes, and issuing or consuming a token revokes older open reset tokens for that operator.
  • Super-admin JWTs include is_super_admin=true, admin_role, and the standard role claim.
  • Owner has full control and exclusively manages operator access. Operations may perform tenant, trial, and delivery recovery mutations. Support and ReadOnly can investigate the control plane but cannot mutate production state.
  • Every authenticated administrator request is revalidated against the live operator record. A deactivated account or stale role claim receives 401 and must sign in again.
  • Every new login creates a server-side session referenced by admin_session_id. Missing, expired, or revoked sessions are rejected even when the JWT signature and lifetime are otherwise valid.
  • Optional TOTP MFA uses an encrypted enrollment secret, a five-minute hashed login challenge, and hashed one-time recovery codes. Recovery codes are shown only once.
  • Five failed password or second-factor attempts within 15 minutes lock the operator for 15 minutes. Responses remain generic; successful login, password recovery, expiry, or an Owner unlock clears the state.
  • At least one active Owner must remain, and an Owner cannot deactivate the account backing the current session.
  • The frontend redirects unauthenticated admin routes to /admin/login.
  • The public admin hostname redirects / to /admin.
  • The configured SuperAdmin__Email value is the authoritative bootstrap and recovery address. When this address changes, update the matching database identity and production environment together, revoke open recovery tokens, then verify login and Resend delivery before handoff.

Administrative APIs

  • GET /api/v1/admin/stats
  • POST /api/v1/admin/forgot-password
  • POST /api/v1/admin/reset-password
  • GET /api/v1/admin/trial-requests
  • GET /api/v1/admin/trial-requests/{id}
  • POST /api/v1/admin/trial-requests/{id}/approve
  • POST /api/v1/admin/trial-requests/{id}/reject
  • POST /api/v1/admin/trial-requests/{id}/resend-setup-email
  • POST /api/v1/admin/trial-requests/{id}/setup-link
  • GET /api/v1/admin/companies
  • POST /api/v1/admin/companies/{id}/suspend
  • POST /api/v1/admin/companies/{id}/activate
  • POST /api/v1/admin/companies/{id}/change-plan
  • GET /api/v1/admin/audit-logs
  • GET /api/v1/admin/apps
  • GET /api/v1/admin/users
  • GET /api/v1/admin/operators
  • GET /api/v1/admin/operators/invitations
  • POST /api/v1/admin/operators/invitations
  • POST /api/v1/admin/operators/invitations/accept
  • POST /api/v1/admin/operators/{id}/activate
  • POST /api/v1/admin/operators/{id}/deactivate
  • PATCH /api/v1/admin/operators/{id}/role
  • POST /api/v1/admin/operators/{id}/unlock
  • GET /api/v1/admin/security/posture
  • GET /api/v1/admin/sessions
  • DELETE /api/v1/admin/sessions/{id}
  • POST /api/v1/admin/sessions/revoke-others
  • GET /api/v1/admin/mfa/status
  • POST /api/v1/admin/mfa/enroll
  • POST /api/v1/admin/mfa/confirm
  • POST /api/v1/admin/mfa/verify-login
  • POST /api/v1/admin/mfa/disable
  • GET /api/v1/admin/api-keys
  • GET /api/v1/admin/campaigns
  • GET /api/v1/admin/delivery-summary
  • GET /api/v1/admin/delivery-engine
  • GET /api/v1/admin/delivery-engine/jobs/{id}
  • POST /api/v1/admin/delivery-engine/jobs/{id}/retry
  • POST /api/v1/admin/delivery-engine/jobs/{id}/dismiss
  • GET /api/v1/admin/provider-fleet — connections with company/app names, credential version, fingerprint prefix, and latest job
  • GET /api/v1/admin/provider-fleet/status — managed flag, effective emergency stop (deployment and operator sources), control-plane readiness, fleet summary, managed projects, preflights
  • GET /api/v1/admin/provider-fleet/preflights/{id} — sanitized preflight checks
  • POST /api/v1/admin/provider-fleet/preflight/{companyId} — read-only Google boundary check (Owner/Operations)
  • POST /api/v1/admin/provider-fleet/emergency-stop — engage or release the operator emergency stop with a reason (Owner/Operations)
  • POST /api/v1/admin/provider-fleet/jobs/{jobId}/retry (Owner/Operations)
  • POST /api/v1/admin/provider-fleet/connections/{id}/suspend — reason required (Owner/Operations)
  • POST /api/v1/admin/provider-fleet/connections/{id}/resume — reason required (Owner/Operations)
  • POST /api/v1/admin/provider-fleet/connections/{id}/verify — non-delivering provider verification (Owner/Operations)
  • GET /api/v1/admin/provider-fleet/connections/{id}/health — latest 25 sanitized health checks

High-risk workflows

Approve a trial request

  1. Open Trial Requests.
  2. Filter to submitted requests.
  3. Review company, requester, website, expected volume, and use case.
  4. Choose the initial plan and trial length.
  5. Approve the request.
  6. Verify the company, owner identity, owner membership, setup-password token, setup-email status, and audit event.
  7. If setup email shows Fallback or Failed, use Resend setup after fixing provider configuration, or use Setup link as a manual recovery path.

Reject a trial request

  1. Open Trial Requests.
  2. Review the request details.
  3. Enter an internal note and customer-facing reason when required.
  4. Reject the request.
  5. Verify no workspace was provisioned and the rejection audit event exists.

Recover setup access for an approved request

  1. Open Trial Requests and filter to approved requests.
  2. Check the Setup email column for attempt count, provider, last attempt time, and error/fallback detail.
  3. Click Resend setup after email configuration is healthy.
  4. Click Setup link only when the owner needs manual recovery before email is healthy.
  5. Copy the generated link from the modal and share it only with the listed workspace owner.
  6. Verify the trial_request.setup_link_generated audit event exists.

Suspend an organization

Suspension takes effect everywhere within 30 seconds: every provider send for the organization stops (CompanySuspended), scheduled campaigns fail instead of retrying, existing workspace sessions and API keys become read-only (403 COMPANY_SUSPENDED on changes), and SDK installations are refused.

  1. Locate the organization in Tenant Directory.
  2. Open its action control.
  3. Enter a meaningful reason of at least five characters.
  4. Confirm suspension.
  5. Verify the new status and the admin.company.suspended audit event.

Reactivate an organization

  1. Locate the suspended organization.
  2. Confirm reactivation.
  3. Verify the active status and the admin.company.activated audit event.

Change a plan

  1. Select the current plan badge in Tenant Directory.
  2. Choose the new plan.
  3. Confirm the entitlement change.
  4. Verify the organization plan and the admin.company.plan_changed audit event.

Stop managed providers in an emergency

  1. Open Provider Fleet.
  2. Select Engage emergency stop and enter a reason of at least five characters.
  3. Verify the banner reports the operator stop as engaged and the admin.provider_emergency_stop.engaged audit event exists.
  4. New managed provisioning and every send through a PushHub-managed connection stop immediately on all API and worker instances. Customer-owned (BYO) connections keep sending.
  5. Releasing the operator stop does not override the deployment setting MANAGED_PROVIDERS_EMERGENCY_STOP; both must be released before managed providers run.

Verify a provider connection

  1. Open Provider Fleet and select Verify on the connection.
  2. PushHub proves provider authority without delivering a notification and records a health check. A definitive rejection marks the connection Failed; an unreachable provider marks it Degraded.
  3. Verify the admin.provider_connection.verified audit event. Scheduled checks run automatically in the worker host every 24 hours (hourly for Degraded).

Suspend or resume a provider connection

  1. Open Provider Fleet and locate the connection.
  2. Select Suspend (or Resume) and enter an operator reason.
  3. A suspended connection stops new sends in any mode; campaigns targeting it fail with ProviderSuspended instead of retrying. Credentials, bundles, and evidence are retained.
  4. Resuming returns a previously verified connection to Ready; a connection that never completed verification returns to Pending.
  5. Verify the admin.provider_connection.suspended or admin.provider_connection.resumed audit event.

Recover a dead-lettered dispatch job

  1. Open Delivery Engine and filter to DeadLettered.
  2. Inspect the job source, company/app ownership, normalized failure, and attempt history.
  3. Correct provider configuration or the reported operational fault before replay.
  4. Enter a recovery reason of at least five characters and retry the job. Final recipient outcomes remain skipped.
  5. Dismiss only when no replay should occur; this is terminal and audited.
  6. Verify admin.dispatch_job_retried or admin.dispatch_job_dismissed in Audit Trail.

Invite and activate a Keme operator

  1. Sign in as an Owner and open Access & Security.
  2. Enter the operator's individual email and name, then choose the least-privilege role that fits their work.
  3. Send the invitation and confirm it appears as Pending in invitation history.
  4. The recipient follows the emailed CTA within three days and creates a strong password.
  5. Confirm the named account appears as Active and the invitation shows Accepted.
  6. Verify admin.operator.invited and admin.operator.invitation_accepted in Audit Trail.

Role changes and deactivation take effect immediately for existing JWTs. Use deactivation when access must stop; do not share or repurpose another operator's credentials.

Enable MFA and store recovery codes

  1. Open Access & Security and choose Set up MFA.
  2. Add the displayed setup key to a standards-compatible authenticator application.
  3. Enter the current six-digit code and confirm enrollment.
  4. Copy all ten recovery codes immediately into the approved secrets manager. They are never displayed again.
  5. Verify MFA shows Enabled and confirm admin.mfa.enabled in Audit Trail.
  6. Sign out and confirm password login advances to the second-factor challenge.

Disabling MFA requires the current password and a valid authenticator or recovery code. The operation revokes other sessions and is audited.

Revoke an administrator session

  1. Open Access & Security and review Active sessions.
  2. Match the user agent, IP, and last-seen time before revoking a device.
  3. Use Sign out others for an incident response or after suspected credential exposure.
  4. Current and revoked tokens are checked against server state on every request.

Recover a locked administrator

  1. Confirm the operator through a separate trusted channel; do not rely on the lockout email alone.
  2. Review recent admin.login.* and admin.mfa.* events, source IPs, and sessions in Access & Security and Audit Trail.
  3. For expected failures, an Owner can use Unlock. Otherwise revoke sessions, require password recovery, and preserve the audit evidence.
  4. Confirm the operator receives the unlock or password-change security alert and can complete MFA.

Rotate credentials and retire bootstrap access

  1. Maintain at least two active named Owner identities. Each must enable MFA, store recovery codes in the approved secrets manager, and test password recovery independently.
  2. In Access & Security, wait for Bootstrap retirement ready. This requires two active non-bootstrap MFA-protected operators and two active Owners.
  3. Rotate Jwt__Key only during a scheduled maintenance window; this signs out every workspace and administrator session. Redeploy all API instances together and verify authentication, health, workers, and callbacks.
  4. Rotate Security__DataEncryptionKey by moving the current value into Security__DataEncryptionKeysRetired, setting a new 32-byte primary key, redeploying, and verifying encrypted provider/MFA reads before removing any retired key. Never remove a retired key until all protected values have been rewritten.
  5. Rotate the bootstrap password through administrator recovery, revoke all of its sessions, store the replacement only in the approved break-glass vault, and stop using it for daily operations.
  6. Final deactivation or removal of the environment bootstrap secret is a deliberate production-owner action after recovery drills pass. Do not automate it from a deployment.
  7. Record the rotation date, operator, affected secret version, verification evidence, and rollback owner in the operational change record—never the secret value.

Implementation

  • Layout: frontend/src/components/layout/AdminLayout.tsx
  • Sign-in: frontend/src/pages/admin/AdminLoginPage.tsx
  • Password recovery: frontend/src/pages/auth/PasswordRecoveryPage.tsx
  • Operator invitation activation: frontend/src/pages/admin/AdminAcceptInvitationPage.tsx
  • Command Center: frontend/src/pages/admin/AdminDashboardPage.tsx
  • Trial Requests: frontend/src/pages/admin/AdminTrialRequestsPage.tsx
  • Tenant Directory: frontend/src/pages/admin/AdminCompaniesPage.tsx
  • Remaining operations pages: frontend/src/pages/admin/AdminOperationsPages.tsx
  • Cross-tenant platform modules: frontend/src/pages/admin/AdminPlatformPages.tsx
  • Delivery engine: frontend/src/pages/admin/AdminDeliveryEnginePage.tsx
  • API controller: backend/PushHub.API/Controllers/SuperAdminController.cs
  • Live operator-session enforcement: backend/PushHub.API/Middleware/ActiveSuperAdminMiddleware.cs
  • Trial request API controller: backend/PushHub.API/Controllers/AdminTrialRequestsController.cs
  • Application handlers: backend/PushHub.Application/Features/SuperAdmin/
  • Trial request handlers: backend/PushHub.Application/Features/TrialRequests/

Cross-tenant read-only inventory endpoints use a dedicated EF Core read model in the super-admin controller. Administrative writes continue through MediatR command handlers so validation and audit creation remain centralized.

Verification

  • Build the API and frontend in Release/production mode.
  • Confirm unauthenticated admin routes redirect to sign-in.
  • Confirm valid super-admin login returns an access token.
  • Confirm stats, companies, plans, audit, and health requests succeed under the admin token.
  • Exercise destructive workflows only with a designated test tenant.
  • Verify desktop and mobile layouts and check the browser console for runtime errors.

Production verification on July 17, 2026 confirmed admin authentication, trial request listing, protected unauthenticated access, public intake, and rejection workflow. The live smoke request was rejected immediately so no test tenant was provisioned.

Production identity verification on September 14, 2026 confirmed the corrected managed-studio administrator mailbox, unchanged password authentication, protected-module access, revocation of older recovery links, and successful Resend delivery to the corrected address. No credential or API key is stored in this repository.

Phase Three verification is documented in docs/PHASE_THREE.md. Delivery-engine smoke checks are read-only; retry and dismissal must use a designated test job.

Managed provider fleet

Keme operators can inspect every provider connection and its latest provisioning job at GET /api/v1/admin/provider-fleet. Failed or scheduled jobs may be requeued through the audited retry endpoint; credentials and configuration payloads are never returned. The feature remains disabled until the infrastructure and physical-device gates in docs/MANAGED_PUSH_INFRASTRUCTURE.md are complete.