[Build] Establish the StreamPark 3.0 architecture baseline

[wolfboys]

Purpose

This PR establishes a clean architecture baseline for StreamPark 3.0.

The change intentionally removes obsolete modules, duplicated implementations, unused dependencies, and historical compatibility layers. It also rebuilds several core boundaries so that subsequent backend, frontend, and runtime improvements can be developed on a simpler and more maintainable foundation.

This is a deliberate 3.0 cleanup. Deprecated internal structures are removed instead of being carried forward indefinitely.

Major Changes

1. Simplify the module structure

• Remove the obsolete streampark-spark-cli module.
• Remove the standalone streampark-flink-udf module.
• Move FlinkShimsProxy into the Flink client API and remove the standalone streampark-flink-proxy module.
• Rename streampark-common-scala-bridge to the clearer streampark-scala module.
• Reorganize Flink client APIs into explicit request, response, configuration, and implementation packages.
• Remove unused classes, utilities, Maven dependencies, and outdated package structures.

2. Retire the legacy E2E implementation

• Remove the deprecated streampark-e2e modules and replace their workflow with a temporary required-check compatibility job.
• Remove E2E infrastructure that is tightly coupled to the previous frontend and backend architecture.
• Reserve the E2E workflow for reconstruction after the StreamPark 3.0 frontend and backend boundaries are stabilized.

The removal is intentional and temporary at the project level. A new E2E workflow will be designed around the refactored architecture.

3. Rebuild the common configuration framework

Replace the historical common.conf implementation with a typed and immutable configuration model under common.configuration.

The new configuration framework provides:

• Typed ConfigOption declarations.
• Immutable configuration snapshots.
• Explicit configuration source precedence.
• Configuration origin tracking and actionable conversion errors.
• Structured YAML, HOCON, properties, environment, system property, command-line, and runtime sources.
• Module-owned option catalogs without global option registries.
• Atomic process-level configuration publication.
• Constant-style workspace access backed by a validated configuration snapshot.
• A dedicated Console-side Spring configuration initializer.

Common configuration remains independent of Flink and Spark runtime APIs.

4. Fix Flink configuration compatibility

Reimplement Flink configuration discovery and parsing according to the target Flink version instead of the StreamPark compile-time Flink version.

The supported behavior is:

• Flink versions before 1.19 load flink-conf.yaml with the legacy Flink line parser.
• Flink 1.19 and 1.20 support both filenames and prefer flink-conf.yaml when both files exist.
• flink-conf.yaml uses legacy parsing, while config.yaml uses standard YAML parsing.
• Flink 2.0 and later load only config.yaml with standard nested YAML parsing.
• Nested maps are flattened into dot-separated Flink configuration keys.
• YAML collections are preserved using the representation expected by Flink.

This resolves the historical compatibility issues when StreamPark manages Flink 1.x and 2.x installations from the same deployment.

5. Consolidate the Flink shims architecture

• Remove the duplicated streampark-flink-shims-base-v2 module.
• Consolidate shared Flink behavior into a single shims-base implementation.
• Remove the obsolete shims test module and redundant implementations.
• Remove the unsupported Flink 1.17 shim.
• Make Flink 1.18, 1.19, 1.20, 2.0, 2.1, 2.2, and 2.3 explicit reactor modules.
• Keep only APIs supported by every version in shared abstract classes.
• Keep legacy APIs such as registerDataStream and table-function registerFunction overloads only in compatible Flink 1.x implementations.
• Preserve classloader isolation through serializable StreamPark request and response contracts.

No general utilities or target-independent configuration logic are placed in the shims layer.

6. Clarify Console responsibilities

• Move Spring configuration composition into SpringConfigurationInitializer.
• Keep Console-specific options in the Console module.
• Move Flink environment and application configuration behavior out of persistence entities.
• Keep database entities focused on table mappings and persisted state.
• Introduce typed REST response envelopes while retaining the existing wire format.
• Separate request DTOs, response DTOs, assemblers, services, and entities more clearly.

7. Improve HTTP client and proxy handling

• Introduce a shared OkHttp client with connection pooling and bounded retries for idempotent requests.
• Rework Console proxy handling to preserve multi-value headers and response status.
• Filter hop-by-hop and connection-declared headers correctly.
• Ensure upstream responses are always closed.
• Avoid creating a new HTTP client for every proxy request.

8. Improve repository and developer tooling

• Add a root development launcher for building, starting, stopping, and inspecting a source checkout.
• Align build and distribution banners with the StreamPark 3.0 version.
• Refresh repository metadata and remove the retired E2E required check and label mapping.
• Update NOTICE files for 2026 and ignore the local pnpm store.

9. Improve maintainability

• Remove stale and inaccurate comments.
• Add class-level and core implementation documentation.
• Simplify method names and keep new production and test method names within 40 characters.
• Remove redundant and overlapping tests.
• Update AGENTS.md to describe the current module boundaries and architecture rules.
• Preserve Apache license, formatting, Checkstyle, and RAT requirements.

10. Harden the Flink client submission path

• Resolve each serialized submit request once into an immutable submission snapshot.
• Keep configuration parsing, build validation, artifact resolution, and deterministic classpath discovery at the request boundary.
• Remove the repetitive FlinkConfigurationOps.setIfPresent wrapper and apply typed Flink options directly.
• Preserve the historical job name and job ID placeholders in job-scoped Flink defaults while leaving shared session and remote defaults unchanged.
• Close packaged programs, cluster clients, and descriptors on all success and failure paths.
• Consolidate savepoint request types and make request validation failures explicit.
• Use job-oriented naming for user Flink programs while retaining official Flink Application Mode terminology and serialized compatibility fields.
• Reorder public and private methods, simplify method names, and document core submission behavior.

11. Keep the E2E check mergeable during migration

The legacy E2E implementation remains removed. A minimal compatibility workflow temporarily publishes the existing required E2E - Result context because the branch-protection change in .asf.yaml cannot take effect until this PR is merged. The compatibility workflow can be removed after ASF synchronizes the updated branch protection.

Compatibility Notes

This PR intentionally introduces internal breaking changes for StreamPark 3.0:

• The legacy org.apache.streampark.common.conf package is replaced by org.apache.streampark.common.configuration.
• Obsolete modules and internal APIs are removed.
• Flink 1.17 shims are removed.
• The previous E2E implementation is removed and will be rebuilt separately.
• Flink client request and response types are reorganized around explicit package boundaries.

User-facing configuration keys retain their defined compatibility aliases where required.

Verification

The following checks have been completed:

• streampark-common: 102 tests passed.
• Console configuration, proxy, Flink environment, SQL, and savepoint tests: 12 tests passed.
• Flink client request resolution, configuration precedence, placeholder compatibility, and REST submission tests passed.
• Full Flink shims reactor build passed for Flink 1.18 through 2.3.
• Spotless and Checkstyle passed for the tested modules.
• Apache RAT passed for the common, client, and shims modules.
• Flink configuration parsing was verified with:
  • Flink 1.16.3 flink-conf.yaml
  • Flink 1.19.1 config.yaml
  • Flink 2.3.0 config.yaml

Follow-up Work

• Rebuild the E2E framework and CI workflow against the finalized 3.0 frontend and backend contracts.
• Continue reducing deprecated compatibility APIs after downstream migrations are complete.
• Run the complete release CI matrix before the StreamPark 3.0 release.

[sonarqubecloud]

Quality Gate passed

Issues
237 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.6% Duplication on New Code

See analysis details on SonarQube Cloud

[VampireAchao]

LGTM