SELECT 
    sd.skills,
    ROUND(MEDIAN(jpf.salary_year_avg)) as median_salary,
    COUNT(jpf.*) as demand_skills,
    COUNT(jpf.salary_year_avg) as corrected_demand_count
FROM job_postings_fact as jpf
INNER JOIN skills_job_dim as sjd
        ON jpf.job_id = sjd.job_id
INNER JOIN skills_dim as sd
    ON sjd.skill_id = sd.skill_id
WHERE 
    jpf.job_title_short LIKE 'Data Engineer'
    AND
    jpf.job_work_from_home = False

GROUP BY sd.skills
HAVING COUNT(jpf.*) > 100
ORDER BY 
    median_salary DESC
LIMIT 25
; 


/*

┌────────────┬───────────────┬───────────────┬────────────────────────┐
│   skills   │ median_salary │ demand_skills │ corrected_demand_count │
│  varchar   │    double     │     int64     │         int64          │
├────────────┼───────────────┼───────────────┼────────────────────────┤
│ rust       │      210000.0 │           232 │                     23 │
│ golang     │      184000.0 │           912 │                     39 │
│ terraform  │      184000.0 │          3248 │                    193 │
│ spring     │      175500.0 │           364 │                     33 │
│ neo4j      │      170000.0 │           277 │                     11 │
│ gdpr       │      169616.0 │           582 │                     22 │
│ zoom       │      168438.0 │           127 │                     12 │
│ graphql    │      167500.0 │           445 │                     28 │
│ mongo      │      162250.0 │           265 │                     14 │
│ fastapi    │      157500.0 │           204 │                      3 │
│ bitbucket  │      155000.0 │           478 │                      9 │
│ django     │      155000.0 │           265 │                      5 │
│ crystal    │      154224.0 │           129 │                      3 │
│ c          │      151500.0 │           444 │                     23 │
│ atlassian  │      151500.0 │           249 │                      9 │
│ typescript │      151000.0 │           388 │                     39 │
│ kubernetes │      150500.0 │          4202 │                    147 │
│ node       │      150000.0 │           179 │                     22 │
│ ruby       │      150000.0 │           736 │                     48 │
│ css        │      150000.0 │           262 │                     13 │
│ airflow    │      150000.0 │          9996 │                    386 │
│ redis      │      149000.0 │           605 │                     17 │
│ vmware     │      148798.0 │           136 │                      2 │
│ ansible    │      148798.0 │           475 │                     14 │
│ jupyter    │      147500.0 │           400 │                     15 │
├────────────┴───────────────┴───────────────┴────────────────────────┤
│ 25 rows                                                   4 columns │
└─────────────────────────────────────────────────────────────────────┘

*/




SELECT 
    sd.skills,
    ROUND(MEDIAN(jpf.salary_year_avg)) as median_salary,
    COUNT(jpf.*) as demand_count,
    ROUND(LN(COUNT(jpf.*)),2) as demand_count,
    ROUND((LN(COUNT(jpf.*)) * MEDIAN(jpf.salary_year_avg))/1_000_000,2) as optimal_score
FROM job_postings_fact as jpf
INNER JOIN skills_job_dim as sjd
        ON jpf.job_id = sjd.job_id
INNER JOIN skills_dim as sd
    ON sjd.skill_id = sd.skill_id
WHERE 
    jpf.job_title_short LIKE 'Data Engineer'
    AND
    jpf.job_work_from_home = True
    AND
    jpf.salary_year_avg IS NOT NULL
GROUP BY sd.skills
HAVING COUNT(jpf.*) > 100
ORDER BY 
    optimal_score DESC
LIMIT 25
; 


/*
┌────────────┬───────────────┬──────────────┬──────────────┬───────────────┐
│   skills   │ median_salary │ demand_count │ demand_count │ optimal_score │
│  varchar   │    double     │    int64     │    double    │    double     │
├────────────┼───────────────┼──────────────┼──────────────┼───────────────┤
│ terraform  │      184000.0 │          193 │         5.26 │          0.97 │
│ python     │      135000.0 │         1133 │         7.03 │          0.95 │
│ aws        │      137320.0 │          783 │         6.66 │          0.91 │
│ sql        │      130000.0 │         1128 │         7.03 │          0.91 │
│ airflow    │      150000.0 │          386 │         5.96 │          0.89 │
│ spark      │      140000.0 │          503 │         6.22 │          0.87 │
│ kafka      │      145000.0 │          292 │         5.68 │          0.82 │
│ snowflake  │      135500.0 │          438 │         6.08 │          0.82 │
│ azure      │      128000.0 │          475 │         6.16 │          0.79 │
│ java       │      135000.0 │          303 │         5.71 │          0.77 │
│ scala      │      137290.0 │          247 │         5.51 │          0.76 │
│ git        │      140000.0 │          208 │         5.34 │          0.75 │
│ kubernetes │      150500.0 │          147 │         4.99 │          0.75 │
│ databricks │      132750.0 │          266 │         5.58 │          0.74 │
│ redshift   │      130000.0 │          274 │         5.61 │          0.73 │
│ gcp        │      136000.0 │          196 │         5.28 │          0.72 │
│ nosql      │      134415.0 │          193 │         5.26 │          0.71 │
│ hadoop     │      135000.0 │          198 │         5.29 │          0.71 │
│ pyspark    │      140000.0 │          152 │         5.02 │           0.7 │
│ mongodb    │      135750.0 │          136 │         4.91 │          0.67 │
│ docker     │      135000.0 │          144 │         4.97 │          0.67 │
│ r          │      134775.0 │          133 │         4.89 │          0.66 │
│ go         │      140000.0 │          113 │         4.73 │          0.66 │
│ github     │      135000.0 │          127 │         4.84 │          0.65 │
│ bigquery   │      135000.0 │          123 │         4.81 │          0.65 │
├────────────┴───────────────┴──────────────┴──────────────┴───────────────┤
│ 25 rows                                                        5 columns │
└──────────────────────────────────────────────────────────────────────────┘


Summary
This analysis examines the optimal skills for remote Data Engineer roles by combining salary 
and demand into a single composite score — calculated as the log of job postings multiplied by 
median salary. This approach rewards skills that are both well-compensated and widely requested, 
avoiding the trap of chasing either high pay in niche roles or high volume in lower-paying ones. 
The dataset covers 25 skills, each appearing in at least 100 remote job postings with a reported 
salary, giving the findings strong statistical grounding.
The results reveal a clear tiering: Terraform, Python, AWS, and SQL occupy the top cluster with 
optimal scores between 0.91–0.97, driven by strong salary floors ($130K–$184K) and massive demand. 
Below them sits a rich mid-tier — Airflow, Spark, Kafka, Snowflake — where slightly lower demand 
is offset by above-average salaries, particularly in streaming and orchestration. The bottom cluster 
(Docker, Go, GitHub, BigQuery) still commands solid $135K+ medians but trails on demand volume, 
making them valuable secondary skills rather than primary targets for career positioning.

Key Findings

Terraform is the highest-value single skill with the top optimal score (0.97) 
and the highest median salary of any skill in the dataset at $184,000 — nearly 
$50K above the group average. Despite relatively modest demand (193 postings), 
its salary premium is so pronounced that it outscores even Python and SQL. 
Infrastructure-as-code expertise is rare, commands a significant wage premium, 
and is directly suited to remote work since all provisioning is done via CLI and 
APIs.
Python and SQL are the volume anchors of the market, each appearing in over 1,100 
remote postings — more than double any other skill — and both scoring 0.91–0.95. 
Their median salaries ($135K and $130K respectively) are solid but not exceptional; 
their dominance comes from ubiquity. For anyone enteri_ng the field, these two skills 
represent the lowest-risk, highest-return investment — nearly every role expects them.

AWS leads the cloud platforms, outscoring Azure and GCP by a notable margin (0.91 vs. 
0.79 and 0.72). All three cloud providers sit in the top half of the table, but AWS 
uniquely combines strong demand (783 postings) with the highest cloud median salary 
($137,320). This reflects AWS's continued dominance in enterprise data infrastructure 
and its tight integration with modern data stacks. Azure and GCP remain important but 
are stronger as complementary skills.

Streaming and orchestration tools (Kafka, Airflow, Spark) offer a high salary-to-demand 
ratio, clustering between $140K–$150K median salaries with moderate but healthy demand 
(292–503 postings each). These are the skills most likely to differentiate a mid-career 
engineer, signalling the ability to manage real-time pipelines and complex DAG-based 
workflows autonomously — exactly the profile remote-first teams are hiring for.

Infrastructure and containerisation skills (Kubernetes, Git, Docker) punch above 
their demand weight on salary — Kubernetes in particular has the second-highest median 
salary in the dataset at $150,500, despite appearing in only 147 postings. This 
niche-but-lucrative pattern suggests that DevOps-adjacent data engineers who can 
manage containerised workloads command a meaningful premium, even if the absolute 
number of such roles is smaller. These are strong specialisation targets for engineers 
already solid in Python/SQL/cloud.

*/
™