Read CSV Average Salary by Department

Medium
File I/O
CSV
Joins
Aggregation

Read CSV Average Salary by Department

Two files are involved: datasets/csv/employees.csv (one row per employee) and datasets/csv/departments.csv (one row per department). How much does a department pay on average?

What to do

  1. Read both CSV files. Their paths are handed to you as paths['employees'] and paths['departments'].
  2. Join the employees with the departments on department_id.
  3. Average salary per department_name and round the result to two decimals.
  4. Sort by avg_salary descending, then by department_name ascending.
  5. Return exactly these columns: department_name, avg_salary.

Signature

File questions receive the paths as a third argument:

from pyspark.sql import functions as F

def solve(spark, dfs, paths):
    employees = spark.read.csv(paths['employees'], header=True, inferSchema=True)
    departments = spark.read.csv(paths['departments'], header=True, inferSchema=True)
    # TODO: join, aggregate and sort
    return employees

The same data is also preloaded as dfs['employees'] and dfs['departments'] if you prefer to skip the file reads.

This question runs in your browser - no sign-in needed. Press Run for the sample data and Submit for the hidden data.

Browser Engine: Evaluates PySpark DataFrame & SQL logic locally in Pyodide WebAssembly.

Input files

2 Files
Loaded automatically

These files are loaded into this browser when you run. Their paths are handed to your code as the third argument: solve(spark, dfs, paths). Read one with spark.read.csv(paths['users'], header=True), or skip the read and use dfs['users'].

paths[employees]
csv
120 rows
datasets/csv/employees.csv
employee_id(int)
first_name(string)
last_name(string)
email(string)
hire_date(date)
department_id(int)
manager_id(int)
salary(double)
paths[departments]
csv
12 rows
datasets/csv/departments.csv
department_id(int)
department_name(string)
location(string)

Return these columns

department_name(string)
avg_salary(double)

Column names and order must match. Sort order is specified in the question.

Browser engine
Local files
Loading Python editor…
Click Run to try the sample cases, or Submit to check the hidden cases. First run caches the browser Python engine (~8–15s once); all code stays on your device.