Spark Style Arena

The manual

Eleven rules, one page

Every challenge in the arena maps to exactly one of these. Read it before a run, or land here from a miss mid-run — the page is always open to you.

Read the original Palantir guide
5 challenges in the bank

Style guide · rule 01

Column Access

Name columns, don’t reach into the DataFrame object.

A column referenced as `df.amount` or `df["amount"]` is bound to that one Python variable. The moment you lift the expression into a helper, reuse it against a differently-named DataFrame, or reorder the pipeline, it stops working — and often fails at runtime rather than at review time. `F.col("amount")` and plain string references describe the column by name and carry no attachment to any particular DataFrame.

  • Reach for `F.col("name")`, or a bare string where the API accepts one.
  • Attribute access (`df.amount`) and subscript access (`df["amount"]`) both bind the expression to one variable.
  • Attribute access is fine in exactly one place: disambiguating same-named columns after a join, via aliases.
  • Column expressions built with `F.col` are portable — you can hoist them into module-level constants and reuse them.
Don’t
  • def enrich(orders):
  • return (
  • orders
  • .withColumn('is_bulk', orders.quantity > 100)
  • .filter(orders['status'] == 'PAID')
  • )
Do
  • def enrich(orders):
  • return (
  • orders
  • .withColumn('is_bulk', F.col('quantity') > 100)
  • .filter(F.col('status') == 'PAID')
  • )

What changed: The expressions no longer mention `orders`, so they survive being extracted into a shared helper.

Palantir style guide
5 challenges in the bank

Style guide · rule 02

Logical Operations

Give complicated booleans a name.

A filter with four clauses on one line is a comment-shaped hole in the code: the reader can see *what* it tests but not *why* it matters. Pulling each clause into a named variable turns the condition into a sentence, and the names survive into the next person’s debugging session. The rough ceiling is about three expressions in a single condition block.

  • Extract multi-clause conditions into named booleans before using them in `filter` or `when`.
  • Aim for at most ~3 expressions in any one condition block.
  • Name the business concept (`is_lapsed_member`), not the mechanics (`cond_1`).
  • Named conditions can be reused across `filter`, `when` and assertions without being retyped.
Don’t
  • def eligible_claims(claims):
  • return claims.filter(
  • (F.col('status') == 'OPEN')
  • & (F.col('amount_usd') > 500)
  • & (F.col('filed_at') >= '2024-01-01')
  • & (F.col('region').isin('EMEA', 'APAC'))
  • )
Do
  • def eligible_claims(claims):
  • is_open = F.col('status') == 'OPEN'
  • is_material = F.col('amount_usd') > 500
  • is_in_scope = (F.col('filed_at') >= '2024-01-01') & F.col('region').isin('EMEA', 'APAC')
  •  
  • return claims.filter(is_open & is_material & is_in_scope)

What changed: The filter now reads as three business conditions instead of four anonymous comparisons.

Palantir style guide
5 challenges in the bank

Style guide · rule 03

Schema Contracts

Use `select` to declare what comes out.

A `select` at the top or bottom of a transform is a contract: these columns, these names, these types. A reader can answer "what does this transform emit?" without executing anything, and a schema change upstream fails loudly in one obvious place instead of leaking through. Keep the contract legible by applying at most one function per column inside it — anything heavier belongs in a `withColumn` above.

  • Open or close every transform with an explicit `select` listing the output columns.
  • Select, rename (`alias`) and cast in that one place.
  • At most one function applied per column inside a `select`.
  • Nested or multi-step derivations go in `withColumn` first, then get selected by name.
Don’t
  • def build_shipments(raw):
  • return raw.withColumn(
  • 'shipped_on',
  • F.to_date(F.regexp_replace(F.trim(F.col('ship_ts')), '/', '-')),
  • )
Do
  • def build_shipments(raw):
  • cleaned_ts = F.regexp_replace(F.trim(F.col('ship_ts')), '/', '-')
  •  
  • return (
  • raw
  • .withColumn('ship_ts_clean', cleaned_ts)
  • .select(
  • F.col('shipment_id').cast('string'),
  • F.to_date('ship_ts_clean').alias('shipped_on'),
  • F.col('carrier_code').alias('carrier'),
  • )
  • )

What changed: The output schema is now readable at a glance, and no column in the `select` stacks three functions.

Palantir style guide
4 challenges in the bank

Style guide · rule 04

Empty Columns

A placeholder is `null`, not a clever string.

When a column has no value yet, say so with `F.lit(None).cast("string")`. Sentinels like `""`, `"NA"` or `0` are real values as far as Spark is concerned: they count in `count()`, they survive `isNotNull()`, they average into `0` in aggregations, and they join to other rows. Every downstream consumer then has to know your private convention — and one of them won’t.

  • Use `F.lit(None).cast("<type>")` for a placeholder column.
  • The cast is not optional: an untyped null column is `void` and breaks writes and unions.
  • Never use `""`, `"NA"`, `"N/A"`, `-1`, `0` or `"UNKNOWN"` as a stand-in for missing.
  • Sentinels quietly corrupt `count`, `avg`, `isNull` and join behaviour downstream.
Don’t
  • def stage_devices(devices):
  • return devices.withColumn('decommissioned_at', F.lit(''))
Do
  • def stage_devices(devices):
  • # Populated by the retirement job; empty until a device is retired.
  • return devices.withColumn('decommissioned_at', F.lit(None).cast('timestamp'))

What changed: `isNull()` now finds these rows, and an average over the column no longer silently includes zeros.

Palantir style guide
5 challenges in the bank

Style guide · rule 05

Comments

Explain why. The code already says what.

A comment that restates the line under it is worse than no comment: it doubles the maintenance surface and goes stale the first time someone edits the code and not the prose. The comments worth writing are the ones the code cannot express — the business rule behind a threshold, the upstream quirk you are working around, the reason an obvious simplification is wrong.

  • Delete comments that paraphrase the code (`# filter the dataframe`).
  • Record business rules, upstream data quirks and non-obvious workarounds.
  • If a comment is needed to explain *what* the code does, rename things instead.
  • A magic number with a comment is usually a named constant waiting to happen.
Don’t
  • def score_leads(leads):
  • # filter the leads
  • active = leads.filter(F.col('is_active'))
  • # add a column called tier
  • return active.withColumn('tier', F.when(F.col('spend') > 20000, 'gold').otherwise('standard'))
Do
  • # Sales counts anything above 20k as gold for commission purposes (see RFC-114).
  • GOLD_SPEND_THRESHOLD_USD = 20000
  •  
  •  
  • def score_leads(leads):
  • active = leads.filter(F.col('is_active'))
  • return active.withColumn(
  • 'tier',
  • F.when(F.col('spend') > GOLD_SPEND_THRESHOLD_USD, 'gold').otherwise('standard'),
  • )

What changed: The surviving comment carries information that is not in the code: where the threshold comes from.

Palantir style guide
5 challenges in the bank

Style guide · rule 06

UDFs

Almost every UDF has a native equivalent.

A Python UDF is a black box to Catalyst: it cannot be reordered, pushed down or fused, and every row has to be serialised out to a Python worker and back. A transform that would have been a single scan becomes a per-row round trip. Native column functions cover the overwhelming majority of what people write UDFs for, and they stay inside the optimiser.

  • Branching → `F.when(...).otherwise(...)`; first non-null → `F.coalesce`.
  • Text → `F.upper`, `F.lower`, `F.trim`, `F.regexp_replace`, `F.regexp_extract`, `F.split`, `F.concat_ws`, `F.substring`.
  • Dates → `F.date_format`, `F.to_date`, `F.datediff`, `F.months_between`.
  • Arithmetic works directly on columns — no UDF needed to add or divide two of them.
  • A UDF blocks predicate pushdown and forces Python serialisation for every row.
Don’t
  • @F.udf(returnType=T.StringType())
  • def normalize_carrier(code):
  • if code is None:
  • return 'UNKNOWN'
  • return code.strip().upper()
  •  
  •  
  • def clean_flights(flights):
  • return flights.withColumn('carrier', normalize_carrier(F.col('carrier_raw')))
Do
  • def clean_flights(flights):
  • return flights.withColumn(
  • 'carrier',
  • F.coalesce(F.upper(F.trim(F.col('carrier_raw'))), F.lit('UNKNOWN')),
  • )

What changed: Same output, but Catalyst can now see through the expression and push the work into the scan.

Palantir style guide
6 challenges in the bank

Style guide · rule 07

Joins

State the join type. Never patch a bad key with `distinct`.

Leaving `how=` off means the reader has to remember the default, and the default is rarely what a reviewer would have chosen deliberately. `how="right"` forces the reader to hold the operands backwards in their head — flip the arguments and use `left`. And if a join fans out unexpectedly, `dropDuplicates()` hides the symptom while the wrong key stays wrong: the row you keep is arbitrary.

  • Always pass `how=` explicitly, even when you want an inner join.
  • Avoid `how="right"` — swap the operand order and use `left`.
  • Disambiguate same-named columns with `.alias()` and qualified references.
  • Never use `.dropDuplicates()` / `.distinct()` to hide a join that fans out.
  • If a join is *meant* to fan out, say so in a comment.
Don’t
  • def attach_customer(orders, customers):
  • joined = orders.join(customers, 'customer_id')
  • return joined.dropDuplicates(['order_id'])
Do
  • def attach_customer(orders, customers):
  • # customers has one row per (customer_id, valid_from); take the current record only.
  • current = customers.filter(F.col('is_current'))
  •  
  • return orders.join(current, on='customer_id', how='left')

What changed: The fan-out is fixed at the key instead of being deduplicated away, and the join type is stated.

Palantir style guide
5 challenges in the bank

Style guide · rule 08

Window Functions

An ordered window without a frame is not what you think.

Add `orderBy` to a window and Spark quietly gives it a default frame of `rangeBetween(unboundedPreceding, currentRow)`. A `F.max` you meant as a partition-wide maximum silently becomes a running maximum, and nothing errors — the numbers are just wrong. Always write the frame you want, even when it matches the default.

  • Specify the frame with `rowsBetween` or `rangeBetween` on every ordered window.
  • An ordered window with no frame defaults to `rangeBetween(unboundedPreceding, currentRow)` — a running aggregate.
  • For a whole-partition aggregate use `rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing)`.
  • `rowsBetween` counts physical rows; `rangeBetween` compares the ordering value, so ties share a frame.
  • Pass `ignorenulls=True` to `F.first` / `F.last` when you want the last *known* value.
Don’t
  • def peak_reading(sensors):
  • window = Window.partitionBy('sensor_id').orderBy('recorded_at')
  • return sensors.withColumn('peak_temp_c', F.max('temp_c').over(window))
Do
  • def peak_reading(sensors):
  • window = (
  • Window
  • .partitionBy('sensor_id')
  • .orderBy('recorded_at')
  • .rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing)
  • )
  • return sensors.withColumn('peak_temp_c', F.max('temp_c').over(window))

What changed: Without the frame this returned a running maximum. With it, every row sees the true partition peak.

Palantir style guide
5 challenges in the bank

Style guide · rule 09

Expression Chaining

Break the chain where the intent changes.

A twelve-call chain has no place to put a breakpoint, no name for any intermediate state, and a stack trace that points at the whole thing. Cap chains at roughly five statements and split at the seams where the *kind* of work changes — ingest and filter, then join, then derive, then aggregate, then the final select. Each named DataFrame is free documentation and a free debugging handle.

  • Keep a chain to about five statements.
  • Split where the kind of operation changes: filter → join → derive → aggregate → select.
  • Name each intermediate DataFrame after what it contains, not `df2`.
  • Named intermediates are inspectable in a notebook and nameable in a stack trace.
Don’t
  • def daily_revenue(orders, refunds):
  • return (
  • orders
  • .filter(F.col('status') == 'PAID')
  • .join(refunds, on='order_id', how='left')
  • .withColumn('net_usd', F.col('total_usd') - F.coalesce(F.col('refund_usd'), F.lit(0)))
  • .withColumn('order_date', F.to_date('placed_at'))
  • .groupBy('order_date')
  • .agg(F.sum('net_usd').alias('revenue_usd'))
  • .orderBy('order_date')
  • )
Do
  • def daily_revenue(orders, refunds):
  • paid_orders = orders.filter(F.col('status') == 'PAID')
  •  
  • with_refunds = paid_orders.join(refunds, on='order_id', how='left')
  •  
  • net_orders = (
  • with_refunds
  • .withColumn('net_usd', F.col('total_usd') - F.coalesce(F.col('refund_usd'), F.lit(0)))
  • .withColumn('order_date', F.to_date('placed_at'))
  • )
  •  
  • return (
  • net_orders
  • .groupBy('order_date')
  • .agg(F.sum('net_usd').alias('revenue_usd'))
  • )

What changed: Four named stages instead of one eight-call expression — each one is inspectable on its own.

Palantir style guide
4 challenges in the bank

Style guide · rule 10

Multi-line Expressions

Parentheses hold the expression together, never a backslash.

A `\` continuation is invisible whitespace-sensitive syntax: one trailing space after it and the file stops parsing, with an error pointing somewhere unhelpful. A single pair of wrapping parentheses does the same job, survives reformatting, and lets you put one method call per line with the dot leading — which makes the diff when someone adds a step exactly one line long.

  • Wrap a multi-line expression in one pair of parentheses.
  • Never use `\` line continuations.
  • One method call per line, with the `.` leading the line.
  • Leading dots keep diffs to a single added line when a step is inserted.
Don’t
  • def active_accounts(accounts):
  • return accounts.filter(F.col('is_active')) \
  • .withColumn('tier', F.upper(F.col('tier'))) \
  • .select('account_id', 'tier')
Do
  • def active_accounts(accounts):
  • return (
  • accounts
  • .filter(F.col('is_active'))
  • .withColumn('tier', F.upper(F.col('tier')))
  • .select('account_id', 'tier')
  • )

What changed: No invisible syntax, and inserting a step now touches exactly one line in the diff.

Palantir style guide
5 challenges in the bank

Style guide · rule 11

Craft & Conventions

The small habits that keep a repo readable.

None of these are subtle on their own, and all of them compound. A magic literal buried in a filter is a business rule nobody can find. A file with six transforms in it is a merge conflict waiting to happen. `df1`/`df2`/`tmp` force every reader to re-derive what the code already knew. Pure helpers that are testable get tested; ones that need a Spark session don’t.

  • Hoist magic literals to module-level constants with meaningful names.
  • `snake_case` for column names and variables, consistently.
  • Keep transform files short and each function focused on one job.
  • Import as `from pyspark.sql import functions as F` — not `import *`, not bare `col`.
  • Extract pure helpers so they can be unit-tested without a Spark session.
  • Name DataFrames for what they hold: `paid_orders`, not `df2`.
Don’t
  • from pyspark.sql.functions import *
  •  
  •  
  • def f(df1):
  • df2 = df1.filter(col('LatencyMs') > 250)
  • return df2.withColumn('IsSlow', lit(True))
Do
  • from pyspark.sql import functions as F
  •  
  • SLOW_REQUEST_THRESHOLD_MS = 250
  •  
  •  
  • def flag_slow_requests(requests):
  • slow_requests = requests.filter(F.col('latency_ms') > SLOW_REQUEST_THRESHOLD_MS)
  • return slow_requests.withColumn('is_slow', F.lit(True))

What changed: Every name now tells you something, and the threshold is findable with one grep.

Palantir style guide