Machine Learning Research Engineer @ Windborne – Take-Home Assignment

Machine Learning Engineer Take-Home – IFS Cycle Upgrade Analysis

Summary:

Initializing AIFS v1.1 with cycle 50R1 degrades certain variables–Q50, Q100, and Q150 are most negatively affected. Since we did not have parallel data, we compared 49R1 to 50R1 at the same season across different years and at different, but nearby periods within the same year. In both cases, for Q50, Q100, and Q150, forecasts initialized from 50R1 showed higher RMSE at day 1, decreasing at longer leads.

Same Season, Different Year:

Within 2026 Cutover:

We focused on training three methods and tested several input adapters. We found that adapting Q50, Q100, and Q150 separately performed best. For Q50, we matched the median and spread to its 49R1 distribution outside of 20°S-20°N, and left tropical unchanged. For Q100 and Q150, we estimated the change in their result. We subtract the median residual between 49R1 and 50R1. Here is a snippet of the corrected 50R1 versus uncorrected 50R1 scorecard:

Note that negative values indicate improvement.

To conclude, Q50, Q100, and Q150 showed the largest 49R1 to 50R1 distribution changes. Similarly, the largest forecast degradations occur in those same humidity fields, so we focused on correcting those inputs. After several ablations, we ended up with our hybrid strategy. We did not have access to parallel 49R1/50R1 analysis, and the experiment uses small datasets. Future work can address those issues.

Other artifacts:

Research Log:

  • [13:12] The assignment states “ECMWF recently upgraded their Integrated Forecasting System (IFS) from cycle 49R2 to cycle 50R1 in May 2026,” but the referenced link and resources suggest that they upgraded from cycle 49R1, not 49R2. So, going forward, I will keep in mind that 49R2 might have been a typo.
  • [13:40] The inputs are the IFS operational analysis, cycle 49R1 for AIFS v1.1 and cycle 50R1 for AIFS v2. According to 5.6 Sol, no public same-time 49R1/50R1 analysis exists, so we are going to have to study the input distributions on the same calendar season across two years, or on the same year across two calendar seasons. Sol has been tasked with both.
  • [13:58] Found two scorecards: AIFS v2 from 50R1 versus AIFS v1.1 from 49R1, and AIFS v2 from 50R1 versus AIFS v1.1 from 50R1. We could combine these two (especially using “against observations” rows). Sol has been tasked with this.
  • [14:35] The AIFS v1.1 input distribution shift seems to be concentrated in humidity at 50 and 100 hPa (in brackets, q50 and q100). This matches the inferred combined scorecards and the ECMWF newsletter.
  • [15:17] After forecasting and verifying, we see that cycle 50R1 produces worse performance at 50 and 100 hPa, especially for Q50 and Q100, keeping in mind that the forecasts are for different periods.
  • [16:10] Several findings:
    • Applying residual correction by subtracting the median of the residual for each variable level worsened Q50 forecast RMSE, but improved Q100 and Q150 forecast RMSEs.
    • What about excluding Q50? Q100/Q150 improved, operational guardrail mean worsened.
    • Adding a parameter to control correction: Q100/Q150 improved less, but operational guardrail mean worsened less.
    • Correcting only Q100 and Q150, leaving everything else fixed: best so far.
    • Applying per-latitude band feature normalization to Q50 by centering the values using 50r1 median, scaling it using the ratio or robust spreads, and then shifting it to the 49r1 median: looked good apart from tropical (+77%).
    • Applying above Q100 and Q150 correction globally and selectively applying feature normalization to Q50: best so far.

Time Log:

  • [12:56] Starting the assignment. Reading the PDF.
  • [13:03] To better understand the problem, I am going to do a little bit of research about the background and go through some of the shared resources.
  • [13:20] I am going to get an agent to find out what data is publicly available while I go through the resources.
  • [13:36] For the first part of the assignment, “Investigate the effect of IFS’s shift to cycle 50R1 on AIFS v1.1’s forecast output,” we can look at how the input distribution has changed, how much AIFS v1.1’s outputs and accuracy changed, whether errors grow with forecast horizon or concentrate in certain variables or regions, etc.
  • [13:50] While Sol is working on studying the input distributions, I will look to see if there are any publicly available scorecards comparing AIFS v1.1 from 50r1 and from 49r1.
  • [14:02] To study how much AIFS v1.1’s outputs and accuracy changed, I am going to need some sort of “ground truth” or reference target–need to find what would be most appropriate as reference targets.
  • [14:07] Sol produced several artifacts comparing the two input distributions and combining scorecards. I will review the artifacts and see if they align with the newsletter: https://www.ecmwf.int/en/newsletter/185/earth-system-science/upgrade-ifs-cycle-50r1.
  • [14:40] We have investigated the AIFS v1.1 input distribution shift. The next step is to download the AIFS v1.1 model, run forecasts and verify against ERA5 (also to be downloaded) which will act as our reference targets.
  • [14:44] Moving code and data to a GPU server.
  • [14:49] After we download the model and data, run forecasts, and check and verify against our reference targets, we can start implementing strategies to tackle the distribution shift problem. There are several approaches we could take to address the distribution shift problem, either by adapting the model or the data. Intuitively, I lean toward keeping the model fixed and adapting the data. From the model side we can update the batch-norm EMAs (not a lot of training-free options). From the data side, we can use normalization or mix some statistics. We can apply different augmentations and then average the outputs. We might be able to withhold certain variables. Going to do a quick search.
  • [14:53] Delegated Sol to investigate training-free distribution shift methods.
  • [15:10] Sol suggests:
    • ECMWF’s cycle-shift experiment argues against relying only on a global mean/variance correction–need to verify this.
    • From parallel 49r1/50r1 analyses, estimate d=median⁡(x50−x49) for each variable, level and suitable stratum; use x50=x50−d. (Requires parallel analysis… will ask Sol to run.)
    • x49=μS49+σS49/σ50 * (​x50−μ50)​. Use area-weighted, seasonally matched statistics, preferably robust median/IQR.
    • Several other suggestions.
  • [15:25] I will download the assignment PDF and pass it to Sol alongside the artifacts generated so far to evaluate task performance.
  • [15:35] Baseline work is good. Need to delegate several ideas for correcting 50r1 to 49r1 to Sol, each idea should produce a scorecard of corrected 50r1 vs uncorrected 50r1.
  • [16:01] I will review the results and then write the summary and answer the questions.
  • [16:38] I will get Sol to iterate on our best correction so far, while I complete the summary.

What would you do with more time?

If I had more time, I would do the following:

  • Data:
    • Verify whether same-time 49r1/50r1 analysis exists (ECMWF/MARS). If it does exist, download it.
    • I generated several “scorecards” comparing 49r1 and 50r1 on several axes, e.g., normalized mean shift, variance chance, etc. Include more variables and review it more carefully.
  • Forecasts:
    • I created two forecast scorecards: 1) same season, different year: 49r1 from 13-20 May 2025, and 50r1 from 13-20 May 2026; and 2) within-2026 cutover: 49r1 from 4-11 May 2026, and 50r1 from 13-20 May 2026. In the case where we have same-time analysis data, one scorecard is enough.
    • Verify the target reference: should it be ERA5 or something else?
    • Develop an evaluation framework, so that we can quickly generate scorecards and a table of corrections from forecasts.
  • Corrections:
    • From the model side, I do not think there are many reasonable training-free methods for dealing with distribution shifts.
    • From the data side, there are a lot of ideas that we can try.

What AI tools did you use and how?

I used GPT-5.6-Sol to search for references, and in Codex, to write all of the code, including downloading data, connecting to my GPU server, running forecasts, etc. Typically, especially for code that I want to reuse, I would more carefully review the code and guide the design to keep it in a maintainable shape. But given the time constraint, I had to delegate all of the code to Sol. I also use OpenWhispr for STT.